FLUID 0.7.6 (preview) — Declarative Packaging Modes

Preview — not yet stable

0.7.6 is a preview. A contract uses it only by declaring fluidVersion: "0.7.6", and its fields and their meaning can still change before it is promoted to stable. The latest stable version is 0.7.5. See Stable and preview versions.

0.7.6 adds declarative ownership of infrastructure containers (packaging), a declared list of downstream consumers, pins for upstreams in another mesh, and binding-level fields that let one cloud-neutral contract carry encryption, identity mapping and data expiry to each platform it is deployed on.

Schema: fluid-schema-0.7.6.json · Reference: specs/0.7.6/fluid-spec.html · Diff: diff-0.7.5-to-0.7.6.md
Compatibility: additive over 0.7.5. scripts/check-compat.py --from 0.7.5 --to 0.7.6 reports no narrowing change, so a valid 0.7.5 contract stays valid when it declares "0.7.6".

What is new

Every addition below is optional. The list is derived from the schemas: these are the property paths present in 0.7.6 and absent in 0.7.5.

Path	What it declares
`packaging`	Contract-wide ownership mode for the containers a product writes into.
`exposes[].binding.packaging`	A per-binding override of `packaging`.
`consumers[]`	The downstream artifacts (dashboards, notebooks, applications…) built on this product.
`consumes[].upstreamWorkspace`, `consumes[].upstreamDigest`	A pin on an upstream that lives in another mesh.
`exposes[].semantics.measures[].aggParams`	Parameters for an aggregation, used by `agg: percentile`.
`exposes[].binding.encryption.kms`	The key that data at rest is encrypted with.
`exposes[].binding.principals`	What each logical principal named in the contract is on this binding's platform.
`exposes[].binding.governance.lakeFormation.bucketPolicy`	Which grantees may read a Lake Formation location directly from object storage.
`exposes[].lifecycle.expire`	Whether data older than the expose's `retention` is deleted.
`exposes[].policy.privacy.masking[].params.{keepFirst, keepLast, saltEnv, keyEnv}`	Typed parameters for masking rules.

fluidVersion accepts "0.7.6" in addition to "0.7.3", "0.7.4" and "0.7.5".

0.7.6 also writes down the meaning of a field that already existed: exposes[].policy.authz.columnRestrictions (below).

Reading the 0.7.6 schema descriptions

Many 0.7.6 field descriptions explain what the reference implementation does with the field (fluid apply, the OpenTofu resources it emits, fluid verify). Those passages are non-normative: they describe one implementation. A FLUID document is valid or invalid by its schema; an implementation that validates the same documents conforms whether or not it provisions the same resources.

`packaging` — who owns the containers

packaging:
  mode: shared            # isolated | shared — the default for every container kind
  pool: analytics-eu      # id of the platform-owned pool the product writes into
  containers:             # per-kind overrides: bucket | database | dataset | schema | warehouse | cluster
    warehouse: isolated   # hybrid tier: shared database, owned warehouse

isolated — the product creates and owns the container.
shared — the container already exists and belongs to the platform; the product writes into it but does not create or own it.
exposes[].binding.packaging has the same shape and overrides packaging key by key for that binding. An absent packaging block declares nothing, and the 0.7.6 schema describes that case as the behaviour from before the block existed.

The schema constrains the shape; it does not require pool. The reference implementation refuses a contract in which any container resolves to shared and no pool is set (packaging (pool-required)), and refuses isolated for the cluster container kind on a Confluent or Kafka binding (cluster-isolated-unsupported). Which platform resource each container kind stands for is implementation-defined; the 0.7.6 schema's description of containers lists the reference implementation's mapping.

`consumers` — what is built on this product

consumers:
  - name: weekly_revenue_dashboard     # required — identifier, unique within the contract
    type: dashboard                    # required — dashboard | notebook | analysis | ml | application
    label: Weekly Revenue Dashboard
    maturity: high                     # high | medium | low
    owner: { team: finance-analytics, email: finance-analytics@example.com }
    url: https://bi.example.com/dashboards/42
    exposeIds: [orders]                # which output ports it reads; absent = all

The shape follows dbt exposures. The block is purely declarative: the 0.7.6 schema says consumers never affect plan or apply, and marks the block as reserved — the reference implementation carries it through but does not yet act on it.

Cross-mesh pins on `consumes[]`

consumes:
  - productId: crm.bronze.accounts
    exposeId: accounts
    upstreamWorkspace: crm-mesh                  # the other mesh that owns this upstream
    upstreamDigest: "sha256:4f1c9a0d6e2b7c8a1f3e5d9b0c2a4e6f8d1b3c5e7a9f0b2d4c6e8a1b3d5f7091"

upstreamWorkspace names the federated workspace that owns the upstream. Omit it for an upstream in the same workspace.
upstreamDigest is the digest of the upstream contract this product was composed against, written sha256: followed by 64 lowercase hex characters.
Setting upstreamWorkspace requires upstreamDigest. The schema states this with dependentRequired, a JSON Schema Draft 2020-12 keyword. 0.7.6 is the first FLUID schema to use a keyword that Draft 7 does not have, so a validator that applies Draft 7 rules ignores this requirement. Every FLUID schema declares Draft 2020-12 (see Validation semantics).

What the digest is computed over is not defined by the standard; it is implementation-defined. In the reference implementation it is a SHA-256 over the parsed, canonicalised upstream contract, printed by fluid contract digest <contract>, and fluid apply compares it with the upstream's current digest to detect that the upstream changed since composition.

`aggParams` on semantic measures

semantics:
  measures:
    - name: p95_amount
      agg: percentile
      expr: amount
      aggParams:
        percentile: 0.95              # in [0, 1]; 0.5 is the median
        useDiscretePercentile: false  # discrete instead of interpolated percentile

aggParams mirrors agg_params in dbt-semantic-interfaces. In 0.7.6 it is used by agg: percentile.

Binding fields: encryption, principals, Lake Formation bucket policy

binding:
  platform: gcp
  format: bigquery_table
  location: { project: acme-analytics, dataset: sales, table: orders, region: europe-west1 }
  encryption:
    kms: product                      # product (default) | none | a key reference
  principals:                         # logical principal -> identity on this platform
    "group:analysts": "group:analysts@example.com"
    "group:stewards": ["group:data-stewards@example.com"]

encryption.kms accepts product, none, an AWS KMS alias (alias/<name>), an AWS KMS key or alias ARN, or a Google Cloud KMS key name (projects/<p>/locations/<l>/keyRings/<r>/cryptoKeys/<k>). The schema refuses an alias/… or arn:… value on a gcp binding, and a projects/… value on an aws binding. In the 0.7.6 schema's description, product means a customer-managed key created for this product, and none means the platform's default encryption.

principals maps the principals the contract names — in accessPolicy.grants[].principal and exposes[].policy.authz.columnRestrictions[].principal — to identities on this binding's platform, so the base contract stays cloud-neutral and each environment binds it. A value is one identity or a list of them; [] says the principal has no identity on this platform. On a gcp binding each identity must be an IAM member (user:, group:, serviceAccount: or domain:), and on an aws binding an IAM principal ARN; the schema enforces both. The schema's description adds that, when the block is present, every principal the contract names for the expose must be mapped; the reference implementation refuses an unmapped one.

governance.lakeFormation.bucketPolicy (cross-account (default) | none | all-grantees) decides which Lake Formation grantees also get a statement in the S3 bucket policy emitted beside the grants. A bucket-policy statement lets its principal read the objects directly, which bypasses Lake Formation's column, row and cell filters; none emits no bucket policy at all.

An emitted bucket policy is authoritative

The 0.7.6 schema's description of bucketPolicy ends: "An emitted aws_s3_bucket_policy is authoritative: it replaces every other statement on the bucket. When no grantee is left none is emitted." Applying a contract that emits a bucket policy therefore replaces every other statement on that bucket.

Whether one is emitted depends on the value. cross-account, the default, gives a statement only to principals "whose ARN names an AWS account other than the one running the apply, decided at plan time against the caller identity"; same-account grantees get none, "since on a registered location Lake Formation vends them credentials". So a contract that never sets bucketPolicy still has an emitted policy replace every other statement on the bucket as soon as one of its grantees is in another AWS account. all-grantees gives every grantee a statement, same-account included, "which is the output before this field existed and reopens the direct S3 read path". none emits no bucket policy, so nothing is replaced.

A statement (s3:ListBucket and s3:GetBucketLocation on the bucket, s3:GetObject under location.path) "lets its principal read the objects straight from S3, which skips Lake Formation's column, row and cell filters and its revocations": Lake Formation's filters do not apply to that read, and revoking the grant in Lake Formation does not close it.

The schema describes none as "the tightest setting when the location is registered (registerLocation) and every reader, cross-account ones included, queries through a Lake Formation integrated engine, or when the bucket policy is managed elsewhere". Like the other 0.7.6 descriptions of what fluid apply emits, this is a non-normative description of the reference implementation (see the tip above).

`exposes[].lifecycle.expire`

lifecycle:
  retention: P400D   # ISO-8601 duration
  expire: true       # default false: delete data older than retention

An expose's lifecycle has the same fields as the contract-root lifecycle, plus expire. With expire: false — the default — retention is a declaration only, so a contract that already declares a retention period does not start deleting data. With expire: true, data older than retention is deleted. How a platform expires data is implementation-defined. The reference implementation honours expire on an AWS S3 binding and on a BigQuery table, and refuses it on other GCP bindings rather than ignoring it.

Typed masking parameters

policy:
  privacy:
    masking:
      - { column: phone, strategy: mask, params: { keepLast: 4 } }                    # ********4567
      - { column: email, strategy: hash, params: { saltEnv: FLUID_PII_HASH_SECRET } }

Param	Strategy	Meaning
`keepFirst`, `keepLast`	`mask`	Leading / trailing characters left unmasked (0–64; defaults 0 and 4).
`saltEnv`	`hash`	Name of the environment variable that holds the salt (default `FLUID_PII_HASH_SECRET`).
`keyEnv`	`tokenize`, `encrypt`	Name of the environment variable that holds the key.

params stays open (additionalProperties: true). The typed keys name secrets by environment variable rather than carrying them; the schema's description adds that the reference implementation's runner refuses any other key, including a literal salt or key. The 0.7.6 description of strategy also gives each strategy's output form — for example, hash is the lowercase hex SHA-256 of salt followed by value — and notes that k_anonymity is accepted by the schema but is a property of a whole table, not a per-value transform.

Column restrictions: semantics written down

exposes[].policy.authz.columnRestrictions exists in 0.7.5 with the one-line description "Column-level access control". The 0.7.6 description defines it:

A column named in any restriction is restricted.
access: deny — the principal may not read the columns. access: allow — the columns are readable only by the principals an allow names.
A deny beats an allow, and a restriction never grants access: the readers are still the expose's readers.
Principals are logical, resolved per platform through binding.principals.

Two complete contracts

Both validate against the 0.7.6 JSON Schema (Draft 2020-12) and with the reference implementation, data-product-forge 0.18.1 (fluid validate prints ✅ Valid FLUID contract (schema v0.7.6)).

BigQuery: column restriction, mapped principals, a product-owned key, expiring partitions, a percentile measure, packaging and a consumer.

fluidVersion: "0.7.6"
kind: DataProduct
id: sales.gold.orders
name: Orders
metadata:
  owner: { team: sales-analytics, email: sales-analytics@example.com }
accessPolicy:
  grants:
    - principal: "group:analysts"
      permissions: [read]
    - principal: "group:stewards"
      permissions: [read]
exposes:
  - exposeId: orders
    kind: table
    contract:
      schema:
        - { name: order_id,       type: STRING,  required: true }
        - { name: order_date,     type: DATE,    required: true }
        - { name: customer_email, type: STRING }
        - { name: amount,         type: NUMERIC }
    policy:
      authz:
        columnRestrictions:
          - principal: "group:analysts"
            columns: [customer_email]
            access: deny
    semantics:
      measures:
        - name: p95_amount
          agg: percentile
          expr: amount
          aggParams: { percentile: 0.95 }
    lifecycle:
      retention: P400D
      expire: true
    binding:
      platform: gcp
      format: bigquery_table
      location: { project: acme-analytics, dataset: sales, table: orders, region: europe-west1 }
      encryption: { kms: product }
      principals:
        "group:analysts": "group:analysts@example.com"
        "group:stewards": "group:data-stewards@example.com"
packaging:
  mode: isolated
consumers:
  - name: weekly_revenue_dashboard
    label: Weekly Revenue Dashboard
    type: dashboard
    maturity: high
    owner: { team: finance-analytics }
    exposeIds: [orders]

S3 + Lake Formation: a cross-mesh upstream pin, masking parameters, an existing key, a shared bucket and no bucket policy.

fluidVersion: "0.7.6"
kind: DataProduct
id: sales.silver.customers
name: Customers
metadata:
  owner: { team: sales-data }
consumes:
  - productId: crm.bronze.accounts
    exposeId: accounts
    upstreamWorkspace: crm-mesh
    upstreamDigest: "sha256:4f1c9a0d6e2b7c8a1f3e5d9b0c2a4e6f8d1b3c5e7a9f0b2d4c6e8a1b3d5f7091"
exposes:
  - exposeId: customers
    kind: table
    contract:
      schema:
        - { name: customer_id, type: STRING, required: true }
        - { name: phone,       type: STRING }
        - { name: email,       type: STRING }
    policy:
      privacy:
        masking:
          - { column: phone, strategy: mask, params: { keepLast: 4 } }
          - { column: email, strategy: hash, params: { saltEnv: FLUID_PII_HASH_SECRET } }
    binding:
      platform: aws
      format: parquet
      location: { bucket: acme-lake, path: sales/customers/, region: eu-west-1, database: sales, table: customers }
      encryption: { kms: "alias/acme-lake" }
      packaging: { mode: shared, pool: lake-pool-eu, containers: { bucket: shared } }
      governance:
        lakeFormation:
          registerLocation: true
          bucketPolicy: none
          grants:
            - principal: "arn:aws:iam::123456789012:role/analyst"
              permissions: [SELECT]

The example sets bucketPolicy: none on purpose. Its bucket is shared, so by the packaging section above it already exists and belongs to the platform, and the product writes into it without owning it; the location is registered (registerLocation: true), so Lake Formation governs the analyst grant. An emitted bucket policy would replace every other statement on that bucket and would give a grantee a path to the objects that skips Lake Formation's filters (see the box above). With the default, cross-account, a grantee in another AWS account would get exactly such a statement. none is the value for a registered location whose readers query through a Lake Formation integrated engine, or whose bucket policy is managed elsewhere.

Remove upstreamDigest from the second contract and it becomes invalid: 'upstreamDigest' is a dependency of 'upstreamWorkspace' (keyword dependentRequired, at /consumes/0).

References

Schema Versions — status of every version, and how to choose fluidVersion.
Changelog: 0.7.5 → 0.7.6.
Reference implementation: the governance parity notes describe what data-product-forge emits and verifies for retention, encryption, column restrictions and principals on AWS and GCP.