What's New in FLUID 0.7.3 — Source-Aligned Acquisition

FLUID 0.7.3 is the Source-Aligned Acquisition release. It is additive over 0.7.2 — a valid 0.7.2 contract stays valid when it declares "0.7.3" — and makes ingestion from external systems a first-class FLUID concept rather than something you bolt on with Airbyte/Meltano configs outside the contract.

🛬 acquisition build pattern (NEW)

The build block gains a fourth pattern, acquisition, designed for source-aligned data products: contracts that ingest an external system (Salesforce, Postgres, Kafka, files, …) into your mesh, with no transformation logic.

build:
  pattern: acquisition            # tells the schema that build.properties below is an acquisitionPattern
  engine: airbyte                 # see "Six engines" below
  capabilities: [incremental_dedup, schema_evolution, dlp_scan]
  properties:                     # ← pattern-specific block; acquisitionPattern shape
    source:
      kind: postgres              # filesystem | postgres | mysql | http | salesforce | stripe | …
      mode: incremental_dedup     # 6 modes: full_refresh | incremental_* (3) | cdc | streaming
      cursor_field: updated_at
      connection:
        secretRef: "vault://pg-prod-readonly"   # a URI (<scheme>://…), e.g. vault://, aws://, gcp://, azure://, env://
      streams: [public.customers, public.orders]
    sink:
      format: iceberg             # iceberg | delta | parquet | snowflake_table | bigquery_table | …
      partitionBy: ["day(ingested_at)"]    # string array (function-form)
    delivery:
      guarantee: at_least_once    # at_most_once | at_least_once | exactly_once
      idempotencyKey: "${stream}|${batch_id}"
      dlq:
        enabled: true
        sink: { format: parquet, location: "s3://acme-dlq/customers/" }
        maxRecordsBeforeAbort: 10000
        alertOn: [pii_classification_failed, schema_violation, quality_gate_failed]
    schemaEvolution:
      policy: evolve_safe         # strict | discover_and_freeze | evolve_safe | evolve_all
      onAddedColumn: include      # include | warn | fail
      onRemovedColumn: warn       # drop | warn | fail
      onTypeChange: fail          # cast | warn | fail
    preLand: [dlp_scan, tokenize_pii, quality_gate, emit_lineage_input]
    airbyte:
      connector_image: airbyte/source-postgres
      version: "3.4.10"
      image_signature:            # ⭐ supply-chain security
        verifier: cosign          # cosign is the only verifier today
        publicKey: "k8s://acme/cosign-pub"
        slsaProvenance: required  # required | optional | disabled
      deployment: { mode: managed }    # embedded | bring-your-own | managed

Why this matters: ingestion is where most data-quality, compliance, and lineage debt is born. Previously a .fluid.yml could describe how data was transformed but stayed silent on how it arrived. 0.7.3 closes that gap with the same contract-as-code discipline applied to the source boundary.

🔌 Six ingestion engines

🏗️ Three deployment modes

• embedded — runs in-process with the FLUID runner (no extra infra)
• bring-your-own — points at an existing Airbyte/Meltano/Kafka server you already operate
• managed — Forge provisions runtime via Helm / Docker Compose / OpenTofu

🤝 Capability-based negotiation

Runners publish what they can do (full_refresh, incremental_dedup, cdc, schema_discovery, dlp_scan, exactly_once, …). The contract publishes what it asks for via build.capabilities. The orchestrator validates ask ⊆ declarations before scheduling — incompatible source/runner pairs fail fast at contract apply, not at 3 AM in production.

📐 Schema evolution as a first-class policy

schemaEvolution.policy has four well-defined values:

🚚 Delivery guarantees + DLQ

delivery.guarantee makes the contract honest about what the runner can promise: at_most_once, at_least_once, or exactly_once. Pair it with idempotencyKey (a template like "${stream}|${batch_id}") and a dlq destination, and re-runs become safe by construction.

🔐 Source supply-chain security

For production-grade ingestion, the connector image you run is part of the data supply chain:

airbyte:
  image_signature:
    verifier: cosign                              # signature verifier (cosign only today)
    publicKey: "k8s://acme/cosign-pub"            # public-key reference
    slsaProvenance: required                      # required | optional | disabled

🧹 Top-level retention block

retention:
  runState: P30D     # ISO-8601 durations
  runLogs:  P90D
  lineage:  P365D
  dlq:      P180D

A single sweeper job honors these — no more per-tool TTL knobs scattered across the stack.

🧩 Also new in 0.7.3

The sections above cover acquisition. The same release added these, all optional:

🔄 Backward compatibility with 0.7.2

• ✅ No narrowing change: scripts/check-compat.py --from 0.7.2 --to 0.7.3 reports none. (It flags the identifier pattern change for review, because pattern containment cannot be decided in general; the change only widens the pattern, and the waiver in scripts/compat-waivers.txt records the exhaustive check that showed it.)
• ✅ Every addition in this release is optional.

Migration:

fluidVersion: "0.7.3"  # was "0.7.2"

The 0.7.3 schema accepts only "0.7.3" as fluidVersion, so this line is the one change a valid 0.7.2 contract needs.

See the Schema Changelog for the full version history, or the 0.7.2 → 0.7.3 diff for the auto-generated change list.