What's New in FLUID 0.7.3 — Source-Aligned Acquisition

FLUID 0.7.3 is the Source-Aligned Acquisition release. It is additive over 0.7.2 — every existing contract remains valid — and finally makes ingestion from external systems a first-class FLUID concept rather than something you bolt on with Airbyte/Meltano configs outside the contract.

🛬 `acquisition` build pattern (NEW)

The build block gains a fourth pattern, acquisition, designed for source-aligned data products: contracts that ingest an external system (Salesforce, Postgres, Kafka, files, …) into your mesh, with no transformation logic.

build:
  pattern: acquisition            # tells the schema that build.properties below is an acquisitionPattern
  engine: airbyte                 # see "Six engines" below
  capabilities: [incremental_dedup, schema_evolution, dlp_scan]
  properties:                     # ← pattern-specific block; acquisitionPattern shape
    source:
      kind: postgres              # filesystem | postgres | mysql | http | salesforce | stripe | …
      mode: incremental_dedup     # 6 modes: full_refresh | incremental_* (3) | cdc | streaming
      cursor_field: updated_at
      connection:
        secretRef: "vault://pg-prod-readonly"   # must be a URI: vault:// aws:// gcp:// azure:// env://
      streams: [public.customers, public.orders]
    sink:
      format: iceberg             # iceberg | delta | parquet | snowflake_table | bigquery_table | …
      partitionBy: ["day(ingested_at)"]    # string array (function-form)
    delivery:
      guarantee: at_least_once    # at_most_once | at_least_once | exactly_once
      idempotencyKey: "${stream}|${batch_id}"
      dlq:
        enabled: true
        sink: { format: parquet, location: "s3://acme-dlq/customers/" }
        maxRecordsBeforeAbort: 10000
        alertOn: [pii_classification_failed, schema_violation, quality_gate_failed]
    schemaEvolution:
      policy: evolve_safe         # strict | discover_and_freeze | evolve_safe | evolve_all
      onAddedColumn: include      # include | warn | fail
      onRemovedColumn: warn       # drop | warn | fail
      onTypeChange: fail          # cast | warn | fail
    preLand: [dlp_scan, tokenize_pii, quality_gate, emit_lineage_input]
    airbyte:
      connector_image: airbyte/source-postgres
      version: "3.4.10"
      image_signature:            # ⭐ supply-chain security
        verifier: cosign          # cosign is the only verifier today
        publicKey: "k8s://acme/cosign-pub"
        slsaProvenance: required  # required | optional | disabled
      deployment: { mode: managed }    # embedded | bring-your-own | managed

Why this matters: ingestion is where most data-quality, compliance, and lineage debt is born. Previously a .fluid.yml could describe how data was transformed but stayed silent on how it arrived. 0.7.3 closes that gap with the same contract-as-code discipline applied to the source boundary.

🔌 Six ingestion engines

Engine	Best for	Footprint
duckdb	files (Parquet/CSV/JSON), JDBC, zero infra	in-process
airbyte	350+ SaaS connectors	container
meltano	600+ Singer taps, Python ecosystem	container
dlt	Python-native, code-defined pipelines	library
kafka-connect	streaming, Confluent ecosystem	cluster
debezium	change data capture (CDC)	cluster

🏗️ Three deployment modes

embedded — runs in-process with the FLUID runner (no extra infra)
bring-your-own — points at an existing Airbyte/Meltano/Kafka server you already operate
managed — Forge provisions runtime via Helm / Docker Compose / OpenTofu

🤝 Capability-based negotiation

Runners publish what they can do (full_refresh, incremental_dedup, cdc, schema_discovery, dlp_scan, exactly_once, …). The contract publishes what it asks for via build.capabilities. The orchestrator validates ask ⊆ declarations before scheduling — incompatible source/runner pairs fail fast at contract apply, not at 3 AM in production.

📐 Schema evolution as a first-class policy

schemaEvolution.policy has four well-defined values:

Policy	Behavior
`strict`	any schema change at source aborts the run
`discover_and_freeze`	discover schema on first run, lock it thereafter
`evolve_safe`	additive changes propagate; type changes / removals require approval
`evolve_all`	propagate everything (with full lineage)

🚚 Delivery guarantees + DLQ

delivery.guarantee makes the contract honest about what the runner can promise: at_most_once, at_least_once, or exactly_once. Pair it with idempotencyKey (a template like "${stream}|${batch_id}") and a dlq destination, and re-runs become safe by construction.

🔐 Source supply-chain security

For production-grade ingestion, the connector image you run is part of the data supply chain:

airbyte:
  image_signature:
    verifier: cosign                              # signature verifier (cosign only today)
    publicKey: "k8s://acme/cosign-pub"            # public-key reference
    slsaProvenance: required                      # required | optional | disabled

🧹 Top-level `retention` block

retention:
  runState: P30D     # ISO-8601 durations
  runLogs:  P90D
  lineage:  P365D
  dlq:      P180D

A single sweeper job honors these — no more per-tool TTL knobs scattered across the stack.

🔄 100% backward compatible with 0.7.2

✅ No breaking changes
✅ acquisition, retention, build.capabilities, and the six new engines are all opt-in
✅ All 0.7.2 semantics + 0.7.1 agentPolicy/sovereignty/accessPolicy fully preserved

Migration:

fluidVersion: "0.7.3"  # was "0.7.2" — that's it

See the Schema Changelog for the full version history, or the 0.7.2 → 0.7.3 diff for the auto-generated change list.