What's New in FLUID 0.7.5 — Streaming Kafka → Iceberg Sink & Confluent Tableflow

FLUID 0.7.5 is the latest stable version. It adds an opt-in Iceberg streaming sink to the kafka-connect acquisition engine, two binding platforms — confluent (Confluent Cloud Tableflow) and pgvector (a vector / embeddings output port) — and location fields for Iceberg catalogs, Redshift Serverless and Kinesis.

✅ Additive over 0.7.4. scripts/check-compat.py --from 0.7.4 --to 0.7.5 finds no narrowing change: every valid 0.7.4 contract stays valid when it declares "0.7.5". To use a 0.7.5 field, the contract must declare fluidVersion: "0.7.5" — see Upgrading.

• Schema: fluid-schema-0.7.5.json · Reference: specs/0.7.5/fluid-spec.html · Diff: diff-0.7.4-to-0.7.5.md

Streaming Iceberg sink on kafka-connect acquisition

The kafka-connect engine's settings gain an opt-in Iceberg streaming sink. They live where every acquisition engine's settings live: under builds[].properties, which the schema validates as an acquisition pattern when pattern: acquisition, in a key named after the engine.

fluidVersion: "0.7.5"
kind: DataProduct
id: sales.bronze.orders_stream
name: Orders (streamed)
metadata:
  owner: { team: sales-data }
builds:
  - id: ingest_orders
    pattern: acquisition
    engine: kafka-connect
    capabilities: [streaming]
    outputs: [orders]
    properties:
      source:                                 # required for pattern: acquisition
        kind: kafka
        mode: streaming
        connection: { secretRef: "env://KAFKA_BOOTSTRAP" }
        streams: [orders, payments]
      kafka-connect:
        iceberg_sink_enabled: true            # NEW — opt in to a derived Iceberg sink connector
        sink_topics: [orders, payments]       # NEW — topics the sink consumes (else derived from source.streams)
        streamingSink:                        # NEW — optional tuning, every key optional
          commitIntervalMs: 60000
          dynamicEnabled: true
          routeField: _topic
          autoCreate: true
          evolveSchema: true
        iceberg_catalog_overrides:            # NEW — string → string, merged last over the derived config
          iceberg.catalog.warehouse: "s3://acme-lake/warehouse"
exposes:
  - exposeId: orders
    kind: table
    contract:
      schema:
        - { name: order_id, type: STRING, required: true }
        - { name: amount,   type: NUMERIC }
    binding:
      platform: aws
      format: iceberg
      location:
        bucket: acme-lake
        database: sales
        table: orders
        region: eu-west-1
        catalog: glue                         # NEW — Iceberg catalog kind
        warehouse: "s3://acme-lake/warehouse" # NEW — Iceberg warehouse

• iceberg_sink_enabled — boolean opt-in. The schema description adds that it defaults to off when a hand-written sink_connector_config is present.
• sink_topics — the topics the derived sink consumes; when absent, they are derived from the source streams.
• streamingSink — a closed object of optional tuning keys: commitIntervalMs, routeField, dynamicEnabled, autoCreate, evolveSchema, upsertMode, controlTopic.
• iceberg_catalog_overrides — a string → string map merged last over the derived sink configuration.

confluent: Confluent Cloud Tableflow binding

binding.platform gains confluent, and binding.location gains the fields Tableflow needs:

binding:
  platform: confluent                 # NEW in 0.7.5
  format: iceberg
  location:
    environment_id: env-abc12         # NEW
    kafka_cluster_id: lkc-7x8y9       # NEW
    confluent_role_arn: "arn:aws:iam::123456789012:role/tableflow-byob"   # NEW
    bucket: acme-tableflow
    database: sales
    topic: orders
    region: eu-west-1

The schema only adds the fields. The reference implementation also requires, for a confluent binding, format: iceberg and all of environment_id, kafka_cluster_id, bucket and confluent_role_arn; fluid validate reports each one that is missing.

pgvector: vector / embeddings output port

binding.platform gains pgvector, binding.format gains pgvector_table, and a binding can carry vectorConfig:

kind: vector
contract:
  schema:
    - { name: ticket_id, type: STRING, required: true }
    - { name: body,      type: TEXT, labels: { ai-embeddable: "true" } }
binding:
  platform: pgvector                  # NEW in 0.7.5
  format: pgvector_table              # NEW in 0.7.5
  location: { database: support, schema: public, table: tickets }
  vectorConfig:                       # NEW in 0.7.5
    dimensions: 1536                  # required — 1 to 16000
    embeddingModel: text-embedding-3-small
    vectorType: vector                # vector | halfvec
    distanceMetric: cosine            # cosine | l2 | inner_product | l1
    indexType: hnsw                   # hnsw | ivfflat | none
    hnsw: { m: 16, efConstruction: 64 }
    sourceKeyColumn: ticket_id

vectorConfig is a closed object; only dimensions is required. hnsw (m, efConstruction) and ivfflat (lists) carry index tuning, and table names the embeddings table. Which columns get embedded is not part of the schema: the reference implementation embeds the columns labelled ai-embeddable: "true", and fluid validate warns when there are none.

New binding.location fields

binding:
  platform: aws
  format: redshift_external_schema
  location:
    region: eu-west-1
    namespace: analytics             # NEW
    workgroup: analytics-wg          # NEW
    iam_role_arn: "arn:aws:iam::123456789012:role/redshift-spectrum"   # NEW
    external_schema: sales_ext       # NEW
    glue_database: sales             # NEW
    table: orders

fluidVersion

The fluidVersion enum is ["0.7.3", "0.7.4", "0.7.5"]. That lets the 0.7.5 schema accept documents that declare an older version; it does not let them use 0.7.5 fields, because a document is validated against the schema of the version it declares.

Upgrading

fluidVersion: "0.7.5"

• To use any field above, declare "0.7.5". A contract that declares "0.7.4" and uses, say, platform: confluent is invalid: its schema is 0.7.4, which does not have that value. The reference implementation reports exactly that against schema v0.7.4.
• If you use no new field, bumping is optional — the contract stays valid either way.
• Point your editor's # yaml-language-server: $schema=… line at fluid-schema-0.7.5.json when the file declares 0.7.5, and at the matching older schema when it does not. See Choosing fluidVersion.

Nothing was removed, so there is nothing to migrate off.

Corrections to this page

• An earlier version of this page said that 0.7.5 widened build.engine to accept adapter-qualified dbt engines such as dbt-databricks. That pattern (^dbt-[a-z0-9]+([_-][a-z0-9]+)*$) has been in the schema since 0.7.2; build.engine is identical in 0.7.4 and 0.7.5.
• Its headline example put the sink under builds[].acquisition with a name key. A build has neither key; the example above shows where the settings go, inside a complete contract that validates.
• The 0.7.5 schema first published on this site was a snapshot taken while 0.7.5 was still a preview in the reference implementation. The stable 0.7.5 also has the pgvector platform, pgvector_table, vectorConfig and the Iceberg-catalog, Redshift Serverless and Kinesis location fields; the published file has been re-synchronised with it and this page now describes it. See the Changelog.

References

• Schema Changelog — the history of every version.
• Full 0.7.4 → 0.7.5 diff — the auto-generated change list.
• What comes next: 0.7.6 (preview).