What's New in FLUID 0.7.5 — Streaming Kafka → Iceberg Sink & Confluent Tableflow

FLUID 0.7.5 extends the acquisition layer with a streaming Kafka → Iceberg sink and a managed Confluent Cloud Tableflow binding, and generalizes build.engine so dbt adapter-qualified engines (e.g. dbt-databricks) validate. It is the contract side of the streaming-ingestion RFC shipped in the forge CLI 0.9.0 line.

✅ 0.7.5 is fully backward-compatible with 0.7.4. This release is purely additive — every valid 0.7.4 contract validates as 0.7.5 unchanged. See Backward compatibility — fully additive for details.

🧊 Streaming Iceberg sink on kafka-connect acquisition (NEW)

The kafka-connect acquisition pattern gains an opt-in Iceberg streaming sink (RFC §6.2). All fields are optional and off by default — a hand-written sink_connector_config is untouched unless you opt in.

builds:
  - name: ingest_orders
    engine: kafka-connect
    acquisition:
      kafka-connect:
        iceberg_sink_enabled: true        # NEW — opt-in; derive the Iceberg sink connector config
        sink_topics: [orders, payments]   # NEW — explicit topics the sink consumes (else derived from source streams)
        streamingSink:                    # NEW — optional tuning (all keys optional)
          commitIntervalMs: 60000
          dynamicEnabled: true
          routeField: _topic
          autoCreate: true
          evolveSchema: true
        iceberg_catalog_overrides:        # NEW — operator escape hatch, merged LAST over the derived config
          iceberg.catalog.warehouse: s3://lake/warehouse
  • iceberg_sink_enabled — boolean opt-in. Defaults OFF when a hand-written sink_connector_config is present.
  • sink_topics — explicit topic list the derived sink consumes; otherwise derived from the source streams.
  • streamingSink — optional sink tuning (commitIntervalMs, routeField, dynamicEnabled, autoCreate, evolveSchema, …). All keys optional.
  • iceberg_catalog_overrides — a string → string map merged last over the derived Iceberg sink config, for operator escape-hatch keys.

☁️ Confluent Cloud Tableflow binding (NEW, additive)

  • binding.platform gains confluent — the Confluent Cloud Tableflow managed Kafka → Iceberg emitter. Existing platforms (gcp, aws, azure, snowflake, databricks, kafka, local, kubernetes, postgres, other) are unchanged.
  • binding.location gains three Tableflow fields:
    • environment_id — Confluent Cloud environment id (env-xxxxx).
    • kafka_cluster_id — the Kafka cluster id (lkc-xxxxx) the Tableflow topic belongs to.
    • confluent_role_arn — ARN of the pre-created AWS IAM role Confluent Tableflow assumes (BYOB AWS).
binding:
  platform: confluent          # NEW in v0.7.5
  location:
    environment_id: env-abc12
    kafka_cluster_id: lkc-7x8y9
    confluent_role_arn: arn:aws:iam::123456789012:role/tableflow-byob

fluidVersion widened (additive)

fluidVersion widens from enum: ["0.7.3", "0.7.4"] to enum: ["0.7.3", "0.7.4", "0.7.5"], so existing 0.7.3 / 0.7.4 values still validate against the 0.7.5 schema.

Backward compatibility — fully additive

0.7.5 is fully backward-compatible with 0.7.4. Every valid 0.7.4 contract validates as 0.7.5 unchanged — the release only adds optional fields and enum values. Nothing was removed:

  • The streaming-sink keys (iceberg_sink_enabled, sink_topics, streamingSink, iceberg_catalog_overrides) and the Confluent Tableflow binding fields are all optional.
  • All 0.7.4 surfaces — exposes[].mcp, the postgres platform, the Lake Formation governance blocks, the postgres_table / athena_table / glue_table formats — are unchanged.

Upgrading

fluidVersion: "0.7.5"   # optional — "0.7.3" / "0.7.4" still validate against the 0.7.5 schema

Bumping the version is all it takes — there are no removed fields to migrate off. The streaming Iceberg sink, the confluent binding, and the adapter-qualified dbt engines are all optional and additive.

References

  • Schema Changelog — the field-by-field machine diff for every version.
  • Full 0.7.4 → 0.7.5 diff — the complete auto-generated change list (additive: the streaming Iceberg sink on kafka-connect, the confluent Tableflow binding, and fluidVersion widened to include 0.7.5).