FLUID Data Product contract (v0.7.5). Streaming Kafka→Iceberg Sink & Confluent Tableflow Release.
🔥 NEW in v0.7.5:
• Streaming Iceberg sink — The kafka-connect acquisition pattern gains an opt-in Iceberg streaming sink: iceberg_sink_enabled (derive the sink connector config; off by default when a hand-written sink_connector_config is present), streamingSink (commit interval, dynamic routing, auto-create, schema-evolution tuning — RFC §6.2), sink_topics, and iceberg_catalog_overrides (operator escape hatch merged last over the derived config).
• Confluent Cloud Tableflow — binding.platform: confluent plus binding.location Tableflow fields (environment_id, kafka_cluster_id, confluent_role_arn) for the managed Kafka→Iceberg emitter.
• Backward compatible with v0.7.4 — All v0.7.4 contracts remain valid; the streaming-sink keys and the confluent binding are additive and optional.
📋 Inherited from v0.7.4 (Runtime agentPolicy Enforcement at the MCP Gateway):
• MCP output-port gateway — An expose.mcp block declares an expose as agent-consumable over MCP (sampling caps + classification); its presence opts the expose into fluid mcp output-port serve.
• Runtime agentPolicy enforcement — When an expose.mcp block is present, policy.agentPolicy.allowedModels / deniedModels and allowedUseCases / deniedUseCases are enforced at the Fluid MCP gateway on every read.
📋 Inherited from v0.7.3 (Source-Aligned Acquisition):
• Acquisition build pattern — First-class support for source-aligned data products that ingest external systems (Salesforce, Postgres, Kafka, files) into the mesh.
• Six ingestion engines — duckdb (zero-infra files/JDBC), airbyte (350+ SaaS connectors), meltano (600+ Singer taps), dlt (Python-native), kafka-connect (streaming), debezium (CDC).
• Three deployment modes — embedded (in-process), bring-your-own (existing server), managed (Forge provisions via Helm/Compose/OpenTofu).
• Capability-based negotiation — Runners declare {fullrefresh, incrementalappend, incrementaldedup, cdc, streaming, schemadiscovery, dlpscan, exactlyonce}; contract asks validated against declarations.
• Schema evolution semantics — schemaPolicy ∈ {strict, discoverandfreeze, evolvesafe, evolveall} with concrete per-change behavior.
• Delivery guarantees — atmostonce | atleastonce | exactly_once with idempotency keys and DLQ semantics.
• Source supply-chain security — Cosign image signature verification + SLSA provenance required in production.
• Top-level retention block — runState, runLogs, lineage, dlq durations.
• Extended sovereignty.dataResidency — region pinning + prohibitTransferTo for runtime enforcement.
📋 Inherited from v0.7.2 (Semantic Layer):
• Semantic model definitions — Optional semantics block on each expose for entities, measures, dimensions, and metrics.
• Agent-consumable business logic — Eliminates semantic hallucination by providing machine-readable metric definitions.
• Industry-aligned primitives — Compatible with dbt MetricFlow, Snowflake Semantic Views, and OSI-aligned formats.
📋 Inherited from v0.7.1 (Provider-First Orchestration + Agentic Governance):
• Agentic governance — AI/LLM usage policies (agentPolicy) declare which models can consume data and for what purposes (runtime enforcement added in v0.7.4 — see above).
• Sovereignty constraints — Top-level jurisdiction and data residency requirements for compliance enforcement.
• Orchestration philosophy — Pragmatic engine enumeration (v0.7.x) with path to capability-based abstraction (v0.8.0+).
• Provider actions as first-class tasks — Direct invocation of provider actions (aws.s3.ensurebucket, sf.table.ensure) without wrapper operators.
• Unified task type taxonomy — provideraction, fluid_*, bash, python, sensor.
• Data product references in datasets — Link trigger datasets to actual exposes[] with strong typing and auto-URI generation.
• Task output → expose mapping — Formal produces[] declaration for automatic dataset registration and lineage.
• Cross-provider workflow patterns — Native support for multi-cloud pipelines with XCom data passing.
• Cost tracking — Track actual vs estimated costs with budget enforcement.
🎯 Design Philosophy:
• Provider actions = orchestration primitives.
• Contracts drive both infrastructure and workflows.
• Strong typing and validation.
• Natural multi-cloud/multi-provider patterns.
• Minimal abstraction layers.
• Agentic governance: data owners control AI/LLM usage boundaries.
• Sovereignty-first: jurisdiction constraints enforced at contract level.
⚙️ Orchestration Philosophy (v0.7.x):
• Pragmatic approach: enumerate known engines (airflow, dagster, prefect) for immediate adoption.
• Trade-off: implementation-specific vs pure abstraction — favouring real-world deployment patterns.
• Future path: v0.8.0+ will evolve toward a capability-based orchestration model.
• Escape hatch: custom engine allows experimentation without schema changes.
{
"fluidVersion": "0.5.7",
"kind": "DataProduct",
"id": "gold.customer.analytics_360_v1",
"name": "Customer 360 Analytics",
"description": "Unified customer profiles with ML-driven insights",
"domain": "Customer Experience",
"tags": [
"customer-data",
"analytics",
"gold-layer"
],
"labels": {
"team": "customer-analytics",
"criticality": "high",
"cost-center": "engineering"
},
"metadata": {
"layer": "Gold",
"owner": {
"team": "customer-analytics",
"email": "customer-analytics@company.com"
},
"businessContext": {
"domain": "Customer Experience",
"subdomain": "Customer Intelligence"
}
},
"exposes": [
{
"exposeId": "customer_profiles",
"title": "Unified Customer Profiles",
"version": "2.1.0",
"kind": "table",
"tags": [
"pii",
"customer-facing",
"real-time"
],
"labels": {
"sensitivity": "high",
"retention": "7-years",
"region": "us-central1"
},
"contract": {
"schema": [
{
"name": "customer_id",
"type": "STRING",
"required": true,
"description": "Unique customer identifier",
"sensitivity": "cleartext",
"tags": [
"identifier",
"primary-key"
],
"labels": {
"business-name": "Customer ID",
"data-category": "identifier"
}
}
]
},
"binding": {
"platform": "gcp",
"format": "bigquery_table",
"location": {
"project": "company-data",
"dataset": "gold_customer",
"table": "profiles_v1"
}
}
}
]
}
Vendor / plugin-namespaced configuration. Each plugin claims a single sub-key (e.g. customScaffold, customCatalog) and validates its own schema there via the fluidbuild.extensionvalidators entry-point.
Additional Properties of any type are allowed.
Type: objectContract schema version. Accepts '0.7.3' (source-aligned data products + acquisition pattern + ingestion engines) or '0.7.4' (adds the MCP output-port gateway + runtime agentPolicy enforcement). 0.7.4 is fully backward-compatible with 0.7.3 — every 0.7.3 contract validates as 0.7.4 unchanged.
^\d+\.\d+(\.\d+)?$
"0.7.4"
"0.7.3"
Kind of contract. 'MLPipeline' added for basic ML support.
Stable product identifier, e.g. 'gold.hr.employee360v1'.
Must match regular expression:^[A-Za-z0-9_][A-Za-z0-9_.-]*[A-Za-z0-9_]$|^[A-Za-z0-9_]$
Must be at least 1 characters long
Business domain/mesh domain (e.g., 'HR', 'Finance').
Product-level labels for metadata and automation.
{
"team": "customer-analytics",
"criticality": "high",
"cost-center": "engineering",
"retention": "7-years",
"region": "us-central1",
"business-name": "Customer Profiles",
"data-category": "analytical"
}
Each additional property must conform to the following schema
Type: stringLabel values are always strings for consistency and tooling compatibility.
Generation envelope injected by fluid forge / fluid init — records how, when and by which tool/command the contract was generated. Machine-written (not user-authored); additive metadata that does not affect the data product's semantic contract.
Envelope schema version.
Envelope discriminator, e.g. 'ContractMetadata'.
ISO 8601 UTC timestamp of generation.
Tool, version and command that generated the contract.
Data layer label (e.g., Bronze/Silver/Gold). Free-form. When set together with productType, the two MUST be consistent under the canonical mapping (Bronze↔SDP, Silver↔ADP, Gold↔CDP).
NEW in v0.7.3: Data Mesh data-product type. SDP = Source-Aligned Data Product (raw ingestion from external systems; corresponds to Bronze). ADP = Aggregated Data Product (cross-source joined / domain-modelled; corresponds to Silver). CDP = Consumption-Aligned Data Product (analytics- or product-shaped; corresponds to Gold). Either metadata.layer or metadata.productType is sufficient — the validator infers the missing one. When both are present they MUST agree.
NEW in v0.7.3: Data classification label propagated to catalog + access policy enforcement.
NEW in v0.7.3: Feature gates the contract opts into (e.g., 'acquisition' while Bronze acquisition is in beta).
All items must be unique
No Additional ItemsOncall rotation or contact for operational issues.
Business alignment information for data mesh organization.
No Additional PropertiesBusiness domain this product belongs to.
Business subdomain for more granular organization.
Business capability this product supports.
Value stream this product participates in.
Must contain a minimum of 1 items
{
"exposeId": "customer_profiles",
"title": "Customer Profiles Table",
"kind": "table",
"tags": [
"pii",
"customer-facing",
"analytical"
],
"labels": {
"sensitivity": "high",
"retention": "7-years",
"team": "customer-analytics"
},
"contract": {
"schema": [
{
"name": "customer_id",
"type": "STRING",
"required": true,
"sensitivity": "cleartext",
"tags": [
"identifier",
"primary-key"
],
"labels": {
"business-name": "Customer ID",
"data-category": "identifier"
}
}
]
},
"binding": {
"platform": "gcp",
"format": "bigquery_table",
"location": {
"project": "company-data",
"dataset": "customer",
"table": "profiles"
}
}
}
Detailed description of the exposed resource.
Semantic version (semver.org).
Must match regular expression:^(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)(?:-[0-9A-Za-z.-]+)?(?:\+[0-9A-Za-z.-]+)?$
Hash of canonical schema/OpenAPI. Used by contract-tests gates.
Must match regular expression:^sha256:[0-9a-fA-F]{64}$
NEW in v0.7.3: Schema evolution policy. strict = fail on any schema change; discoverandfreeze = capture from source on first run, fail thereafter; evolvesafe = accept additive changes; evolveall = accept all changes (cast on type narrow, route failures to DLQ).
Tabular schema (for table/view/file).
No Additional ItemsMust be at least 1 characters long
Column data type. Accepts a canonical FLUID/SQL type name (case-insensitive) or a parameterized form such as decimal(18,4) or varchar(255). The enum lists the canonical bare types; the pattern additionally permits case variants and parameter suffixes. Intentionally generous — it rejects obvious garbage (e.g. MYSTERY_TYPE) while admitting the cross-dialect superset.
(?i)^\s*(string|text|varchar|varchar2|nvarchar|char|nchar|character|clob|int|integer|int2|int4|int8|int16|int32|int64|tinyint|smallint|mediumint|bigint|long|longint|serial|bigserial|float|float4|float8|float32|float64|double|real|double\s+precision|decimal|dec|numeric|number|bignumeric|money|boolean|bool|bit|date|time|datetime|datetime2|smalldatetime|timestamp|timestamptz|timestamp_tz|timestamp_ntz|timestamp_ltz|timestampntz|timestamp\s+with(out)?\s+time\s+zone|interval|year|variant|object|array|struct|map|record|row|json|jsonb|super|binary|varbinary|bytes|blob|bytea|raw|geography|geometry|geom|point|uuid|uniqueidentifier|guid|enum|hll)\s*(\(\s*[0-9A-Za-z_,\s'\"-]*\s*\))?\s*$
Field-level sensitivity classification for improved policy ergonomics.
Semantic meaning (e.g., 'email', 'phone', 'identifier').
Business-friendly name for this field.
Business definition and meaning of this field.
Field-level validation constraints.
No Additional ItemsOpenAPI document URL (for APIs).
Compatibility promise for non-breaking evolution.
Enhanced data quality rules including anomaly detection.
No Additional PropertiesISO-8601 duration (e.g., P1D, PT15M, P2Y6M).
Must match regular expression:^P(?!$)(\d+Y)?(\d+M)?(\d+W)?(\d+D)?(T(\d+H)?(\d+M)?(\d+S)?)?$
Simplified SQL-expression data quality rules (alternate form of dq.rules for inline assertions).
No Additional ItemsRule identifier.
SQL or boolean expression that must evaluate to true.
Severity when the rule fails.
Availability percentage (e.g., 99.9%).
Must match regular expression:^(100(\.0+)?|\d{2}(\.\d+)?|\d{1}\d(\.\d+)?)%$
e.g., '0 rows'
Target completeness ratio (0.0 to 1.0).
Value must be greater or equal to 0 and lesser or equal to 1
Acceptable error rate (0.0 to 1.0).
Value must be greater or equal to 0 and lesser or equal to 1
Column-level access control.
No Additional ItemsAdditional Properties of any type are allowed.
Type: objectProvider-specific predicate
NEW in v0.7.1: AI/LLM usage governance - controls which AI models can consume this data and for what purposes.
No Additional PropertiesWhitelist of AI models permitted to consume this data. Examples: ['gpt-4', 'claude-3-opus', 'gemini-pro']. Empty array = no AI access.
All items must be unique
No Additional Items^[a-z0-9][a-z0-9-_.]*[a-z0-9]$|^[a-z0-9]$
"gpt-4"
"claude-3-opus"
"claude-3-sonnet"
"gemini-pro"
"gemini-ultra"
"llama-3"
"mixtral-8x7b"
Blacklist of AI models explicitly prohibited from consuming this data. Takes precedence over allowedModels.
All items must be unique
No Additional ItemsMaximum tokens per AI request. Prevents excessive data exposure in single queries.
Value must be greater or equal to 1
4096
8192
16384
Daily token limit for AI consumption. Enforces usage quotas.
Value must be greater or equal to 1
Permitted AI use cases. Controls whether data can be used for training, inference, reasoning, etc.
All items must be unique
No Additional Items[
"inference",
"reasoning",
"summarization"
]
[
"search",
"qa",
"rag"
]
Prohibited AI use cases. Example: Allow reporting but deny model training.
All items must be unique
No Additional Items[
"training",
"fine_tuning"
]
Whether AI agents can use this data for multi-step reasoning/chain-of-thought. False = simple retrieval only.
Whether AI systems can cache/store this data. False = ephemeral processing only.
Data retention requirements for AI systems consuming this data.
Maximum days AI system can retain this data. 0 = no retention allowed.
Value must be greater or equal to 0
Whether AI system must delete data after use.
Whether AI consumption must be logged/audited.
Free-text description of allowed purposes. Example: 'Customer service chatbot only, not for marketing'.
Target platform. NEW in v0.7.5: 'confluent' for the Confluent Cloud Tableflow managed Kafka→Iceberg emitter. NEW in v0.7.4: 'postgres' for the MCP output-port PostgreSQL driver (AWS Athena uses platform 'aws').
Binding format. NEW in v0.7.4: 'postgrestable' (PostgreSQL driver), 'athenatable' / 'glue_table' (AWS Athena driver) for the MCP output port.
Apache Iceberg table format configuration
No Additional PropertiesIceberg table format version
Underlying file format for data files
Partition transformation specifications
No Additional ItemsPartition field name
Source column for partitioning
Partition transform function
Number of buckets (for bucket transform)
Truncate width (for truncate transform)
Sort order specifications for data files
No Additional ItemsColumn to sort by
Sort direction
NULL value ordering
Iceberg table properties
Each additional property must conform to the following schema
Type: stringProvider-native addressing with tightened structure for better interop.
No Additional PropertiesCloud account ID
GCP project ID
BigQuery dataset or database name
Database name
Schema name
Table name
Storage bucket name
File or object path
API gateway endpoint
Base URL for API endpoints
Pub/Sub or Kafka topic name
Pub/Sub subscription name
Cloud region
Cloud zone
NEW in v0.7.5: Confluent Cloud environment id (env-xxxxx) for the Tableflow emitter.
NEW in v0.7.5: Confluent Cloud Kafka cluster id (lkc-xxxxx) the Tableflow topic belongs to.
NEW in v0.7.5: ARN of the pre-created AWS IAM role Confluent Tableflow assumes (byobaws). Its trust policy is updated post-apply with the provider integration's externalid (two-phase IAM).
Provider-specific binding properties (e.g., cluster_by, comment for Snowflake).
Additional Properties of any type are allowed.
Type: objectPer-resource governance: registerLocation, principal grants, tag associations, row/column filters. Account-wide governance (admins, tag definitions) lives at the contract top-level governance block.
No Additional PropertiesPer-resource LF settings: location registration, principal grants, LF-tag associations, row/column filters. Applies only to formats backed by S3 + Glue catalog (iceberg/parquet/csv/json/avro/orc/delta).
No Additional PropertiesWhen true, emit awslakeformationresource that registers the binding's S3 path with Lake Formation. Required before LF can manage access to objects under that path.
Principal-based LF grants on this exposure's database/table. Each entry maps a principal to a permission set (and optional column projection). Emitted as awslakeformationpermissions resources. Field idiom adapted from ODCS v3 roles[] (principal + access-type), bound to IAM ARNs since LF is cloud-specific.
No Additional ItemsIAM principal ARN (role or user) receiving the grant.
Must match regular expression:^arn:aws[a-z0-9-]*:iam::
LF permissions granted. Valid values per the hashicorp/aws awslakeformationpermissions resource.
Must contain a minimum of 1 items
All items must be unique
No Additional ItemsSubset of permissions the principal can re-grant. Maps to awslakeformationpermissions.permissionswithgrant_option.
All items must be unique
No Additional ItemsOptional column-level restriction. When set, the grant is emitted against awslakeformationpermissions.tablewithcolumns instead of .table, allowing the principal to read only the listed columns.
All items must be unique
No Additional ItemsOpposite of columns: every column EXCEPT these is granted. Maps to tablewithcolumns.excludedcolumnnames. Mutually exclusive with columns.
All items must be unique
No Additional ItemsRow-level (and optionally column-level) filter applied to all reads of this table via LF. Emitted as awslakeformationdatacellsfilter. The filter must then be granted to principals via grants[].permissions including SELECT — LF enforces both layers.
No Additional PropertiesFilter name (must be unique per table).
PartiQL-flavoured row predicate. ALL lets every row through (use when only column filtering is intended).
Columns visible through the filter. Mutually exclusive with allColumns / excludedColumnNames.
All items must be unique
No Additional ItemsColumns hidden through the filter (everything else is visible).
All items must be unique
No Additional ItemsConvenience for the LF column-wildcard form (every column visible). When true, columnNames / excludedColumnNames must be omitted.
Unified lifecycle state vocabulary.
Reference to replacement data product or expose.
Human/machine-readable SLI check description
Default SLI trio (freshness, completeness, latency) by kind.
No Additional PropertiesEnable auto-generation of default SLIs.
Default freshness SLI configuration.
No Additional PropertiesFreshness threshold (auto-set by kind: table=PT6H, stream=PT1M, api=PT1S)
Same definition as windowDefault completeness SLI configuration.
No Additional PropertiesCompleteness threshold (auto-set by kind: table=0.95, stream=0.99)
Value must be greater or equal to 0 and lesser or equal to 1
Default latency SLI configuration.
No Additional PropertiesP95 latency threshold (auto-set by kind: api=PT200MS, table=PT5S, stream=PT100MS)
Same definition as windowLatency percentile to measure.
Value must be greater or equal to 0 and lesser or equal to 100
Alert dispatcher config consumed by the acquisition Alerter (DLQ overflow, schema-fingerprint changes, anomaly signals).
No Additional PropertiesFor kind=file: NDJSON sink path (env-vars expanded).
For kind=webhook: Slack-compatible incoming webhook URL. Validated against the SSRF guard at construction time.
NEW in v0.7.2: Semantic model definition mapping physical columns to business concepts. Defines entities, measures, dimensions, and metrics for agent-consumable business logic. Eliminates semantic hallucination by providing machine-readable KPI definitions. Compatible with dbt MetricFlow, Snowflake Semantic Views, and OSI-aligned formats.
No Additional Properties{
"name": "orders_revenue_model",
"description": "Canonical semantic model for revenue analytics",
"defaultAggTimeDimension": "order_date",
"entities": [
{
"name": "order",
"type": "primary",
"expr": "order_id"
},
{
"name": "customer",
"type": "foreign",
"expr": "customer_id"
}
],
"measures": [
{
"name": "order_amount",
"agg": "sum",
"expr": "amount",
"description": "Sum of all order amounts"
},
{
"name": "order_count",
"agg": "count_distinct",
"expr": "order_id",
"createMetric": true
}
],
"dimensions": [
{
"name": "order_date",
"type": "time",
"expr": "completed_at",
"typeParams": {
"timeGranularity": "day"
}
},
{
"name": "region",
"type": "categorical"
}
],
"metrics": [
{
"name": "net_revenue",
"description": "GAAP-compliant net revenue: completed orders minus discounts, excluding refunds",
"type": "derived",
"inputMetrics": [
"gross_revenue",
"total_discounts"
],
"expr": "gross_revenue - total_discounts",
"owner": "finance-data@company.com"
},
{
"name": "gross_revenue",
"description": "Total value of completed orders",
"type": "simple",
"measure": "order_amount",
"filter": "status = 'completed'"
}
]
}
Human-readable name for this semantic model.
Business context explaining what this semantic model represents.
Default time dimension used for measure aggregation. Can be overridden per-measure.
Join keys with type annotations. Entities connect this table to others in the semantic graph.
No Additional ItemsEntity name (e.g., 'order', 'customer').
Entity key type: primary (unique, complete), foreign (references another table), unique (unique, possibly partial), natural (real-world identifier).
Column expression. Defaults to entity name if omitted.
Aggregatable column expressions. Measures are the atomic building blocks from which metrics are composed.
No Additional ItemsMeasure name (e.g., 'totalamount', 'ordercount').
Aggregation function applied to the expression.
SQL expression to aggregate. Defaults to measure name if omitted.
Override the model-level default time dimension for this measure.
Dimension over which this measure cannot be naively aggregated (e.g., account balances).
No Additional PropertiesWhich value to pick when the non-additive dimension is present.
If true, automatically create a simple metric from this measure.
Categorical and time-based grouping axes. Dimensions define how data can be sliced in queries.
No Additional ItemsDimension name (e.g., 'orderdate', 'region', 'productcategory').
Dimension type: categorical (text/enum grouping) or time (temporal axis).
SQL expression. Defaults to dimension name if omitted.
Type-specific parameters.
No Additional PropertiesDefault time granularity for this dimension.
Composable KPI definitions built from measures. Metrics are the named business calculations that agents and tools resolve.
No Additional ItemsMetric name (e.g., 'netrevenue', 'customerchurn_rate').
Business definition of this metric.
Metric type: simple (single measure + filter), derived (expression over other metrics), ratio (numerator/denominator).
Reference to a measure name (for simple metrics).
SQL WHERE clause filter (e.g., "status = 'completed' AND is_refunded = FALSE").
References to other metric names (for derived/ratio metrics).
No Additional ItemsMathematical expression combining input metrics (for derived metrics).
Numerator metric name (for ratio metrics).
Denominator metric name (for ratio metrics).
Business owner of this metric definition.
AWS Glue crawler configuration for auto-discovery of schema from S3 paths.
Glue crawler name.
IAM role ARN for the crawler.
Cron schedule expression (e.g., 'cron(0 6 * * ? *)').
Custom classifier names to apply.
No Additional ItemsPolicy for schema changes detected by the crawler.
AWS Glue Iceberg table management configuration (snapshots, compaction).
Underlying file format for Iceberg data files.
Snapshot retention policy.
Maximum age of snapshots in milliseconds.
Minimum number of snapshots to retain.
Iceberg compaction settings.
Target file size in MB after compaction.
NEW in v0.7.4: Declares this expose as agent-consumable over MCP, with per-expose overrides for the consumer-side output-port server (fluid mcp output-port serve). Its presence opts the expose into the Fluid MCP gateway, where policy.agentPolicy.allowedModels / allowedUseCases are enforced at every read.
Hard cap for the sample MCP tool. Defends against an over-curious agent asking for petabyte-scale row counts. Default: 100.
Value must be greater or equal to 1
Surfaced on the describe tool so consumer agents can declare downstream handling. Advisory; the real gate is policy.agentPolicy.
Upstream data product identifier (e.g., 'silver.hr.people_v2'). Renamed from 'provider' for clarity.
Same definition as idSemver range (e.g., ^1.2, ~1.4, >=1.0.0). Not strictly validated by regex; tooling enforces.
Minimum completeness expectation.
Value must be greater or equal to 0 and lesser or equal to 1
Business purpose for consuming this data.
NEW in v0.5.5: Multiple build configurations for multi-modal data products (e.g., batch + streaming, SQL + ML).
No Additional ItemsEnhanced build configuration supporting v0.4.0 patterns with v0.5.5 improvements.
No Additional PropertiesIf the conditions in the "If" tab are respected, then the conditions in the "Then" tab should be respected. Otherwise, the conditions in the "Else" tab should be respected.
"hybrid-reference"
Hybrid-reference pattern from v0.4.0 (dbt-style).
No Additional Propertiesdbt profile target to run against. Forwarded to dbt as --target <name> by the build runner (allowlisted against the profile's real targets; falls back to the profile default when the requested target is unavailable).
dbt node selector(s) forwarded to dbt --select. Accepts a single selector string or an array of selector strings; the build runner drops flag-like values (those starting with '-').
Legacy alias for select (dbt's older --models flag). Same string-or-array shape; select takes precedence when both are set.
Additional Properties of any type are allowed.
Type: objectEach additional property must conform to the following schema
Type: enum (of string)Key-value labels for structured metadata and automation. Consistent pattern used throughout FLUID objects.
Same definition as labelsIf the conditions in the "If" tab are respected, then the conditions in the "Then" tab should be respected. Otherwise, the conditions in the "Else" tab should be respected.
"embedded-logic"
Embedded logic pattern from v0.4.0.
No Additional PropertiesAdditional Properties of any type are allowed.
Type: objectKey-value labels for structured metadata and automation. Consistent pattern used throughout FLUID objects.
Same definition as labelsIf the conditions in the "If" tab are respected, then the conditions in the "Then" tab should be respected. Otherwise, the conditions in the "Else" tab should be respected.
"multi-stage"
NEW in v0.5.5: Multi-stage orchestration pattern.
No Additional PropertiesUnified identifier pattern: starts/ends with alphanumeric or underscore, allows dots and hyphens in middle. Both casing conventions accepted (snake_case / dotted lowercase / PascalCase / UPPER) so existing catalog ids round-trip unchanged.
Same definition as idKey-value labels for structured metadata and automation. Consistent pattern used throughout FLUID objects.
Same definition as labelsValue must be greater or equal to 1
Key-value labels for structured metadata and automation. Consistent pattern used throughout FLUID objects.
Same definition as labelsIf the conditions in the "If" tab are respected, then the conditions in the "Then" tab should be respected. Otherwise, the conditions in the "Else" tab should be respected.
"acquisition"
NEW in v0.7.3: Source-aligned acquisition pattern. The build ingests data from an external system (no transformation). Engine ∈ {duckdb, airbyte, meltano, dlt, kafka-connect, debezium}.
No Additional PropertiesExternal source system specification.
No Additional PropertiesSource system kind: filesystem, postgres, mysql, sqlite, http, salesforce, stripe, github, kafka, etc. Engine-specific catalog of supported kinds.
Connection details. Use ${VAR} placeholders for env values; use secretRef for credentials. Inline secrets are forbidden by validator.
Allows ${ENV_VAR} placeholders that resolve to an integer at runtime.
Reference to a secret in the configured backend (vault://, aws://, gcp://, azure://, env://).
Must match regular expression:^[a-z][a-z0-9_+.-]*://.+
NEW in v0.7.3: Username/principal for SQL- or REST-style sources. Required by most database drivers (Postgres/MySQL/Oracle/etc.) — without it the underlying client falls back to the OS-user identity and typically fails authentication. Goes alongside secretRef (or an inline password) for the credential pair.
NEW in v0.7.3: Default schema/namespace within the source database to read from. Maps to engine-specific config: dlt sqldatabase schema=, Meltano tap-postgres filterschemas=[], Airbyte source-postgres schemas=[], Debezium schema.include.list, Kafka Connect JDBC schema.pattern. Use schemas (array) to scope to multiple schemas at once.
NEW in v0.7.3: Multi-schema variant of schema. When both are set, schemas wins and schema is treated as a deprecated alias. Engines that don't natively support multi-schema reads will iterate the list.
All items must be unique
No Additional ItemsAdditional Properties of any type are allowed.
Type: objectColumn used for incremental cursor (e.g., updated_at, SystemModstamp).
List of streams/tables/objects to ingest.
No Additional ItemsAdditional Properties of any type are allowed.
Type: objectAdditional Properties of any type are allowed.
Type: objectDestination format for ingested data. binding.platform on the expose remains the source of truth for where; sink.format describes how.
No Additional PropertiesDelivery semantics + idempotency + DLQ configuration.
No Additional PropertiesTemplate for the idempotency key. Default: {runid}:{stream}:{recordpk}
Value must be greater or equal to 0
Schema evolution behavior. Stricter-wins between policy and per-change overrides.
No Additional PropertiesPre-land hook chain - runs on each batch before destination write.
No Additional ItemsPre-land quality gates + anomaly signals.
No Additional PropertiesAdditional Properties of any type are allowed.
Type: objectCost tracking + budget enforcement.
No Additional PropertiesValue must be greater or equal to 0
Human-readable size with unit suffix (e.g., '50GB', '500MB').
Value must be greater or equal to 0
Additional Properties of any type are allowed.
Type: objectCatalog auto-registration on first apply.
No Additional PropertiesSingle-flight concurrency control.
No Additional PropertiesAdditional Properties of any type are allowed.
Type: objectConnector image supply-chain verification.
No Additional PropertiesEngine deployment mode and target.
No Additional PropertiesURL for bring-your-own mode.
Additional Properties of any type are allowed.
Type: objectAdditional Properties of any type are allowed.
Type: objectAdditional Properties of any type are allowed.
Type: objectAdditional Properties of any type are allowed.
Type: objectAdditional Properties of any type are allowed.
Type: objectAdditional Properties of any type are allowed.
Type: objectValue must be greater or equal to 1
Confluent-compatible Schema Registry config. When url is set, the runner injects key.converter/value.converter for Avro into the emitted connector config.
Schema Registry HTTP endpoint, e.g. http://schema-registry:8081
Additional Properties of any type are allowed.
Type: objectOpt-in: derive the Iceberg sink connector config. Defaults OFF when a hand-written sinkconnectorconfig is present (RFC §6.2).
Explicit topics the derived Iceberg sink consumes (else derived from source streams).
No Additional ItemsOperator escape hatch: connector config keys merged LAST over the derived Iceberg sink config.
Each additional property must conform to the following schema
Type: stringOptional Iceberg streaming-sink tuning (RFC §6.2). All keys optional.
No Additional PropertiesValue must be greater or equal to 1
Additional Properties of any type are allowed.
Type: objectAdditional Properties of any type are allowed.
Type: objectBuild identifier for multi-build scenarios (e.g., 'batch-processing', 'ml-training').
Same definition as idHuman-readable description of this build configuration.
Build pattern: hybrid-reference (dbt-style), embedded-logic (raw SQL/code), multi-stage (complex pipelines), or acquisition (source-aligned ingestion from external systems).
Build/ingestion engine. Transformation engines: dbt, sql, python, spark, glue, custom. Adapter-qualified dbt engines (dbt-<adapter>, e.g. dbt-glue, dbt-snowflake, dbt-bigquery) are accepted too and route through the dbt build runner, which derives the adapter from the suffix. Ingestion engines (acquisition pattern only): duckdb (zero-infra files/JDBC), airbyte (350+ SaaS), meltano (Singer 600+ taps), dlt (Python-native), kafka-connect (streaming), debezium (CDC). AWS Glue: glue (Spark ETL jobs managed by Glue).
^dbt-[a-z0-9]+([_-][a-z0-9]+)*$
Capabilities the build asks for (acquisition pattern). Validation enforces capabilities ⊆ runner.declared_capabilities.
All items must be unique
No Additional ItemsPattern-specific build properties with conditional validation.
Build execution configuration (triggers, runtime, retries).
No Additional PropertiesNEW in v0.7.0: Can reference data products directly via exposeRef for strong typing and auto-URI generation. Backward compatible with raw URIs.
No Additional ItemsRaw dataset URI (legacy, e.g., 'dataset://gcp/project/dataset/table')
NEW in v0.7.0: Data product reference with strong typing
Cron expression for schedule trigger
Custom timetable configuration
Each additional property must conform to the following schema
Type: stringContainer image for execution
Service account for cloud execution
Value must be greater or equal to 1
^[a-z0-9_][a-z0-9_.-]*[a-z0-9_]$|^[a-z0-9_]$
Value must be greater or equal to 1
Default configuration for all tasks (can be overridden per task)
Value must be greater or equal to 0
NEW in v0.7.0: Enhanced error categorization and handling
Error types that should trigger retries
No Additional Items[
"TemporaryFailure",
"RateLimitExceeded",
"ConnectionTimeout",
"DeadlineExceeded"
]
Error types that should fail immediately
No Additional Items[
"ValidationError",
"SchemaViolation",
"PermissionDenied",
"InvalidCredentials"
]
Must contain a minimum of 1 items
Backward compatibility: If 'operator' is specified but not 'type', map operator to type
If the conditions in the "If" tab are respected, then the conditions in the "Then" tab should be respected. Otherwise, the conditions in the "Else" tab should be respected.
Operator field is deprecated but still works. Maps: FluidExecuteOperator→fluid_execute, BashOperator→bash, etc.
NEW in v0.7.0: Validate provider_action tasks have required fields
If the conditions in the "If" tab are respected, then the conditions in the "Then" tab should be respected. Otherwise, the conditions in the "Else" tab should be respected.
"provider_action"
Validate fluid_execute tasks
If the conditions in the "If" tab are respected, then the conditions in the "Then" tab should be respected. Otherwise, the conditions in the "Else" tab should be respected.
"fluid_execute"
FluidExecute
Validate bash tasks
If the conditions in the "If" tab are respected, then the conditions in the "Then" tab should be respected. Otherwise, the conditions in the "Else" tab should be respected.
"bash"
Bash
Validate python tasks
If the conditions in the "If" tab are respected, then the conditions in the "Then" tab should be respected. Otherwise, the conditions in the "Else" tab should be respected.
"python"
Python
^[a-z0-9_][a-z0-9_.-]*[a-z0-9_]$|^[a-z0-9_]$
NEW in v0.7.0: Simplified task type taxonomy. Replaces 'operator' field (backward compatible). Types: provideraction (NEW!), fluidvalidate, fluidplan, fluidapply, fluidexecute, fluidverify, fluiddqcheck, bash, python, sensor, or provider-specific operators.
DEPRECATED in v0.7.0: Use 'type' instead. Maintained for backward compatibility with v0.6.1 contracts. Maps to 'type' automatically (FluidExecuteOperator → fluid_execute, BashOperator → bash).
NEW in v0.7.0: Provider name when type=provider_action. Examples: aws, gcp, azure, snowflake, databricks.
NEW in v0.7.0: Provider action when type=provideraction. Format: service.operation (e.g., s3.ensurebucket, table.ensure, kinesis.create_stream).
Must match regular expression:^[a-z0-9_]+\.[a-z0-9_]+$
"s3.ensure_bucket"
"table.ensure"
"database.ensure"
"kinesis.create_stream"
"glue.create_job"
"bigquery.create_table"
NEW in v0.7.0: Reference to transformation step in builds[].transformations[]. Links orchestration tasks to build steps for clearer mapping.
Additional Properties of any type are allowed.
Type: objectNEW in v0.7.0: Declares what exposes[] this task produces. Enables automatic dataset registration and lineage tracking.
No Additional ItemsOptional: Airflow dataset URI. Auto-generated from binding.location if not specified.
How this task updates the data
NEW in v0.7.0: Tasks can declare dependencies on other data products with version constraints and freshness requirements.
No Additional ItemsTask-specific overrides for taskDefaults
XCom configuration for inter-task communication (from v0.6.1)
Task-level access control (from v0.6.1)
Cost tracking (enhanced in v0.7.0)
NEW in v0.7.0: Track actual costs vs estimate
NEW in v0.7.0: Alert if actual > threshold * estimate (e.g., 1.5 = 150%)
NEW in v0.7.0: Budget enforcement
Value must be greater or equal to 1
Value must be greater or equal to 1
List of exposeIds produced by this build.
No Additional ItemsUnified identifier pattern: starts/ends with alphanumeric or underscore, allows dots and hyphens in middle. Both casing conventions accepted (snake_case / dotted lowercase / PascalCase / UPPER) so existing catalog ids round-trip unchanged.
Same definition as idDependencies on other builds (for multi-build orchestration).
No Additional ItemsUnified identifier pattern: starts/ends with alphanumeric or underscore, allows dots and hyphens in middle. Both casing conventions accepted (snake_case / dotted lowercase / PascalCase / UPPER) so existing catalog ids round-trip unchanged.
Same definition as idLegacy transformations array for backward compatibility.
No Additional ItemsPath/script identifier (dbt model, sql file, etc.)
List of exposeIds materialized by this step. VALIDATION: Tools must validate that every output exists in exposes[].exposeId.
No Additional ItemsUnified identifier pattern: starts/ends with alphanumeric or underscore, allows dots and hyphens in middle. Both casing conventions accepted (snake_case / dotted lowercase / PascalCase / UPPER) so existing catalog ids round-trip unchanged.
Same definition as idKey-value labels for structured metadata and automation. Consistent pattern used throughout FLUID objects.
Same definition as labelsSingle build configuration (legacy). Use 'builds' array for multi-modal.
Same definition as builds_itemsOptional lineage tracking for better data governance.
No Additional PropertiesLevel of lineage tracking granularity.
Upstream dependencies and their field mappings.
No Additional ItemsField-level lineage mappings (optional).
No Additional ItemsTransformation logic or SQL expression.
Type of data relationship.
Known downstream consumers (optional).
No Additional ItemsImpact level if this data product changes.
Optional schema evolution strategy for managing changes.
No Additional PropertiesVersioning strategy for schema changes.
Compatibility approach for schema evolution.
Advance notice period for breaking changes.
Value must be greater or equal to 0
Whether schema changes require approval.
List of approvers for schema changes.
No Additional ItemsOptional ML model specifications for MLPipeline kind.
No Additional PropertiesList of feature names used by this model.
No Additional ItemsTarget variable for supervised learning.
Validation metrics (e.g., accuracy, auc_roc).
No Additional ItemsMinimum acceptable metric values.
Each additional property must conform to the following schema
Type: numberOptional environment-specific overrides (dev, staging, prod).
Each additional property must conform to the following schema
Type: objectEnvironment-specific metadata overrides.
No Additional PropertiesEnvironment-specific expose overrides.
No Additional ItemsUnified identifier pattern: starts/ends with alphanumeric or underscore, allows dots and hyphens in middle. Both casing conventions accepted (snake_case / dotted lowercase / PascalCase / UPPER) so existing catalog ids round-trip unchanged.
Same definition as idNEW in v0.7.1: Jurisdiction and data residency requirements for compliance enforcement.
No Additional Properties{
"jurisdiction": "EU",
"allowedRegions": [
"eu-west-1",
"eu-central-1"
],
"dataResidency": true,
"crossBorderTransfer": false,
"regulatoryFramework": [
"GDPR"
],
"enforcementMode": "strict"
}
{
"jurisdiction": "US",
"allowedRegions": [
"us-east-1",
"us-west-2"
],
"dataResidency": true,
"regulatoryFramework": [
"CCPA",
"HIPAA"
],
"enforcementMode": "strict"
}
Required legal jurisdiction for data storage and processing. Used to validate binding.location matches sovereignty intent.
"EU"
"US"
"Global"
Explicit list of allowed cloud regions. Example: ['eu-west-1', 'eu-central-1'] for EU-only.
All items must be unique
No Additional Items[
"eu-west-1",
"eu-central-1"
]
[
"us-east-1",
"us-west-2"
]
Explicit list of prohibited cloud regions. Takes precedence over allowedRegions.
All items must be unique
No Additional ItemsWhether data must remain within jurisdiction boundaries at rest and in transit. True = strict residency.
Whether cross-border data transfer is permitted. False = data never leaves jurisdiction.
Allowed legal mechanisms for cross-border transfer (if permitted). Example: ['SCCs', 'BCRs', 'Adequacy'].
All items must be unique
No Additional ItemsApplicable regulatory frameworks. Example: ['GDPR', 'CCPA', 'HIPAA'].
All items must be unique
No Additional Items[
"GDPR"
]
[
"GDPR",
"CCPA"
]
[
"HIPAA"
]
How sovereignty violations are handled. strict = block deployment, advisory = warn, audit = log only.
Whether binding.location must be validated against sovereignty rules before deployment.
Root-level access policy for automated IAM binding generation. Defines principals, permissions, and resources for data product access control.
No Additional Properties{
"grants": [
{
"principal": "group:data-analytics@company.com",
"permissions": [
"read",
"select",
"query"
]
},
{
"principal": "serviceAccount:pipeline@project.iam.gserviceaccount.com",
"permissions": [
"write",
"insert",
"update",
"delete"
]
}
]
}
List of access grants to principals (users, groups, service accounts).
No Additional ItemsIdentity receiving access. Format: 'user:email@domain.com', 'group:name@domain.com', 'serviceAccount:name@project.iam.gserviceaccount.com'
"group:data-analytics@company.com"
"serviceAccount:pipeline@project.iam.gserviceaccount.com"
"user:analyst@company.com"
Permissions to grant. Will be mapped to provider-specific roles (e.g., BigQuery roles).
All items must be unique
No Additional Items[
"read",
"select"
]
[
"write",
"insert",
"update"
]
Resources this grant applies to. JSONPath expressions referencing exposes[] items. If omitted, applies to all exposed resources.
No Additional Items"$.exposes[?(@.exposeId=='customer_table')]"
"$.exposes[?(@.kind=='table')]"
Optional conditions for conditional access (e.g., IP restrictions, time windows).
Additional Properties of any type are allowed.
Type: objectTop-level orchestration configuration for scheduling and workflow engines.
NEW in v0.7.3: Retention durations for run state, run logs, lineage events, and DLQ records. Honored by the retention sweeper invoked from policy-apply.
No Additional PropertiesContract-level governance: account/project-wide settings (e.g. AWS Lake Formation admins and LF-tag definitions). Per-resource grants and tag associations live under binding.governance.
No Additional PropertiesAWS Lake Formation account-wide settings: data-lake admins and LF-tag (TBAC) definitions. Applies to the AWS account the contract's IaC apply targets.
No Additional PropertiesIAM principal ARNs (roles or users) granted Lake Formation admin privileges. Emitted as awslakeformationdatalakesettings.admins. Idempotent: re-applying with the same set is a no-op; principals not listed are NOT removed (LF resource semantics preserve other admins).
All items must be unique
No Additional Items^arn:aws[a-z0-9-]*:iam::
LF-TBAC tag-key definitions. Each entry is a tag key mapped to its allowed string values. Emitted as one awslakeformationlf_tag per key; values are the tag's allowed-values list. Field idiom borrowed from OpenMetadata's classification-label model — see ODCS roles + OpenMetadata data contracts spec.
Each additional property must conform to the following schema
Type: array of stringMust contain a minimum of 1 items
All items must be unique
No Additional Items