FLUID Data Product contract (v0.7.1). Provider-First Orchestration + Agentic Governance Release:
🔥 NEW in v0.7.1:
• Agentic governance - AI/LLM usage policies (agentPolicy) control which models can consume data and for what purposes
• Sovereignty constraints - Top-level jurisdiction and data residency requirements for compliance enforcement
• Orchestration philosophy - Pragmatic engine enumeration (v0.7.x) with path to capability-based abstraction (v0.8.0+)
🔥 NEW in v0.7.0:
• Provider actions as first-class tasks - Direct invocation of provider actions (aws.s3.ensurebucket, sf.table.ensure) without wrapper operators
• Unified task type taxonomy - Simplified type system: provideraction, fluid_*, bash, python, sensor
• Data product references in datasets - Link trigger datasets to actual exposes[] with strong typing and auto-URI generation
• Task output → expose mapping - Formal produces[] declaration for automatic dataset registration and lineage
• Cross-provider workflow patterns - Native support for multi-cloud pipelines with XCom data passing
• Build step references - Link orchestration tasks to transformation steps via buildStepRef
• Data product dependencies - Tasks can declare dependencies on other data products with version constraints
• Enhanced error categorization - Retryable vs non-retryable errors with intelligent handling
• Provider-specific operators - Native support for SnowflakeOperator, BigQueryOperator, GlueOperator, etc.
• Cost tracking enhancements - Track actual vs estimated costs with budget enforcement
✅ BACKWARD COMPATIBLE with v0.5.7:
• All v0.5.7 contracts remain valid (no breaking changes)
• New features are opt-in via new fields
• Existing patterns (hybrid-reference, embedded-logic, multi-stage) fully preserved
• All data quality, observability, policy, and lineage features retained
📋 Inherited from v0.5.7:
• Multiple build patterns with conditional validation
• Enhanced build section for multi-modal concerns
• Comprehensive data quality specs with anomaly detection
• Detailed observability with default SLIs
• Fine-grained policy controls (privacy, RBAC, column-level access)
• Field-level lineage tracking
• Schema evolution strategies
• Machine learning model specifications
• Environment-specific overrides
• Simplified tagging maintained throughout
🎯 Design Philosophy:
• Provider actions = orchestration primitives
• Contracts drive both infrastructure and workflows
• Strong typing and validation
• Natural multi-cloud/multi-provider patterns
• Minimal abstraction layers
• Agentic governance: Data owners control AI/LLM usage boundaries
• Sovereignty-first: Jurisdiction constraints enforced at contract level
⚙️ Orchestration Philosophy (v0.7.x):
• Pragmatic approach: Enumerate known engines (airflow, dagster, prefect) for immediate adoption
• Trade-off: Implementation-specific vs pure abstraction - favoring real-world deployment patterns
• Future path: v0.8.0+ will evolve toward capability-based orchestration model
• Escape hatch: 'custom' engine allows experimentation without schema changes
{
"fluidVersion": "0.5.7",
"kind": "DataProduct",
"id": "gold.customer.analytics_360_v1",
"name": "Customer 360 Analytics",
"description": "Unified customer profiles with ML-driven insights",
"domain": "Customer Experience",
"tags": [
"customer-data",
"analytics",
"gold-layer"
],
"labels": {
"team": "customer-analytics",
"criticality": "high",
"cost-center": "engineering"
},
"metadata": {
"layer": "Gold",
"owner": {
"team": "customer-analytics",
"email": "customer-analytics@company.com"
},
"businessContext": {
"domain": "Customer Experience",
"subdomain": "Customer Intelligence"
}
},
"exposes": [
{
"exposeId": "customer_profiles",
"title": "Unified Customer Profiles",
"version": "2.1.0",
"kind": "table",
"tags": [
"pii",
"customer-facing",
"real-time"
],
"labels": {
"sensitivity": "high",
"retention": "7-years",
"region": "us-central1"
},
"contract": {
"schema": [
{
"name": "customer_id",
"type": "STRING",
"required": true,
"description": "Unique customer identifier",
"sensitivity": "cleartext",
"tags": [
"identifier",
"primary-key"
],
"labels": {
"business-name": "Customer ID",
"data-category": "identifier"
}
}
]
},
"binding": {
"platform": "gcp",
"format": "bigquery_table",
"location": {
"project": "company-data",
"dataset": "gold_customer",
"table": "profiles_v1"
}
}
}
]
}
Contract schema version. Must be exactly '0.7.1' for agentic governance + provider-first orchestration with full backward compatibility to 0.5.7.
Specific value:"0.7.1"Must match regular expression: ^\d+\.\d+(\.\d+)?$
"0.7.1"
Kind of contract. 'MLPipeline' added for basic ML support.
Stable product identifier, e.g. 'gold.hr.employee360v1'.
Must match regular expression:^[a-z0-9_][a-z0-9_.-]*[a-z0-9_]$|^[a-z0-9_]$
Must be at least 1 characters long
Business domain/mesh domain (e.g., 'HR', 'Finance').
Product-level labels for metadata and automation.
{
"team": "customer-analytics",
"criticality": "high",
"cost-center": "engineering",
"retention": "7-years",
"region": "us-central1",
"business-name": "Customer Profiles",
"data-category": "analytical"
}
Each additional property must conform to the following schema
Type: stringLabel values are always strings for consistency and tooling compatibility.
Data layer label (e.g., Bronze/Silver/Gold). Free-form.
Oncall rotation or contact for operational issues.
Business alignment information for data mesh organization.
No Additional PropertiesBusiness domain this product belongs to.
Business subdomain for more granular organization.
Business capability this product supports.
Value stream this product participates in.
Must contain a minimum of 1 items
{
"exposeId": "customer_profiles",
"title": "Customer Profiles Table",
"kind": "table",
"tags": [
"pii",
"customer-facing",
"analytical"
],
"labels": {
"sensitivity": "high",
"retention": "7-years",
"team": "customer-analytics"
},
"contract": {
"schema": [
{
"name": "customer_id",
"type": "STRING",
"required": true,
"sensitivity": "cleartext",
"tags": [
"identifier",
"primary-key"
],
"labels": {
"business-name": "Customer ID",
"data-category": "identifier"
}
}
]
},
"binding": {
"platform": "gcp",
"format": "bigquery_table",
"location": {
"project": "company-data",
"dataset": "customer",
"table": "profiles"
}
}
}
Semantic version (semver.org).
Must match regular expression:^(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)(?:-[0-9A-Za-z.-]+)?(?:\+[0-9A-Za-z.-]+)?$
Hash of canonical schema/OpenAPI. Used by contract-tests gates.
Must match regular expression:^sha256:[0-9a-fA-F]{64}$
Tabular schema (for table/view/file).
No Additional ItemsMust be at least 1 characters long
Must be at least 1 characters long
Field-level sensitivity classification for improved policy ergonomics.
Semantic meaning (e.g., 'email', 'phone', 'identifier').
Business-friendly name for this field.
Business definition and meaning of this field.
Field-level validation constraints.
No Additional ItemsOpenAPI document URL (for APIs).
Compatibility promise for non-breaking evolution.
Enhanced data quality rules including anomaly detection.
No Additional PropertiesISO-8601 duration (e.g., P1D, PT15M, P2Y6M).
Must match regular expression:^P(?!$)(\d+Y)?(\d+M)?(\d+W)?(\d+D)?(T(\d+H)?(\d+M)?(\d+S)?)?$
Availability percentage (e.g., 99.9%).
Must match regular expression:^(100(\.0+)?|\d{2}(\.\d+)?|\d{1}\d(\.\d+)?)%$
e.g., '0 rows'
Target completeness ratio (0.0 to 1.0).
Value must be greater or equal to 0 and lesser or equal to 1
Acceptable error rate (0.0 to 1.0).
Value must be greater or equal to 0 and lesser or equal to 1
Column-level access control.
No Additional ItemsAdditional Properties of any type are allowed.
Type: objectProvider-specific predicate
NEW in v0.7.1: AI/LLM usage governance - controls which AI models can consume this data and for what purposes.
No Additional PropertiesWhitelist of AI models permitted to consume this data. Examples: ['gpt-4', 'claude-3-opus', 'gemini-pro']. Empty array = no AI access.
All items must be unique
No Additional Items^[a-z0-9][a-z0-9-_.]*[a-z0-9]$|^[a-z0-9]$
"gpt-4"
"claude-3-opus"
"claude-3-sonnet"
"gemini-pro"
"gemini-ultra"
"llama-3"
"mixtral-8x7b"
Blacklist of AI models explicitly prohibited from consuming this data. Takes precedence over allowedModels.
All items must be unique
No Additional ItemsMaximum tokens per AI request. Prevents excessive data exposure in single queries.
Value must be greater or equal to 1
4096
8192
16384
Daily token limit for AI consumption. Enforces usage quotas.
Value must be greater or equal to 1
Permitted AI use cases. Controls whether data can be used for training, inference, reasoning, etc.
All items must be unique
No Additional Items[
"inference",
"reasoning",
"summarization"
]
[
"search",
"qa",
"rag"
]
Prohibited AI use cases. Example: Allow reporting but deny model training.
All items must be unique
No Additional Items[
"training",
"fine_tuning"
]
Whether AI agents can use this data for multi-step reasoning/chain-of-thought. False = simple retrieval only.
Whether AI systems can cache/store this data. False = ephemeral processing only.
Data retention requirements for AI systems consuming this data.
Maximum days AI system can retain this data. 0 = no retention allowed.
Value must be greater or equal to 0
Whether AI system must delete data after use.
Whether AI consumption must be logged/audited.
Free-text description of allowed purposes. Example: 'Customer service chatbot only, not for marketing'.
Provider-native addressing with tightened structure for better interop.
No Additional PropertiesCloud account ID
GCP project ID
BigQuery dataset or database name
Database name
Schema name
Table name
Storage bucket name
File or object path
API gateway endpoint
Base URL for API endpoints
Pub/Sub or Kafka topic name
Pub/Sub subscription name
Cloud region
Cloud zone
Unified lifecycle state vocabulary.
Reference to replacement data product or expose.
Human/machine-readable SLI check description
Default SLI trio (freshness, completeness, latency) by kind.
No Additional PropertiesEnable auto-generation of default SLIs.
Default freshness SLI configuration.
No Additional PropertiesFreshness threshold (auto-set by kind: table=PT6H, stream=PT1M, api=PT1S)
Same definition as windowDefault completeness SLI configuration.
No Additional PropertiesCompleteness threshold (auto-set by kind: table=0.95, stream=0.99)
Value must be greater or equal to 0 and lesser or equal to 1
Default latency SLI configuration.
No Additional PropertiesP95 latency threshold (auto-set by kind: api=PT200MS, table=PT5S, stream=PT100MS)
Same definition as windowLatency percentile to measure.
Value must be greater or equal to 0 and lesser or equal to 100
Upstream data product identifier (e.g., 'silver.hr.people_v2'). Renamed from 'provider' for clarity.
Same definition as idSemver range (e.g., ^1.2, ~1.4, >=1.0.0). Not strictly validated by regex; tooling enforces.
Minimum completeness expectation.
Value must be greater or equal to 0 and lesser or equal to 1
Business purpose for consuming this data.
NEW in v0.5.5: Multiple build configurations for multi-modal data products (e.g., batch + streaming, SQL + ML).
No Additional ItemsEnhanced build configuration supporting v0.4.0 patterns with v0.5.5 improvements.
No Additional PropertiesIf the conditions in the "If" tab are respected, then the conditions in the "Then" tab should be respected. Otherwise, the conditions in the "Else" tab should be respected.
"hybrid-reference"
Hybrid-reference pattern from v0.4.0 (dbt-style).
No Additional PropertiesAdditional Properties of any type are allowed.
Type: objectEach additional property must conform to the following schema
Type: enum (of string)Key-value labels for structured metadata and automation. Consistent pattern used throughout FLUID objects.
Same definition as labelsIf the conditions in the "If" tab are respected, then the conditions in the "Then" tab should be respected. Otherwise, the conditions in the "Else" tab should be respected.
"embedded-logic"
Embedded logic pattern from v0.4.0.
No Additional PropertiesAdditional Properties of any type are allowed.
Type: objectKey-value labels for structured metadata and automation. Consistent pattern used throughout FLUID objects.
Same definition as labelsIf the conditions in the "If" tab are respected, then the conditions in the "Then" tab should be respected. Otherwise, the conditions in the "Else" tab should be respected.
"multi-stage"
NEW in v0.5.5: Multi-stage orchestration pattern.
No Additional PropertiesUnified identifier pattern: starts/ends with alphanumeric or underscore, allows dots and hyphens in middle.
Same definition as idKey-value labels for structured metadata and automation. Consistent pattern used throughout FLUID objects.
Same definition as labelsValue must be greater or equal to 1
Key-value labels for structured metadata and automation. Consistent pattern used throughout FLUID objects.
Same definition as labelsBuild identifier for multi-build scenarios (e.g., 'batch-processing', 'ml-training').
Same definition as idHuman-readable description of this build configuration.
Build pattern: hybrid-reference (dbt-style), embedded-logic (raw SQL/code), or multi-stage (complex pipelines).
Pattern-specific build properties with conditional validation.
Build execution configuration (triggers, runtime, retries).
No Additional PropertiesNEW in v0.7.0: Can reference data products directly via exposeRef for strong typing and auto-URI generation. Backward compatible with raw URIs.
No Additional ItemsRaw dataset URI (legacy, e.g., 'dataset://gcp/project/dataset/table')
NEW in v0.7.0: Data product reference with strong typing
Cron expression for schedule trigger
Custom timetable configuration
Each additional property must conform to the following schema
Type: stringContainer image for execution
Service account for cloud execution
Value must be greater or equal to 1
^[a-z0-9_][a-z0-9_.-]*[a-z0-9_]$|^[a-z0-9_]$
Value must be greater or equal to 1
Default configuration for all tasks (can be overridden per task)
Value must be greater or equal to 0
NEW in v0.7.0: Enhanced error categorization and handling
Error types that should trigger retries
No Additional Items[
"TemporaryFailure",
"RateLimitExceeded",
"ConnectionTimeout",
"DeadlineExceeded"
]
Error types that should fail immediately
No Additional Items[
"ValidationError",
"SchemaViolation",
"PermissionDenied",
"InvalidCredentials"
]
Must contain a minimum of 1 items
Backward compatibility: If 'operator' is specified but not 'type', map operator to type
If the conditions in the "If" tab are respected, then the conditions in the "Then" tab should be respected. Otherwise, the conditions in the "Else" tab should be respected.
Operator field is deprecated but still works. Maps: FluidExecuteOperator→fluid_execute, BashOperator→bash, etc.
NEW in v0.7.0: Validate provider_action tasks have required fields
If the conditions in the "If" tab are respected, then the conditions in the "Then" tab should be respected. Otherwise, the conditions in the "Else" tab should be respected.
"provider_action"
Validate fluid_execute tasks
If the conditions in the "If" tab are respected, then the conditions in the "Then" tab should be respected. Otherwise, the conditions in the "Else" tab should be respected.
"fluid_execute"
FluidExecute
Validate bash tasks
If the conditions in the "If" tab are respected, then the conditions in the "Then" tab should be respected. Otherwise, the conditions in the "Else" tab should be respected.
"bash"
Bash
Validate python tasks
If the conditions in the "If" tab are respected, then the conditions in the "Then" tab should be respected. Otherwise, the conditions in the "Else" tab should be respected.
"python"
Python
^[a-z0-9_][a-z0-9_.-]*[a-z0-9_]$|^[a-z0-9_]$
NEW in v0.7.0: Simplified task type taxonomy. Replaces 'operator' field (backward compatible). Types: provideraction (NEW!), fluidvalidate, fluidplan, fluidapply, fluidexecute, fluidverify, fluiddqcheck, bash, python, sensor, or provider-specific operators.
DEPRECATED in v0.7.0: Use 'type' instead. Maintained for backward compatibility with v0.6.1 contracts. Maps to 'type' automatically (FluidExecuteOperator → fluid_execute, BashOperator → bash).
NEW in v0.7.0: Provider name when type=provider_action. Examples: aws, gcp, azure, snowflake, databricks.
NEW in v0.7.0: Provider action when type=provideraction. Format: service.operation (e.g., s3.ensurebucket, table.ensure, kinesis.create_stream).
Must match regular expression:^[a-z0-9_]+\.[a-z0-9_]+$
"s3.ensure_bucket"
"table.ensure"
"database.ensure"
"kinesis.create_stream"
"glue.create_job"
"bigquery.create_table"
NEW in v0.7.0: Reference to transformation step in builds[].transformations[]. Links orchestration tasks to build steps for clearer mapping.
Additional Properties of any type are allowed.
Type: objectNEW in v0.7.0: Declares what exposes[] this task produces. Enables automatic dataset registration and lineage tracking.
No Additional ItemsOptional: Airflow dataset URI. Auto-generated from binding.location if not specified.
How this task updates the data
NEW in v0.7.0: Tasks can declare dependencies on other data products with version constraints and freshness requirements.
No Additional ItemsTask-specific overrides for taskDefaults
XCom configuration for inter-task communication (from v0.6.1)
Task-level access control (from v0.6.1)
Cost tracking (enhanced in v0.7.0)
NEW in v0.7.0: Track actual costs vs estimate
NEW in v0.7.0: Alert if actual > threshold * estimate (e.g., 1.5 = 150%)
NEW in v0.7.0: Budget enforcement
Value must be greater or equal to 1
Value must be greater or equal to 1
List of exposeIds produced by this build.
No Additional ItemsUnified identifier pattern: starts/ends with alphanumeric or underscore, allows dots and hyphens in middle.
Same definition as idDependencies on other builds (for multi-build orchestration).
No Additional ItemsUnified identifier pattern: starts/ends with alphanumeric or underscore, allows dots and hyphens in middle.
Same definition as idLegacy transformations array for backward compatibility.
No Additional ItemsPath/script identifier (dbt model, sql file, etc.)
List of exposeIds materialized by this step. VALIDATION: Tools must validate that every output exists in exposes[].exposeId.
No Additional ItemsUnified identifier pattern: starts/ends with alphanumeric or underscore, allows dots and hyphens in middle.
Same definition as idKey-value labels for structured metadata and automation. Consistent pattern used throughout FLUID objects.
Same definition as labelsSingle build configuration (legacy). Use 'builds' array for multi-modal.
Same definition as builds_itemsOptional lineage tracking for better data governance.
No Additional PropertiesLevel of lineage tracking granularity.
Upstream dependencies and their field mappings.
No Additional ItemsField-level lineage mappings (optional).
No Additional ItemsTransformation logic or SQL expression.
Type of data relationship.
Known downstream consumers (optional).
No Additional ItemsImpact level if this data product changes.
Optional schema evolution strategy for managing changes.
No Additional PropertiesVersioning strategy for schema changes.
Compatibility approach for schema evolution.
Advance notice period for breaking changes.
Value must be greater or equal to 0
Whether schema changes require approval.
List of approvers for schema changes.
No Additional ItemsOptional ML model specifications for MLPipeline kind.
No Additional PropertiesList of feature names used by this model.
No Additional ItemsTarget variable for supervised learning.
Validation metrics (e.g., accuracy, auc_roc).
No Additional ItemsMinimum acceptable metric values.
Each additional property must conform to the following schema
Type: numberOptional environment-specific overrides (dev, staging, prod).
Each additional property must conform to the following schema
Type: objectEnvironment-specific metadata overrides.
No Additional PropertiesEnvironment-specific expose overrides.
No Additional ItemsUnified identifier pattern: starts/ends with alphanumeric or underscore, allows dots and hyphens in middle.
Same definition as idNEW in v0.7.1: Jurisdiction and data residency requirements for compliance enforcement.
No Additional Properties{
"jurisdiction": "EU",
"allowedRegions": [
"eu-west-1",
"eu-central-1"
],
"dataResidency": true,
"crossBorderTransfer": false,
"regulatoryFramework": [
"GDPR"
],
"enforcementMode": "strict"
}
{
"jurisdiction": "US",
"allowedRegions": [
"us-east-1",
"us-west-2"
],
"dataResidency": true,
"regulatoryFramework": [
"CCPA",
"HIPAA"
],
"enforcementMode": "strict"
}
Required legal jurisdiction for data storage and processing. Used to validate binding.location matches sovereignty intent.
"EU"
"US"
"Global"
Explicit list of allowed cloud regions. Example: ['eu-west-1', 'eu-central-1'] for EU-only.
All items must be unique
No Additional Items[
"eu-west-1",
"eu-central-1"
]
[
"us-east-1",
"us-west-2"
]
Explicit list of prohibited cloud regions. Takes precedence over allowedRegions.
All items must be unique
No Additional ItemsWhether data must remain within jurisdiction boundaries at rest and in transit. True = strict residency.
Whether cross-border data transfer is permitted. False = data never leaves jurisdiction.
Allowed legal mechanisms for cross-border transfer (if permitted). Example: ['SCCs', 'BCRs', 'Adequacy'].
All items must be unique
No Additional ItemsApplicable regulatory frameworks. Example: ['GDPR', 'CCPA', 'HIPAA'].
All items must be unique
No Additional Items[
"GDPR"
]
[
"GDPR",
"CCPA"
]
[
"HIPAA"
]
How sovereignty violations are handled. strict = block deployment, advisory = warn, audit = log only.
Whether binding.location must be validated against sovereignty rules before deployment.
Root-level access policy for automated IAM binding generation. Defines principals, permissions, and resources for data product access control.
No Additional Properties{
"grants": [
{
"principal": "group:data-analytics@company.com",
"permissions": [
"read",
"select",
"query"
]
},
{
"principal": "serviceAccount:pipeline@project.iam.gserviceaccount.com",
"permissions": [
"write",
"insert",
"update",
"delete"
]
}
]
}
List of access grants to principals (users, groups, service accounts).
No Additional ItemsIdentity receiving access. Format: 'user:email@domain.com', 'group:name@domain.com', 'serviceAccount:name@project.iam.gserviceaccount.com'
"group:data-analytics@company.com"
"serviceAccount:pipeline@project.iam.gserviceaccount.com"
"user:analyst@company.com"
Permissions to grant. Will be mapped to provider-specific roles (e.g., BigQuery roles).
All items must be unique
No Additional Items[
"read",
"select"
]
[
"write",
"insert",
"update"
]
Resources this grant applies to. JSONPath expressions referencing exposes[] items. If omitted, applies to all exposed resources.
No Additional Items"$.exposes[?(@.exposeId=='customer_table')]"
"$.exposes[?(@.kind=='table')]"
Optional conditions for conditional access (e.g., IP restrictions, time windows).
Additional Properties of any type are allowed.
Type: object