Azure Databricks Roadmap: Key Updates for Lakeflow Connect, Billing, and Compliance Security

Summary

Azure Databricks has updated its platform roadmap with several major feature additions, security profile changes, and billing updates slated for mid-to-late 2026. The key transitions involve enabling the Zerobus Ingest connector in Lakeflow Connect and Lakeflow Designer by default for compliance security profile workspaces (HIPAA, HITRUST, etc.) starting mid-to-late July. Databricks is also implementing new cross-region and public-internet data egress charges under OpenSharing SecureConnect, migrating Declarative Automation Bundles (DABs) to a direct deployment engine, and moving Genie products to a pay-as-you-go pricing model.

What happened?

The official Azure Databricks release documentation outlines multiple critical updates and timeline schedules:

• Lakeflow Compliance Integration: The Zerobus Ingest push-based ingestion API and Lakeflow Designer (a visual no-code data preparation canvas) will be enabled by default in workspaces with the compliance security profile in mid-to-late July 2026.
• SQL AI Functions Defaulting: The ai_extract and ai_classify SQL functions will become available by default in workspaces with compliance security profiles enabled under HIPAA, HITRUST, C5, and TISAX controls starting mid-July 2026.
• OpenSharing SecureConnect Egress Fees: Starting mid-2026, Azure Databricks will bill providers for cross-region and public-internet data egress. Same-region egress and SecureConnect ingress remain free.
• DABs Deployment Transition: On July 24, 2026, Declarative Automation Bundles will migrate default configurations to the direct deployment engine, deprecating the Terraform-backed deployment engine.
• Workspace Entitlements Restructuring: Beginning June 15, 2026 (opt-in) and enforced on September 14, 2026, users will no longer inherit workspace privileges from the users system group. Entitlements must be explicitly granted on a per-principal basis.
• Genie Pay-As-You-Go Pricing: On July 6, 2026, Genie products (Genie Spaces, Genie Code, Genie One) will transition to a pay-as-you-go model. Each user gets 150 DBUs of LLM usage per month free, with extra usage billed by DBU.
• Compliance Enforcement: Starting September 1, 2026, the compliance security profile will be strictly required to process data protected under HIPAA, HITRUST, and IRAP.

Why it matters

These changes represent a double-sided shift towards tighter security compliance and granular cost controls:

• Security and Auditing: By removing automatic privilege inheritance from the users system group and forcing explicit entitlement provisioning, Databricks addresses a critical security gap where read-only or consumer-only users accidentally inherited write/authoring privileges.
• Architecture Migration: Deprecating the Terraform deployment engine in favor of the direct deployment engine for DABs simplifies deployment pipelines and reduces dependency overhead.
• Data Egress Billing: Data transfer costs can spiral quickly. Egress fees for OpenSharing SecureConnect mean providers must carefully plan spatial data architecture and recipient locations.
• Genie Monetization: Shifting Genie products to pay-as-you-go reflects the scaling costs of running state-of-the-art LLMs, requiring teams to monitor DBU consumption.

Evidence

The changes are documented in the official Azure Databricks release planning documentation:

• Release Documentation: Azure Databricks “What’s coming” documentation page.
• System Group Entitlements: Detail pages on workspace entitlement migration starting June 15, 2026.
• Pricing Guides: Data transfer and connectivity pricing details on the official Databricks website.

Analysis

The updates reflect Azure Databricks maturing its governance capabilities. For instance, the default inclusion of ai_extract and ai_classify in compliance security profile workspaces shows that enterprise customers are increasingly demanding secure AI capabilities that conform to strict HIPAA and HITRUST standards. Furthermore, the deprecation of the Terraform deployment engine for DABs is a strategic pivot. While Terraform provided flexibility, it introduced state file drift and security overhead in automated environments. The direct deployment engine promises faster, more robust deployments directly integrated with the Databricks platform APIs. Finally, the changes to system groups represent a major security hygiene improvement. In large organizations, directory synchronization via SCIM often auto-populated the default users group, granting developers and business analysts broad workspace authoring access by default. Restricting this ensures least-privilege access principles are maintained.

Practical Takeaways

Organizations utilizing Azure Databricks should immediately take the following actions:

1. Audit Workspace Entitlements: Review all Terraform, SCIM API, or custom scripts managing system groups. Ensure they do not target the users or admins system groups, and prepare for the migration to users-clone-<TIMESTAMP> groups.
2. DABs Engine Migration: Check all Declarative Automation Bundles to ensure compatibility with the direct deployment engine before the July 24, 2026 deadline.
3. Genie Budgeting: Estimate LLM usage for active Genie Spaces and Genie Code users. Set up monitoring to manage DBU usage beyond the 150 DBU monthly free tier.
4. Review Data Sharing Geography: Map out existing OpenSharing SecureConnect endpoints and check if they cross regional boundaries or utilize the public internet to forecast future egress billing.

Open Questions

• What is the definitive deprecation timeline for the Terraform deployment engine after the July 24 default switch?
• How will the pay-as-you-go pricing for Genie products scale for large-scale enterprise deployments with hundreds of users?

Sources

1. What’s coming? - Azure Databricks
2. Data transfer and connectivity pricing
3. Migrate workspace entitlement control