Azure Databricks: Microsoft Fabric Integration and Enhanced Cloud Security

🔄 Update — 09. June 2026: Cross-Tenant Firewall Limitations in Mirrored Databricks Catalogs

A major compatibility gap has emerged when using Microsoft Fabric’s Mirrored Azure Databricks Catalog with firewalled ADLS Gen2 accounts in cross-tenant environments. Querying the catalog triggers a PowerBINotAuthorizedException because Trusted Workspace Access (TWA) cannot traverse tenant boundaries. Consequently, no direct workaround exists when the storage account is protected by an Azure Storage firewall in a different tenant.

What’s new?

PowerBINotAuthorizedException: Users experience blocking errors when configuring Mirrored Azure Databricks Catalogs to query cross-tenant firewalled ADLS Gen2 storage accounts.
TWA Cross-Tenant Incompatibility: Trusted Workspace Access (TWA), which allows Fabric Workspace Identity to authenticate through firewalls, is currently restricted to single-tenant configurations.
Lack of Alternative Paths: Alternative networking features such as VNet data gateways and Managed Private Endpoints are not supported for the Mirrored Catalog flow.

Why this adds to the article

This update directly expands on the “Enforce Network Security” and “Leverage Mirroring” sections of the article by outlining a critical architectural limit of zero-ETL integration in complex, multi-tenant enterprise environments.

Summary

Growing demand for integrated cloud data platforms has accelerated deep interoperability between Azure Databricks and Microsoft Fabric. At the same time, security and governance mechanisms like Unity Catalog and Entra ID are at the center of recent updates. New educational resources and bootcamps are responding to the surging demand for skilled data engineering talent.

What happened?

Several key updates and announcements have recently shaped the Azure and Databricks ecosystem:

Microsoft Fabric Integration: Database mirroring of Azure Databricks (Mirrored Catalogs) allows organizations to expose Unity Catalog tables as read-only replicas directly within Fabric OneLake, without physical data movement.
Entra ID RBAC for Azure Managed Redis: Azure update ID 564873 introduced Microsoft Entra ID-based Role-Based Access Control (RBAC) for data management in Azure Managed Redis (Public Preview), eliminating the need for shared access keys.
Cross-Platform MLflow Logging: Data scientists can now log machine learning experiments and artifacts from Azure Databricks directly into Microsoft Fabric.
Surging Educational Content: New training offerings, including Vaanii’s Free Azure Databricks & PySpark Bootcamp and updated courses on DataCamp, highlight the industry’s push to upskill engineers for modern data stacks.
Terraform Updates: Latest versions of the HashiCorp Terraform azurerm provider continue to enhance the deployment and security configuration of azurerm_databricks_workspace resources.

Why it matters

The debate of choosing Azure Databricks versus Microsoft Fabric has transitioned into a collaborative approach where enterprises leverage both depending on “ecosystem gravity”. Fabric is preferred for deep integration with Microsoft 365, Power BI, and Copilot, while Databricks remains the go-to for complex multi-cloud data engineering and advanced MLOps. Security is the critical link here: since data flows dynamically, access governance must be synchronized. Furthermore, updates like Entra ID RBAC for Managed Redis demonstrate a platform-wide shift by Microsoft toward passwordless, audit-compliant architectures.

Evidence

The trend is supported by several recent indicators:

Skills & Enablement: Vaanii’s “Free Azure Databricks & PySpark Bootcamp” and DataCamp’s “Introduction to Databricks” course demonstrate significant developer interest.
Infrastructure-as-Code: Continued updates to HashiCorp’s Terraform provider for deploying azurerm_databricks_workspace safely.
Zero-ETL & Governance: Microsoft’s Fabric updates detailed how to securely mirror Azure Databricks data into Fabric, using OneLake data access roles and Trusted Workspace Access.
Cloud Security Roadmap: The official Azure updates catalog (ID 564873) announcing the Entra ID-based RBAC for Redis.

Analysis

The integration of Databricks and Fabric via mirroring delivers massive agility (zero-ETL) but introduces governance hurdles. Because Unity Catalog permissions do not automatically replicate to Fabric, security teams must treat Fabric as a separate security perimeter. Syncing Microsoft Entra ID groups across both platforms and mapping them to OneLake data access roles is currently the recommended blueprint. Similarly, migrating Azure Managed Redis to Entra ID-based RBAC aligns with this broader philosophy of eliminating static secrets and ensuring all data assets are governed by a single identity provider.

Practical Takeaways

For data engineers and cloud architects:

Leverage Mirroring: Evaluate Mirrored Azure Databricks Catalogs in Fabric to expose Databricks tables directly to Power BI without writing complex ETL pipelines.
Synchronize Identity Governance: Use Microsoft Entra ID groups as the single source of truth. Manually map these groups to both Unity Catalog privileges and Fabric OneLake Data Access Roles.
Enforce Network Security: Enable “Trusted workspace access” in Fabric for ADLS Gen2 storage accounts behind firewalls, and use VNet data gateways where appropriate.
Go Passwordless: Transition Azure Managed Redis connections to Entra ID-based RBAC to eliminate static access keys.
Automate Deployments: Implement Databricks workspaces using the latest Terraform azurerm provider to ensure consistent security baselines.

Open Questions

How well does catalog mirroring scale in terms of latency and cost when dealing with petabyte-scale, high-velocity streaming data?
When will Microsoft and Databricks introduce native, bi-directional governance synchronization to eliminate the manual mapping of security roles?