Unleashing Data for OneLake with Mirroring in Microsoft Fabric

What is Mirroring in Microsoft Fabric?

Mirroring in Microsoft Fabric is a powerful mechanism that enables near real-time data replication from operational sources or data warehouses into OneLake. Unlike shortcuts that merely point to storage, mirroring creates a direct connection to relational sources with logging transaction capabilities. This allows for automatic and seamless data movement.

With mirroring, you can consolidate your data from different sources, including Azure SQL Database and Azure Cosmos DB, into a unified analytics platform. This integration eliminates data silos and enhances your ability to derive insights from your data efficiently.

Setting Up Mirroring in Microsoft Fabric

To set up mirroring, you’ll need access to a workspace in Fabric that is backed by premium capacity. In this guide, we’ll go through the steps to mirror data from both Azure SQL Database and Azure Cosmos DB.

Step 1: Accessing the Workspace

Start by navigating to your Fabric workspace. Here, you’ll find the option to set up new mirroring connections. Click on “New” and select “More options” to access the data warehouse section.

Step 2: Mirroring Azure Cosmos DB

To mirror data from Azure Cosmos DB, you’ll need the URI and the account key. Copy the URI from your Cosmos DB instance and create a new connection in Fabric by pasting the URI into the endpoint field. After entering the account key, connect to your database.

Step 3: Mirroring Azure SQL Database

Next, you’ll set up mirroring for Azure SQL Database. Similar to Cosmos DB, you’ll need to create a new connection. Input the server name and database name, select your authentication method, and connect to your database.

On this screen, you have the option to choose multiple tables or a single table for mirroring. You can also opt to automatically mirror any new tables created in the source database. However, for this example, we’ll focus on mirroring a specific table called “Orders.”

Monitoring Mirroring

Once you’ve set up mirroring, monitoring the replication process is crucial. In your workspace, you can view the status of your mirrored databases. This includes the number of rows replicated and any potential issues that may arise during the process.

For instance, you can check how many rows have been moved from Cosmos DB and Azure SQL Database. If you encounter any discrepancies, you can stop replication or adjust your configuration as needed.

Handling Schema Changes with Mirroring

One of the standout features of mirroring in Microsoft Fabric is its handling of schema changes. If you need to modify the schema—say, by adding a new column—you can do so without disrupting the mirroring process.

For example, if you add a column called “Store ID” to your Azure SQL Database, this change will be replicated to your mirrored database in Fabric. This capability significantly reduces the overhead typically associated with data integration.

Querying Across Mirrored Sources

Now that you have your data mirrored from both Azure SQL Database and Cosmos DB, you can perform queries across these disparate data sources. This is where the magic of mirroring truly shines.

Using T-SQL, you can write queries that join data from both sources seamlessly. For instance, you can pull order details from Azure SQL and combine them with customer data from Cosmos DB in a single query. This allows for comprehensive reporting and analysis without the need for complex data pipelines.

Benefits of Mirroring in Microsoft Fabric

Implementing mirroring in Microsoft Fabric comes with numerous advantages:

• Near Real-Time Data Access: Data is continuously replicated, ensuring that you always have access to the most current information.
• Simplified Data Management: Mirroring eliminates the need for complex ETL processes, allowing for a more streamlined data integration strategy.
• Unified Analytics Platform: By consolidating data from various sources, you can leverage advanced analytics tools within Fabric for deeper insights.
• Flexibility with Schema Changes: Easily update your schema without affecting ongoing data replication.

Conclusion

Mirroring in Microsoft Fabric is an innovative solution for organizations looking to enhance their data strategy by enabling seamless data integration and real-time analytics. By following the steps outlined in this post, you can set up mirroring for both Azure SQL Database and Cosmos DB, monitor your data replication, and leverage the power of unified analytics for better business insights.

What do you think about mirroring in Microsoft Fabric? Have you tried it out yet? Let us know in the comments below!