Understanding Direct Lake
Direct Lake is a new storage mode that fundamentally changes how data is processed in Power BI. Instead of relying on traditional Import or DirectQuery methods, Direct Lake allows you to work directly with data stored in a Delta format within the Fabric Lakehouse. This means you get the best of both worlds: the performance of Import mode and the real-time data access of DirectQuery.
Many users are concerned about the migration process. Questions like, “Will I have to rebuild my models?” or “How do I handle incremental refresh and composite models?” are common. Fortunately, with the right tools and guidance, migrating to Direct Lake can be straightforward.
Prerequisites for Migration
Before you start the migration process, ensure you have the following prerequisites in place:
- A Fabric workspace with a Lakehouse or Warehouse
- Access rights on the workspace to deploy datasets
- XMLA Read/Write enabled on your Fabric trial or tenant
- Tabular Editor 2 or 3 installed
Setting Up Your Environment
The first step in the migration process is to create your workspace and deploy your existing model. Here’s how to get started:
- Create your workspace in Power BI.
- Deploy your existing model, ensuring that you have all the necessary partitions set up.
Using Semantic Link Labs
Michael Kovalsky has extended SemPy to assist with the migration process, making it easier to transition from Import or DirectQuery to Direct Lake models. This tool wraps the necessary APIs into Python, streamlining the migration process.
To begin, download the Semantic Link Labs notebook from the GitHub repository and import it into your workspace. Remember to switch your persona from Power BI to Data Engineering to import notebooks correctly.
Migration Steps
The migration process involves several key steps:
1. Install Semantic Links Lab
Once you have imported the notebook, the first step is to install the Semantic Links Lab. This may take a minute or so. Once installed, you need to set the dataset name for both the original and the new Direct Lake model.
2. Create Power Query Template
The next step is to create a Power Query template of your existing semantic model. This is done through the notebook, which generates a PQT file that will be used to build your Direct Lake model.
3. Configure Dataflows
Once you have the PQT file, navigate to your lakehouse and select “Get Data” then “New Dataflows Gen2.” Import the Power Query template you just created. Ensure that the connection is properly configured in the data sources under the admin portal.
4. Publish and Refresh
After you have set up your dataflows, it’s time to publish the dataflow to your workspace. This will initiate the data transfer from the source to your lakehouse. Once the dataflow is complete, you can click refresh to ensure everything is up to date.
5. Create the Semantic Model
After refreshing, the next step is to run the script that will create your new semantic model. This involves executing the final code block in the notebook, which will take some time to complete.
Handling Incremental Refresh
One common concern during migration is how to handle incremental refresh. With Direct Lake, the approach to incremental loading changes. You won’t need incremental refresh for the Direct Lake model since it can handle data dumps directly. However, you should ensure that your data loading processes to the lakehouse are set up for incremental loads.
If you had custom parameters for incremental refresh in your original model, be sure to remove them before publishing your new model, as they can interfere with the migration process.
Verifying Migration Success
After completing the migration process, it’s crucial to verify that everything has transitioned smoothly. Check that all measures, calculated tables, and hierarchies have been migrated correctly. Note that while calculated columns are not supported in Direct Lake, calculated tables that do not depend on other data can be migrated successfully.
Final Thoughts
Migration to Direct Lake opens up new possibilities for Power BI users, allowing for improved performance and real-time data access without the hassles of traditional methods. While the process may seem daunting, following these steps will help ensure a successful transition. For those looking to enhance their Power BI skills further, consider enrolling in expert-led training courses. You can boost your data skills with Power BI training that is tailored to help you succeed.
