Fabric Notebook with PySpark: Easy Setup Guide

Fabric Notebook with PySpark is an incredibly powerful way to begin your journey into big data processing with Microsoft Fabric. Whether you’re a data engineer or just exploring what’s possible with Spark, Microsoft Fabric now makes it easier than ever to interact with your data using notebooks, Spark clusters, and a user-friendly interface.

In this post, we’ll walk through the exact steps to set up your Fabric environment, create a Lakehouse, upload data, and start running your first PySpark notebook.

Prerequisites

To follow along, make sure you have:

• A Microsoft Fabric license (or a trial)
• A Fabric-enabled workspace
• Access to a CSV file (you can download it from the video’s description or get Power BI training from DataBear to supplement your skills)

Step 1: Set Up Your Fabric Workspace

Navigate to your Power BI Service, switch to the Data Engineering Persona, and create a new Lakehouse:

• Click the Power BI icon in the lower left
• Open the Persona switcher
• Select Data Engineering
• Choose Lakehouse and give it a unique name (e.g., Lakehouse_PySpark)

This Lakehouse will be your data container for files and tables you interact with in your notebook.

Step 2: Upload a CSV File to the Lakehouse

Once your Lakehouse is created:

1. Select Get Data from the ribbon
2. Click Upload Files
3. Choose your CSV file (e.g., holiday.csv)
4. Upload it into the Files folder

To work with the data more dynamically, drag and drop the file into the Tables folder. Microsoft Fabric will automatically convert it into a Delta table.

Step 3: Open and Explore a Fabric Notebook

With your data in place:

• Click Open Notebook in your Lakehouse UI
• Create a new notebook and name it appropriately
• The notebook will automatically tie to your Lakehouse

You’ll now be inside the Fabric Notebook interface, where you can write, execute, and manage code.

Step 4: Work with Code Cells and Markdown

Fabric Notebooks support two main types of cells:

• Code Cells (live PySpark, Scala, SQL, or R)
• Markdown Cells (for documentation and notes)

To start with PySpark:

1. Add a new code cell
2. Drag and drop your Delta table (e.g., holiday.csv) into the code cell
3. Fabric will auto-generate PySpark code to load a DataFrame
4. Click Run Cell to execute the query

This will launch a Spark session in the background (may take 30–90 seconds initially). Once the compute is ready, the cell will return a preview of your data.

Step 5: Manipulate and Explore Your DataFrame

To continue working with your DataFrame, you can:

# Reference the DataFrame and display it
df.show()

This displays the data in a tabular format, allowing you to preview it, apply transformations, and explore insights.

Understanding Spark in Fabric

Spark in Microsoft Fabric uses Spark clusters distributed computing environments that process data in memory. This architecture allows for fast, parallel processing of large datasets.

While Spark is designed for big data, it works just as well with smaller files. It’s perfect for:

• Interactive analytics
• Machine learning pipelines
• Large-scale ETL operations

Summary & Next Steps

Setting up a Fabric Notebook with PySpark is easier than ever with Microsoft’s new interface. In just minutes, you can:

• Create a Lakehouse
• Upload your own files
• Convert them into Delta tables
• Run PySpark queries inside a notebook

For further learning, explore Power BI training at DataBear or Pragmatic Works’ bootcamps, and stay tuned for the next part in this Fabric + PySpark series!