Spark settings in Microsoft Fabric are not new, but they are often misunderstood. Configuring them correctly can significantly improve performance, reduce costs, and prevent capacity conflicts. In this guide, we’ll break down Spark settings in Microsoft Fabric step by step, explaining what each option does and when you should use it.
This article is based on a deep technical walkthrough by Zayn Goodman, Data Engineer Trainer at Pragmatic Works.
Core Apache Spark Concepts You Need to Know
Before diving into Spark settings in Microsoft Fabric, it’s important to understand the building blocks of Apache Spark.
Nodes
A node is a machine (virtual or physical) that provides compute resources such as CPU and memory. In Spark, nodes are where work actually happens.
Executors
Executors live inside nodes. They use the node’s allocated resources to perform tasks. Typically, each node has one executor.
Driver Node
The driver node orchestrates everything. It receives jobs from a Fabric notebook, splits data into smaller tasks, and assigns those tasks to executors. Once processing is complete, the driver gathers results and returns them to the notebook.
Parallel Processing
Spark’s power comes from parallelism. Instead of one machine processing data, multiple nodes and executors work at the same time, dramatically speeding up workloads.
Accessing Spark Settings in Microsoft Fabric
To configure Spark settings in Microsoft Fabric:
- Open your Fabric workspace
- Go to Workspace settings
- Navigate to Data Engineering and Science
- Select Spark settings
From here, you’ll find several tabs that control how Spark behaves in your workspace.
Spark Pools and the Starter Pool
What Is the Starter Pool?
The starter pool is the default Spark pool in Microsoft Fabric. Its biggest advantage is speed—it allows Spark sessions to start in about five seconds because the nodes are prehydrated.
This makes it ideal for:
- Interactive notebooks
- Development and testing
- Quick data exploration
Pool Configuration Basics
A Spark pool defines:
- Node family (currently memory optimized)
- Node size (small, medium, large, etc.)
- Number of nodes
- Number of executors
In the starter pool, some settings are fixed, but you can adjust node and executor counts.
Understanding Node Size, Capacity Units, and vCores
Node size directly controls how much compute power each node has.
Key rules to remember:
- 1 Capacity Unit (CU) = 2 vCores
- Fabric capacity (for example F64) determines the total vCores available
- Node size determines how many vCores each node consumes
For example:
- An F64 capacity provides 128 vCores
- A medium node uses 8 vCores
- That means you can provision up to 16 medium nodes
Choosing the right node size is critical for balancing performance and cost.
Nodes vs Executors: Why the Numbers Don’t Match
You may notice that the number of executors is always one less than the number of nodes.
This is because:
- One node is reserved as the driver node
- The remaining nodes host executors
For example:
- 11 nodes = 10 executors
- 15 nodes = 14 executors
This is expected behavior and ensures Spark can coordinate tasks properly.
Autoscale vs Dynamic Executor Allocation
These two settings sound similar but serve different purposes.
Autoscale (Nodes)
Autoscale controls how many nodes are provisioned. Fabric will:
- Add nodes when workload increases
- Remove nodes when activity drops
This helps reduce wasted resources.
Dynamic Executor Allocation
This setting allows Spark to decide how many executors are actually needed for a job. If fewer executors can handle the workload efficiently, Spark will scale down automatically.
Together, these settings can significantly improve performance while lowering costs.
Environments in Microsoft Fabric
Environments allow you to define reusable Spark configurations.
With environments, you can:
- Install custom Python libraries
- Set Spark properties once
- Share configurations across multiple notebooks
- Improve collaboration and consistency
When enabled, new notebooks can automatically use a predefined environment instead of Fabric’s default runtime.
Runtime Versions and Spark Jobs
If environments are disabled, you can manually choose a Spark runtime version, such as:
- Spark 3.5 with Delta 3.2
- Experimental newer runtimes
Choosing the right runtime ensures compatibility and stability for your workloads.
Reserving Cores for Active Spark Jobs
The Reserve maximum cores for active Spark jobs setting determines how Fabric allocates capacity.
- On: Reserves maximum required vCores upfront
- Off: Starts with minimum cores and scales as needed
You should enable this if you run mission-critical jobs that must not be interrupted by other workloads.
High Concurrency for Notebooks and Pipelines
High concurrency allows multiple notebooks or pipeline steps to reuse the same Spark session.
Benefits include:
- Faster startup times
- Reduced compute overhead
- Lower cost for multi-notebook pipelines
This is especially important when using custom pools, which can take several minutes to start.
Spark Session Timeout
This setting controls how long inactive Spark sessions remain alive. Shorter timeouts reduce wasted capacity, while longer timeouts are useful for active development sessions.
Capacity-Level Spark Settings and Job Bursting
At the capacity level, administrators can enable job bursting.
What Is Job Bursting?
Job bursting allows Spark jobs to temporarily use up to 3× the available vCores.
Pros:
- Faster execution for very large jobs
Cons:
- Can consume capacity quickly
- May block other jobs from running
- Increases job queuing risk
When to Use Bursting
- Enable for isolated, large workloads
- Disable for high-concurrency environments
This setting is enabled by default and should be reviewed carefully.
Final Thoughts
Understanding Spark settings in Microsoft Fabric is essential for anyone running data engineering or analytics workloads. From pools and executors to environments and bursting, each option directly impacts performance, scalability, and cost.
If you want hands-on guidance and structured learning, explore professional Power BI and Fabric training
