Apache Spark and Notebooks

Apache Spark is an open-source parallel-processing framework for large-scale analytics. Fabric provides an integrated environment for Spark-based ingestion, transformation and analysis within a lakehouse.

Core Spark Components

• Head node / driver: coordinates the application and distributed processing
• Worker nodes / executors: perform the actual processing tasks
• Spark pool: the compute cluster used to execute workloads
• Runtime: defines Spark, Delta Lake and Python versions
• Environment: stores libraries and custom configuration for consistent execution

Fabric also provides starter pools and configurable pools with autoscale and dynamic allocation. Native Execution Engine can accelerate Parquet and Delta processing, while high-concurrency mode can share a Spark session across notebooks when appropriate.

A DataFrame is Spark's primary structured-data abstraction. It supports chained transformations and can load from files or tables.

Spark Operations and Their SQL Equivalents

• select() maps to a SQL SELECT column list
• where() maps to a SQL WHERE condition
• groupBy().count() maps to GROUP BY with aggregation
• write.mode("overwrite") replaces the existing target data
• partitionBy() physically organises files by a selected column

Schema Choice

• Schema inference is convenient, but an explicit schema is more reliable and can be more efficient for repeatable production workloads.

Saving and partitioning

• Parquet is a compressed columnar format suited to analytical workloads.
• write.mode("overwrite") replaces the target contents; append adds new output.
• partitionBy() organizes files by a selected column and can reduce disk I/O for selective queries.
• Avoid excessive partitions and choose columns with sensible cardinality and common filtering patterns.

Spark SQL allows relational querying inside notebooks. A temporary view is session-based, while a table saved to the catalog is persistent.

SQL Server Concepts vs Spark SQL

• A local temporary table maps to a temporary view created with createOrReplaceTempView
• A permanent table maps to a managed table, commonly stored in Delta format
• An external table maps to metadata in the catalog with files stored elsewhere
• An SSMS query tab maps to a notebook cell using the %%sql magic command

Saving and Partitioning Data

• Parquet is a compressed columnar format suited to analytical workloads
• Delta tables add transactions, schema controls and version history on top of Parquet files
• Partitioning can improve selective queries but should be reserved for large tables and sensible low-cardinality filter columns

Notebook visualization

• Fabric can display query results as a table or a built-in chart without additional code.
• Use matplotlib or seaborn for customized Python visualizations.
• Convert a manageable Spark result to Pandas with toPandas() only when the result fits in driver memory.