Apache Spark and Notebooks

Apache Spark is an open-source parallel-processing framework for large-scale analytics. Fabric provides an integrated environment for Spark-based ingestion, transformation and analysis within a lakehouse.

Core Spark Components - Head node / driver: coordinates the application and distributed processing - Worker nodes / executors: perform the actual processing tasks - Spark pool: the compute cluster used to execute workloads - Runtime: defines Spark, Delta Lake and Python versions - Environment: stores libraries and custom configuration for consistent execution

A DataFrame is Spark's primary structured-data abstraction. It supports chained transformations and can load from files or tables.

Spark Operations and Their SQL Equivalents - select() maps to a SQL SELECT column list - where() maps to a SQL WHERE condition - groupBy().count() maps to GROUP BY with aggregation - write.mode("overwrite") replaces the existing target data - partitionBy() physically organises files by a selected column

Schema Choice - Schema inference is convenient, but an explicit schema is more reliable and can be more efficient for repeatable production workloads

Spark SQL allows relational querying inside notebooks. A temporary view is session-based, while a table saved to the catalog is persistent.

SQL Server Concepts vs Spark SQL - A local temporary table maps to a temporary view created with createOrReplaceTempView - A permanent table maps to a managed table, commonly stored in Delta format - An external table maps to metadata in the catalog with files stored elsewhere - An SSMS query tab maps to a notebook cell using the %%sql magic command

Saving and Partitioning Data - Parquet is a compressed columnar format suited to analytical workloads - Delta tables add transactions, schema controls and version history on top of Parquet files - Partitioning can improve selective queries but should be reserved for large tables and sensible low-cardinality filter columns