30-35% of exam
Monitor and Maintain Data
Monitor pipeline runs, optimize performance, and maintain data quality
Ingest and Transform DataFabric Foundations and OneLakeApache Spark and NotebooksLakehouse, Delta Lake and Medallion ArchitectureFabric Data WarehouseStreaming Data and Real-Time IntelligenceMonitor and Maintain DataCI/CD and Lifecycle ManagementSecure DataFinal Revision: Scenario and Component Guide
Pipeline Monitoring
Monitor and troubleshoot data pipelines:
Pipeline Monitoring Hub
- View all pipeline runs across workspaces
- Filter by status, date range, and pipeline name
- Drill into run details and error messages
Logging and Diagnostics
- Enable diagnostic settings for detailed logs
- Send logs to Log Analytics or Event Hub
- Query logs using KQL
Alerts and Notifications
- Set up alerts for pipeline failures
- Configure email/Teams notifications
- Use Fabric Activator for business rule-based alerts
Performance Optimization
Optimize query and pipeline performance:
Spark Optimization
- Configure cluster sizes based on workload
- Use adaptive query execution
- Cache frequently accessed DataFrames
- Optimize partition sizes (128MB-1GB recommended)
SQL Endpoint Optimization
- Create appropriate indexes
- Use statistics for query optimization
- Partition large tables
- Materialized views for repeated queries
Lakehouse Optimization
- OPTIMIZE tables regularly
- VACUUM old snapshots
- Use Z-ORDER for frequently filtered columns
- Monitor table sizes and file counts
Data Quality
Implement data quality checks:
Great Expectations Integration
- Define expectations for data validation
- Run validation checks in notebooks
- Generate data quality reports
Data Profiling
- Use built-in data profiling in Lakehouse explorer
- Identify nulls, uniqueness, and distributions
Lineage Tracking
- View end-to-end data lineage
- Impact analysis for changes
- Column-level lineage in notebooks