Real-Time Intelligence

Real-time analytics processes and acts on data as it is generated, usually within seconds or minutes. An event is one digital occurrence; a stream is the continuous chronological sequence of events.

Core Real-Time Components - Real-Time Hub: discover streaming sources - Eventstream: capture, transform and route events - Eventhouse with KQL databases: store high-velocity time-series data - KQL queryset: write and save analytical queries - Real-Time Dashboard: display live operational metrics - Activator: trigger actions from conditions

Eventstream Stages - Sources: Azure Event Hubs, IoT Hub, Service Bus, Kafka, MQTT, Google Cloud Pub/Sub and Fabric events - Transformations: filter, manage fields, aggregate, group by time window, union, join and expand - Destinations: Eventhouse, Lakehouse, Activator, derived stream or custom endpoint

Scenario Guide - Use Eventstream when data must be transformed or routed while in transit - Use direct ingestion when the priority is rapidly landing high-volume data, transforming it afterwards with update policies

An Eventhouse is a container for real-time data stores. Its KQL databases are optimised for append-heavy, time-series workloads and automatically organise data for efficient recent-data analysis.

KQL Practices Worth Remembering - Filter early, especially by time, to reduce scanned data and use indexing effectively - Project only required columns to reduce memory and processing overhead - Use the smaller input on the left of a join to improve join efficiency - Use limit for exploration to prevent unnecessary large result displays - Use materialized views to precompute recurring aggregations while incorporating new delta data - Use stored functions to standardise reusable and parameterised KQL logic

Real-Time Dashboards - Tiles execute KQL queries and refresh automatically - Base queries allow several tiles to reuse the same core query logic - Parameters make dashboards interactive for viewers - Multiple pages organise complex operational monitoring scenarios

Activator Activator detects conditions in changing data and performs actions, following a Connect, Monitor, Act model. - Object: a real-world entity such as a device, package or customer - Property: a tracked attribute such as temperature or location - Event: an incoming record that updates the object state - Rule: the condition, threshold or pattern that triggers action - Action: email, Teams, Power Automate, notebook or pipeline execution

Rule Design - Use summarisation windows and duration conditions to avoid triggering on short-lived noise - Property filters can narrow a rule to selected objects, priorities or locations