20-25% of exam
Secure, Monitor, and Troubleshoot Azure Solutions
Protect secrets and configuration with Key Vault and App Configuration, then trace and diagnose production issues with OpenTelemetry and KQL.
Develop Containerized Solutions on AzureDevelop AI Solutions Using Azure Data Management ServicesConnect to and Consume Azure ServicesSecure, Monitor, and Troubleshoot Azure Solutions
Securing secrets with Azure Key Vault
Key Vault is the standard place to keep anything an AI application must never hard-code: API keys for a model endpoint, database connection strings, certificates, and encryption keys.
Storing and retrieving secrets
- Secrets, keys, and certificates are distinct object types in Key Vault with different intended uses — a *secret* is an arbitrary string (a connection string, an API key), a *key* is used for cryptographic operations without ever leaving the vault in plaintext, and a *certificate* combines a key and its public certificate for TLS scenarios.
- Applications should authenticate to Key Vault with a managed identity rather than a client secret — the whole point of Key Vault is removing hard-coded credentials, and authenticating to it *with* a hard-coded credential undermines that.
- Once authenticated, the SDK (`SecretClient` and equivalents) retrieves a secret's current value by name; App Service, Functions, and Container Apps can also reference a Key Vault secret directly from an app setting so application code never has to call the Key Vault SDK explicitly.
Rotation
- A secret in Key Vault can have an expiration date and, for supported secret types, an automatic rotation policy that generates a new version on a schedule before the old one expires.
- Rotation only actually protects you if consumers pick up the new version — an app that cached a secret's value at startup and never re-reads it won't benefit from rotation until it restarts, which is why some architectures poll for the latest version or subscribe to Key Vault's change notifications instead of reading once and holding on indefinitely.
Common confusion
- Enabling soft-delete and purge protection on a vault protects against accidental permanent deletion of secrets — it's a data-protection setting, unrelated to rotation, which is about periodically replacing a secret's *value*, not protecting the vault's contents from deletion.
Azure App Configuration: centralizing application settings
App Configuration is a dedicated service for the non-secret configuration values an application needs — feature flags, connection endpoints, retry policies, service URLs — separate from Key Vault, which is reserved for values that are actually sensitive.
Storing and retrieving configuration
- Configuration is stored as key-value pairs, optionally organized with a *label* (for example, separating `Development` and `Production` values for the same key) so one App Configuration store can serve multiple environments.
- Applications read configuration through the App Configuration provider/SDK, typically at startup, and can be configured to poll for changes on an interval — letting a running application pick up a configuration change (like a feature flag flip) without a redeploy or restart.
- A key can hold a *Key Vault reference* rather than a literal value — the application resolves it through App Configuration, but the actual secret is still fetched from and protected by Key Vault, giving you one place to browse all configuration while secrets stay properly isolated.
Why not just use app settings everywhere
- App Configuration adds a management layer app settings alone don't have: point-in-time snapshots, configuration change history, feature flag management with targeting rules, and a single store shared across multiple services or environments instead of duplicating settings per App Service/Function app/Container App.
Common confusion
- App Configuration and Key Vault are complementary, not competing — the rule of thumb is App Configuration for values that are fine to see in a config blade, and Key Vault (referenced from App Configuration or directly) for anything that would be a security incident if it leaked.
Distributed tracing with OpenTelemetry
A single user-facing AI request often fans out across several services — a Function, a Container App, a Cosmos DB query, a call to a model endpoint — and distributed tracing is what lets you reconstruct that whole path as one coherent timeline instead of a pile of disconnected logs.
How a trace is structured
- A *trace* represents one end-to-end operation; it's made up of *spans*, each representing one unit of work (an HTTP request, a database call, a function execution) with a start time, duration, and its own attributes.
- Spans are linked in a parent-child hierarchy — a span for an incoming HTTP request might have a child span for the Cosmos DB query it triggers, which shows up nested under the request in the trace view, making it obvious which downstream call is responsible for the overall latency.
- OpenTelemetry is the vendor-neutral standard for producing this data: language SDKs instrument common frameworks and libraries automatically (auto-instrumentation) and let you add custom spans by hand for anything auto-instrumentation doesn't cover.
Getting traces into Azure Monitor
- The Azure Monitor OpenTelemetry Distro (or an exporter package like `azure-monitor-opentelemetry-exporter`) sends span data to an Application Insights resource via its connection string, where it powers the *transaction diagnostics* view (one request's full call tree) and the *application map* (how services interact across many requests).
- Many Azure SDKs — including the Cosmos DB SDK — emit their own OpenTelemetry spans for operations like queries, which show up automatically in a trace once the exporter is configured, without you writing any manual instrumentation for that call.
- Every telemetry item carries an operation id shared across the whole distributed operation, which is what lets Application Insights stitch together spans emitted by completely different processes into one trace.
Common confusion
- Enabling an OpenTelemetry SDK and pointing it at *some* backend (like a local Jaeger instance for development) is not the same as it reaching Application Insights — the exporter and connection string must specifically target Azure Monitor for traces to appear there, and forgetting to initialize the OpenTelemetry SDK before other instrumented libraries load is a common reason traces come out incomplete.
KQL for analyzing logs and metrics
Once diagnostic data — logs, traces, and metrics — lands in a Log Analytics workspace, Kusto Query Language (KQL) is how you actually ask questions of it, whether that's during an active incident or while writing an alert rule.
Query fundamentals
- A KQL query starts from a table (`AppTraces`, `AppRequests`, `AppDependencies`, `ContainerAppConsoleLogs`, and so on) and pipes (`|`) data through a sequence of operators, each transforming the result of the previous one — similar in spirit to a Unix pipeline.
- Common operators: `where` filters rows, `summarize` aggregates (counts, averages, percentiles) often combined with `by` to group per dimension, `project` selects and renames columns, and `order by` sorts results.
- A troubleshooting query typically looks like: start from requests or traces, filter to a time window and a specific operation or severity, then summarize failure count or latency percentiles by dependency name or status code to find the actual bottleneck or error source.
Logs vs. metrics for diagnosis
- Metrics are lightweight, pre-aggregated numeric time series (CPU percentage, request count) — fast to query and well suited to real-time alerting, but limited in the questions they can answer.
- Logs are full structured records queried with KQL, which can express arbitrarily complex conditions (correlate a specific customer's failed requests with the exact downstream dependency call that failed) that a metric alone can't represent.
- A production troubleshooting workflow commonly starts with a metric-based alert firing (something is wrong, cheaply detected), then pivots to a KQL query against logs to find out specifically what and why.
Common confusion
- `AppExceptions` (unhandled application exceptions) and `AppTraces` (explicit log statements, at whatever severity the code logged them) are different tables answering different questions — a failing operation that was caught and logged as a warning shows up in `AppTraces`, not `AppExceptions`, so searching only one table can miss real failures the application handled gracefully but still needs attention.