Implement Machine Learning Model Lifecycle and Operations

Azure Machine Learning uses MLflow as its native experiment-tracking API — logging with the open-source MLflow SDK inside a job automatically lands metrics, parameters, and artifacts in the workspace, with no Azure-specific logging code required.

MLflow tracking inside a job

• `mlflow.set_tracking_uri(...)` (or simply running inside an Azure Machine Learning job, where it's configured automatically) points the MLflow client at the workspace instead of a local or external tracking server.
• `mlflow.autolog()` instruments common frameworks (scikit-learn, PyTorch, LightGBM, and others) to log parameters, metrics, and the trained model automatically, without explicit `log_metric`/`log_param` calls scattered through training code.
• For anything autolog doesn't capture, `mlflow.log_metric(name, value)`, `mlflow.log_param(name, value)`, and `mlflow.log_artifact(path)` log explicitly inside a run — and a sweep job's primary metric (covered next) must be logged with exactly this API for the sweep controller to read it.

From notebook to command job

• An exploratory notebook doesn't scale to repeatable, scheduled, or CI-triggered training — the production pattern converts that logic into a standalone Python script accepting arguments (via `argparse` or similar), then runs it as a `command` job.
• A command job YAML specifies the `code` directory, the `command` to execute (for example `python train.py --learning_rate ${{inputs.learning_rate}}`), the `environment`, and the `compute` target — the same script then runs identically whether launched from the CLI, a pipeline step, or a GitHub Actions workflow.
• Job inputs/outputs use the `${{inputs.<name>}}` / `${{outputs.<name>}}` binding syntax, letting a job's parameters (a learning rate, a registered data asset) be swapped without touching the script itself.

Common confusion

• `mlflow.autolog()` captures what a specific framework integration knows how to log automatically — it won't pick up a custom metric your code computes that isn't part of that framework's standard training loop; anything domain-specific still needs an explicit `mlflow.log_metric()` call.

Sweeping and AutoML solve related but distinct problems: AutoML searches over *which model and featurization* to use, while a sweep job searches over *hyperparameter values* for a training script you've already written.

Automated ML

• An AutoML job takes a task type (classification, regression, forecasting, or an image/NLP task), a target column, and training data, then trials multiple algorithms and featurization strategies automatically, ranking candidates by the configured primary metric.
• The Responsible AI dashboard can be generated for an AutoML (or any MLflow) model afterward, surfacing fairness, explainability, and error-analysis views for the trained model — this is a separate step from training, not something that runs automatically inside every job.

Sweep jobs: sampling algorithms

• `random`: draws values uniformly from the search space (or via the Sobol quasi-random sequence with `rule: sobol` for better space coverage and reproducibility); supports early termination and is a common starting point.
• `grid`: exhaustively tries every combination of `choice`-type hyperparameters — accurate but expensive, and unusable with continuous distributions.
• `bayesian`: picks each new trial's values based on the results of previous trials, converging on promising regions faster than random search, but is incompatible with early termination policies since it needs completed trials to inform the next one.

Early termination policies

• `BanditPolicy`: cancels a trial if its primary metric falls outside a `slack_factor` (relative) or `slack_amount` (absolute) distance from the best trial so far, checked every `evaluation_interval`.
• `MedianStoppingPolicy`: cancels a trial if its performance is worse than the median of all trials at the same point — a conservative choice, commonly cited as saving 25-35% of compute with minimal impact on the final result.
• `TruncationSelectionPolicy`: cancels the worst `truncation_percentage` of running trials at each evaluation interval — more aggressive than Bandit or Median stopping.
• The sweep job's `objective.primary_metric` must exactly match the name used in the script's `mlflow.log_metric()` call, and `objective.goal` (`maximize` or `minimize`) tells the sweep controller which direction is "better."

Common confusion

• Grid and random sampling both support early termination; Bayesian sampling does not, because each new trial's parameter choices depend on the full results of prior trials rather than being independent — cutting a trial short would corrupt the very history the algorithm relies on.

A single command job is fine for one step; anything with multiple stages (prep, train, evaluate) or that needs to scale across many machines calls for a pipeline or a distributed training configuration instead.

Pipelines built from components

• A pipeline job wires multiple components together, binding one component's output to the next component's input (`${{parent.jobs.prep_step.outputs.clean_data}}`) so Azure Machine Learning can resolve the dependency graph and run independent steps in parallel automatically.
• Because each step is a versioned component, the exact same "featurize" step can be reused across a training pipeline and a batch-scoring pipeline without duplicating its definition — and a pipeline run's lineage shows exactly which component version produced which output.
• Pipelines are the unit that gets scheduled (covered in the next section) for recurring retraining, and the unit registered as an endpoint for batch inference.

Distributed training

• For a single command job that needs multiple nodes or multiple GPUs, the job's `distribution` block specifies a framework: `PyTorch` (`process_count_per_instance` plus the job's `instance_count`), `MPI`, or `TensorFlow` (with separate worker and parameter-server counts).
• Azure Machine Learning sets up the distributed environment variables (rank, world size, master address) automatically based on this configuration — the training script only needs to read them through the framework's own APIs (for example, PyTorch's `torch.distributed`), not hand-roll cluster coordination.
• Distributed training is a scaling decision independent of hyperparameter tuning — a sweep job's individual trials can themselves each be distributed multi-node jobs, for large models where even a single trial doesn't fit on one machine.

Common confusion

• A pipeline's steps run as independent jobs connected by data dependencies (parallelizing *stages* or independent branches); distributed training splits *one* training job's compute across multiple nodes or GPUs working on the same task together. Needing both is common — the "train" step of a pipeline can itself be a distributed job — but they solve different scaling problems.

Registering a model is what turns a run's output into a durable, versioned asset other steps — deployment, another team's pipeline — can reference by name, independent of whether the original run or its compute still exists.

Registering from a run

• `mlflow.register_model(f"runs:/{run_id}/{artifact_path}", model_name)` registers a model MLflow logged inside a run, using the MLflow `runs:/` URI scheme — this preserves lineage back to the exact run and its logged parameters/metrics.
• The Azure CLI equivalent, `az ml model create --name my-model --version 1 --path runs:/<run-id>/model/ --type mlflow_model`, or the Python SDK's `Model(path="runs:/<run-id>/model/", type=AssetTypes.MLFLOW_MODEL)` followed by `ml_client.models.create_or_update(...)`, does the same thing outside the MLflow API directly.
• The `azureml://jobs/<job-name>/outputs/<output-name>/paths/<path>` URI form registers a model from any named output of a job, useful when a model wasn't logged via MLflow directly inside the run but still needs lineage back to the job that produced it.

Versioning behavior

• Registering under a model name that doesn't exist yet creates version 1; registering again under the same name creates version 2, and so on — versions are immutable once created, so "updating" a registered model always means adding a new version, never overwriting an old one.
• Consumers reference either a pinned version (`azureml:my-model:3`) for reproducibility, or `azureml:my-model@latest` to always resolve to whatever was registered most recently — the same latest-alias pattern used for environments.

Responsible AI evaluation and lifecycle

• A Responsible AI dashboard (fairness, explainability, error analysis, causal analysis components) can be generated against a registered MLflow model, typically as a step between registration and production deployment, so a model with a known fairness or explainability problem is caught before it serves traffic.
• Archiving a model version (rather than deleting it) keeps it visible in history and still resolvable by exact version for auditing or rollback, while removing it from "latest" resolution and default listings — the lifecycle equivalent of deprecating without destroying evidence of what was once in production.

Common confusion

• `mlflow.register_model` and `mlflow.<flavor>.log_model(..., registered_model_name=...)` both end up registering a model, but only within the *same workspace where the run itself was tracked* — Azure Machine Learning doesn't support registering a model into a different workspace's registry directly from an MLflow call; cross-workspace sharing goes through a registry (see the MLOps infrastructure topic), not through MLflow's registration API.

A managed online endpoint is a stable, versioned HTTPS URL; the actual serving code and model live in one or more *deployments* underneath it — separating the stable address from the thing currently answering requests is what makes safe rollout possible.

Endpoints and deployments

• Creating an endpoint alone doesn't serve traffic — at least one deployment (model, environment, scoring script or MLflow model, and instance count/size) must exist and receive traffic allocation before it responds to requests.
• `traffic` on the endpoint is a percentage map across deployments (`blue: 90 green: 10`) that must sum to 100 (or 0, to disable all traffic) — this is the mechanism behind blue/green deployment: create `green` with new code or a new model version, give it 0% traffic, validate it in isolation, then shift the split gradually.
• A request can bypass the traffic split entirely by setting the `azureml-model-deployment` HTTP header to a specific deployment name — useful for direct testing of a 0%-traffic deployment without touching the live split.

Traffic mirroring (shadow testing)

• `mirror_traffic` copies a percentage of *live* traffic to a second deployment without returning its results to the caller — the caller still only ever sees the primary deployment's response, but the mirrored deployment's logs and metrics can be inspected for latency or error-rate problems under real production load.
• Mirroring is capped at 50% of traffic, applies to only one deployment at a time, and isn't available for Kubernetes online endpoints (managed online endpoints only) — `az ml online-endpoint update --name $ENDPOINT_NAME --mirror-traffic "green=10"` mirrors 10% to `green` while `blue` continues serving 100% of live responses.

Rollback

• Because traffic allocation is just a percentage map, rollback is the same mechanism run in reverse: set the previous deployment back to 100% and the failing one to 0% — no redeploy is required, since both deployments are still running side by side until one is explicitly deleted.

Common confusion

• Mirrored traffic and a 10%-live-traffic split look similar operationally but behave very differently for the caller: a live split actually returns the new deployment's predictions to whichever fraction of callers land on it, while mirroring never changes what any caller sees — it exists purely to observe a new deployment's behavior under real traffic before it's trusted with a live split at all.

Deploying a model isn't the end of the lifecycle — its input data and its own predictions can silently drift from what it was trained and validated on, and monitoring is what turns that into something actionable instead of something discovered from a customer complaint.

How monitoring works

• Monitoring compares a *production* data window against a *reference* (baseline) data window — usually the training data, or a recent past production window — using a statistical test or distance metric per signal.
• Production inference data collection has to be explicitly enabled on the online deployment first; without it, there's no production data for a monitor to compare against at all.

Built-in monitoring signals

• Data drift: compares the distribution of input feature values in production against the reference window, using metrics like `jensen_shannon_distance`, `population_stability_index`, `normalized_wasserstein_distance` (numerical features), or `chi_squared_test` (categorical features).
• Prediction drift: the same style of comparison, but applied to the model's *output* distribution rather than its inputs — a model can have stable inputs but drifting predictions if the relationship it learned no longer holds.
• Data quality: flags problems in the incoming data itself independent of drift — `null_value_rate`, `data_type_error_rate`, `out_of_bounds_rate` catch pipeline problems upstream of the model, like a feature that started arriving as a string instead of a number.

Scheduling and alerting

• `az ml schedule create -f monitor.yaml` runs a monitoring job on a recurring schedule against a Spark compute instance; the YAML's `monitoring_target.endpoint_deployment_id` (format `azureml:<endpoint>:<deployment>`) ties the monitor to a specific live deployment.
• Each signal's `metric_thresholds` defines when a metric counts as an anomaly; `alert_enabled: true` sends an email through `alert_notification.emails` when any threshold is breached, which is the trigger a team uses to decide whether retraining is warranted.
• An out-of-box configuration needs no explicit `monitoring_signals` at all — Azure Machine Learning defaults to data drift, prediction drift, and data quality with sensible thresholds, which is enough to catch the most common production problems without hand-tuning every metric up front.

Common confusion

• A monitoring alert firing doesn't mean the model is definitely wrong — it means the *statistical properties* of production data or predictions have shifted from the reference window by more than the configured threshold. Confirming whether that shift actually degraded real-world accuracy (versus a benign, expected seasonal change) is a separate investigation step before deciding to retrain.