Design and Implement an MLOps Infrastructure

The Azure Machine Learning workspace is the top-level container every job, model, endpoint, and asset lives inside — provisioning it correctly is the first MLOps decision, since several of its settings (identity type, associated resources) can't be changed after creation.

Datastores

• A datastore is a saved reference to an existing storage account (blob container, ADLS Gen2 container, file share) — it stores the connection information, not the data itself, so registering a datastore doesn't copy anything.
• Every workspace gets a default datastore backed by its associated storage account automatically; additional datastores are registered explicitly to point at other storage, including storage owned by another team or subscription.
• Datastores authenticate either with a stored credential (account key or SAS token) or, the preferred production pattern, with the workspace's managed identity via Microsoft Entra ID — removing a secret that would otherwise need rotation.

Compute targets

• Compute instance: a single-user development VM for notebooks and interactive testing — not meant to run unattended production jobs.
• Compute cluster (`AmlCompute`): an autoscaling pool of VMs for training jobs, scaling between a configured `minNodeCount` and `maxNodeCount`; setting `minNodeCount: 0` means the cluster costs nothing while idle, at the cost of a cold-start delay on the next job.
• Kubernetes compute (attaching an existing AKS cluster) and serverless compute (Azure Machine Learning manages the compute lifecycle entirely, with no cluster to size or manage) are the other two options, most relevant for teams standardizing on Kubernetes or wanting to skip capacity planning altogether.

Identity and access for the workspace itself

• The workspace has its own system-assigned or user-assigned managed identity, used for the workspace to reach its associated Key Vault, storage account, and container registry without embedded credentials.
• RBAC roles assigned *on* the workspace (Contributor, the built-in AzureML Data Scientist, or a custom role) control who can submit jobs, register models, and manage compute — separate from the identity the workspace itself uses to reach its dependencies.

Common confusion

• A datastore's own authentication (how the *workspace* reaches storage) and a user's RBAC role on the workspace (what *that person* is allowed to do) are independent layers — a user with Contributor on the workspace can still be blocked from the underlying storage account if its network rules or access policies don't separately permit it.

Assets are the versioned building blocks a job references — the point of registering something as an asset instead of using a raw path is that every job that consumed a specific version stays reproducible even after the underlying file changes.

Data assets

• A data asset wraps a path (`uri_file`, `uri_folder`, or `mltable`) with a name and version, so a training job can reference `azureml:sales-features:3` instead of a mutable path that might point at different content next week.
• `mltable` additionally captures schema and read/transform logic (column types, a delimiter, which files to include from a folder) as part of the asset definition, not just a location.

Environments

• An environment pins the Python packages, base Docker image, and any environment variables a job's code runs against — defined from a conda YAML plus a base image, or from a fully custom Dockerfile.
• Curated environments (Microsoft-maintained, prefixed `AzureML-`) cover common frameworks out of the box; a custom environment is built once and then reused by every job that needs the same dependency set, keeping training and deployment consistent.
• Environments are versioned automatically on rebuild, so a job specification like `environment: azureml:my-training-env@latest` always resolves to the most recently built image without editing every job definition.

Components

• A component packages a single reusable pipeline step — its inputs, outputs, code, environment, and command — so the same "train a model" or "validate data" step can be dropped into multiple pipelines without copy-pasting its definition.

Registries: sharing assets across workspaces

• By default, models, environments, components, and data assets live in one workspace and aren't visible from another. A registry is a separate, workspace-independent container for exactly these asset types, replicated across whichever regions you configure.
• A central platform team publishes a validated environment or a training component to a registry once; every team's workspace then references it as `azureml://registries/<registry-name>/environments/<name>/versions/<version>`, guaranteeing every workspace trains against the identical, approved definition instead of copies that can drift.

Common confusion

• A workspace's default asset store and a registry solve different problems: the workspace store versions assets *for that workspace's own jobs*; a registry exists specifically so multiple workspaces can share the same asset without each team re-registering and maintaining its own copy.

Standing up an Azure Machine Learning workspace by hand in the portal doesn't scale past a single team, and it leaves no record of exactly how the environment was configured — Bicep and the Azure CLI make that provisioning declarative and repeatable.

Bicep for the workspace and its dependencies

• The workspace itself deploys as a `Microsoft.MachineLearningServices/workspaces` resource, but it also requires an associated storage account, Key Vault, and (for full telemetry) an Application Insights resource — a workspace Bicep template typically provisions all four together, wiring the workspace's `properties` to their resource IDs.
• Compute can be declared inline on the workspace resource's `computes` array (creating a new `AmlCompute` cluster with a `scaleSettings` block for `minNodeCount`/`maxNodeCount`) or attached separately as its own resource — both approaches are idempotent, so re-running the same deployment converges to the same state instead of duplicating resources.
• Setting the workspace identity block (`systemAssignedIdentity: true` or a specific user-assigned identity) at deployment time is what lets the workspace authenticate to its dependencies without a stored secret from the moment it's created.

Azure CLI for day-to-day and scripted operations

• The `az ml` extension (CLI v2) drives everything after the workspace exists: `az ml workspace create`, `az ml datastore create -f datastore.yml`, `az ml compute create -f cluster.yml`, and so on, each backed by a YAML file matching that resource's schema.
• CLI commands are the natural fit for a GitHub Actions step, since they're scriptable and produce predictable exit codes — a workflow step failing on a bad `az ml job create` is easy to gate a pipeline on, in a way a manual portal click isn't.

Common confusion

• Bicep provisions the *infrastructure* (the workspace, its compute, its network configuration) — it doesn't submit training jobs or register models. Those day-to-day MLOps operations run through the CLI or SDK against an already-provisioned workspace, which is why real pipelines combine both: Bicep for the one-time (or infrequently changed) environment, CLI/SDK for the jobs that run against it continuously.

Automating provisioning and training through GitHub Actions only pays off if the authentication behind it is secure and the workspace's network posture doesn't quietly undermine everything else.

Authenticating GitHub Actions to Azure without long-lived secrets

• The recommended pattern is OpenID Connect (OIDC) via the `azure/login` action: a Microsoft Entra application (or a user-assigned managed identity) is configured with a federated identity credential that trusts tokens GitHub issues for a specific repository, branch, or environment.
• With OIDC configured, the workflow authenticates using only a client ID, tenant ID, and subscription ID — no client secret is stored in GitHub at all, which removes an entire class of leaked-secret risk compared to a classic service-principal-with-secret setup.
• Role assignments (typically `Contributor` or a narrower custom role, scoped to the resource group holding the workspace) are still made against that same Entra application or managed identity — OIDC changes *how* the workflow authenticates, not *what* it's authorized to do once authenticated.

Trunk-based development and branch protection

• Short-lived feature branches merged frequently into a protected `main`/`trunk` branch, gated by required PR reviews and passing CI checks, keeps the workspace's actual state (what Bicep and job YAML define) from drifting far from what's deployed.
• A CI workflow typically lints and validates job/pipeline YAML on every pull request, and only a merge to the protected branch triggers the workflow that actually deploys infrastructure or submits a production training job.

Restricting network access

• A workspace can require private endpoint access only (`public_network_access: Disabled`), forcing all traffic — studio, SDK, CLI — through a private link inside a virtual network rather than the public internet.
• Once network-restricted, GitHub-hosted runners (which run outside your virtual network) can no longer reach the workspace directly — this is a common reason a working local `az ml` command fails identically from a GitHub Actions workflow, and is solved with a self-hosted runner placed inside the VNet, or a hosted-runner network peering/tunnel solution.

Common confusion

• Workload identity federation secures *who* can trigger a deployment or job; network restriction secures *where the traffic can come from*. Configuring OIDC correctly doesn't help a workflow reach a network-isolated workspace from a public GitHub-hosted runner — the two controls have to be solved together, not interchangeably.