Study Guides/AI-300/Design and Implement a GenAIOps Infrastructure

20-25% of exam

Design and Implement a GenAIOps Infrastructure

Configure Microsoft Foundry projects, identities, and network isolation, deploy foundation models for production, and version prompts with Git.

Design and Implement an MLOps Infrastructure Implement Machine Learning Model Lifecycle and Operations Design and Implement a GenAIOps Infrastructure Implement Generative AI Quality Assurance and Observability Optimize Generative AI Systems and Model Performance

Foundry resources, projects, managed identities, and RBAC

A Foundry resource is the Azure-billed account that hosts one or more projects — getting its identity and role assignments right up front avoids re-architecting access control after agents and evaluations are already in production.

Resources and projects

The Foundry resource (an `Microsoft.CognitiveServices/accounts` resource under the hood) is what carries region, networking, and billing; a project inside it scopes a specific team or workload's agents, deployments, and connections.
A Basic agent project uses platform-managed data resources (storage, Cosmos DB, AI Search provisioned automatically); a Standard agent project instead brings your own Storage, Cosmos DB, and AI Search resources — the right choice when data residency, existing infrastructure, or tighter control over those dependencies matters.

Managed identities

The Foundry resource's system-assigned managed identity is what it uses to reach dependent resources (Storage, Cosmos DB, AI Search) without embedded credentials, the same pattern as an Azure Machine Learning workspace.
When network isolation is enabled, that managed identity additionally needs the built-in Azure AI Enterprise Network Connection Approver role (role ID `b556d68e-0be0-4f35-a333-ad7ee1ce17ea`) assigned at the resource group (or subscription) scope, so it can auto-approve the private endpoints its managed network creates.

RBAC roles

Foundry Owner / Foundry Account Owner (recently renamed from Azure AI Owner / Azure AI Account Owner — both names still appear during the rollout): full control including role assignment, typically held by a small platform/IT admin group.
Foundry User (formerly Azure AI User, sometimes still called Azure AI Developer in hub-based docs): the working role for someone building and deploying agents and model connections day to day, without subscription-wide administrative rights.
Narrower, model-specific roles like Cognitive Services OpenAI User (inference calls and viewing deployments only) versus Cognitive Services OpenAI Contributor (can also create/edit deployments and fine-tune) let you separate "can call this model" from "can change what's deployed," independent of the broader Foundry-level roles.
A typical enterprise layout assigns Owner to IT admins at the hub/resource scope, Foundry User (or Contributor) to team leads who create projects, and a narrower per-project role to individual developers — mirroring the same least-privilege-at-the-right-scope principle as Azure RBAC generally.

Common confusion

The Foundry-native roles (Foundry Owner/User) and the Cognitive Services roles (Cognitive Services OpenAI User/Contributor) overlap in purpose but aren't interchangeable — a role assignment made through the classic Cognitive Services resource RBAC doesn't automatically show up as an equivalent Foundry role, and some scenarios (like Entra ID inference calls) specifically require one family or the other.

Network security and private networking

Foundry supports two distinct approaches to network isolation, and the exam expects you to know which problem each one solves rather than treating them as interchangeable "make it private" checkboxes.

Managed virtual network

A managed virtual network is provisioned and operated by Microsoft on your behalf — you choose an isolation mode (`AllowInternetOutbound`, permitting general outbound internet access, or `AllowOnlyApprovedOutbound`, restricting egress to explicitly approved destinations) and Foundry builds the network plumbing for you.
It's created via Bicep/Terraform templates, `az rest`, or the newer `az cognitiveservices account managed-network create` command — notably, the Azure portal UI does not currently support creating it, so this is CLI/IaC territory even for a first deployment.
Once enabled, it cannot be disabled and there's no upgrade path from a custom (BYO) VNet setup to a managed one — the resource has to be redeployed from scratch, which makes this a decision to get right at initial provisioning rather than something to reconfigure later.

Bring-your-own virtual network

BYO VNet injection places the Foundry resource's networking inside a virtual network you already control, giving you direct control over route tables, NSGs, and peering (including a hub-and-spoke design with a centralized firewall inspecting egress).
Inbound isolation (a private endpoint so the resource itself isn't reachable from the public internet) and outbound isolation (routing agent/evaluation traffic through your VNet) are configured somewhat independently — a Standard agent project with BYO VNet is the template to reach for when you need full control over both directions plus your own Storage/Cosmos DB/AI Search.

Agent tool connectivity under network isolation

Tools that use Microsoft's backbone network (Code Interpreter, function calling) need no extra networking configuration even when the Foundry resource is network-isolated.
Tools reaching public endpoints (Bing grounding, web search, SharePoint) still work without private endpoints, but only because their traffic stays on the public internet — an organization that wants to block that public egress does so with Azure Policy, not by relying on the Foundry network configuration alone.
Azure AI Search used as a private grounding source needs its own private endpoint set up separately (Foundry doesn't auto-create it), and any indexer feeding that search index must set `executionEnvironment: "Private"` explicitly — otherwise it silently defaults to multitenant execution that can't cross the private endpoint, producing a quietly empty index rather than an obvious error.

Common confusion

Enabling network isolation on the Foundry resource doesn't automatically secure every dependency it talks to — Azure AI Search, Storage, and Cosmos DB each need their own private endpoints created separately; the managed or BYO VNet controls the Foundry resource's own boundary, not every downstream service's.

Deploying foundation models for production

Foundry's model catalog offers two fundamentally different ways to put a model into production, and picking between them (and then between deployment types within the chosen path) is a recurring exam scenario.

Serverless API vs. managed compute

Serverless API is the default, preferred path for essentially all Foundry Models — Azure OpenAI models and select partner/community models — where Microsoft hosts the inference infrastructure entirely; you never provision or size a VM, and billing is per-token (or per-PTU, covered below).
Managed compute is reserved for open-source, partner, and custom models (including NVIDIA NIM and other industry-specific models) that need to run on dedicated GPU capacity Foundry manages on your behalf — billing here is hourly per accelerator SKU rather than per-token, and it's the option when a model simply isn't offered as a serverless API.
Both support private networking and keyless (Microsoft Entra ID) authentication; only serverless API currently offers built-in, customizable content filtering and the full set of regional/data-zone/global data-processing choices.

Deployment types within serverless API

Standard (pay-per-token): the default, best for development, testing, and variable or unpredictable production traffic — no capacity is reserved, so throughput can vary with overall platform demand.
Global / Data Zone / Regional variants of Standard trade off where data is processed (worldwide, a specific data zone like US/EU/APAC, or a single region) against latency and compliance requirements — a data-residency requirement is what pushes you toward a non-Global option even though Global is typically cheaper and has faster queueing.
Batch: discounted, asynchronous, no latency SLA — fits bulk offline scoring workloads, not interactive agent traffic.

Provisioned throughput units (PTUs)

A PTU reserves a fixed amount of model-processing capacity exclusively for your deployment, billed per PTU-hour whether or not you actually send traffic — the trade you're making is guaranteed, predictable low latency in exchange for paying for idle reserved capacity.
PTU quota is model-independent (any supported model can use the same PTU pool) but region- and deployment-type-specific, and each model has its own minimum PTU count and its own PTU-to-tokens-per-minute ratio — a heavier model needs more PTUs to hit the same throughput as a lighter one.
`az cognitiveservices account deployment create --sku-name GlobalProvisionedManaged --sku-capacity 50 ...` creates a 50-PTU provisioned deployment; the right trigger for reaching for PTUs is predictable, high-volume, latency-sensitive production traffic — not development or bursty, hard-to-forecast usage, where Standard's per-token billing is more efficient.

Common confusion

Choosing "provisioned" doesn't just change how you're billed — it changes the latency profile itself. A Standard deployment has no latency SLA at all (it shares capacity and can slow down under platform-wide demand); only provisioned and priority-processing deployment types come with a defined latency target per model, which is the actual reason mission-critical workloads use them, not just the cost-at-scale argument.

Prompt versioning and management with Git

Treating a prompt as a first-class, versioned artifact — reviewed, diffed, and rolled back exactly like application code — is what keeps a GenAI application's behavior auditable as it evolves, instead of "the prompt currently in production" being whatever someone last pasted into a portal text box.

Prompts as version-controlled assets

Storing prompts as files in a Git repository (rather than only inside a portal UI) means every change goes through the same pull-request review, diff, and history that application code does — a regression in agent behavior can be traced to the exact commit that changed the prompt.
A repository structure typically separates prompt templates from the code that calls them, so a prompt change doesn't require a full application redeploy, and so multiple agents or environments can reference different prompt versions from the same repo.

Variants and comparison

A prompt *variant* is an alternative phrasing, instruction set, or few-shot example set tested against the same task — keeping variants side by side in version control (rather than only as ephemeral playground experiments) is what makes a later "why did we choose this wording" question answerable.
Comparing variants against the same evaluation dataset (see the quality-assurance topic) closes the loop: a prompt change is proposed as a PR, evaluated against quality/cost/performance metrics, and only merged once its results are at least as good as the version it replaces.

Safe, staged promotion

The same trunk-based, PR-gated workflow used for infrastructure code applies here: a new prompt version is developed on a branch, evaluated automatically (often via a GitHub Actions step that runs the evaluation SDK against the changed prompt), and only promoted to the environment agents actually call after that check passes.
Because prompts are just files, rolling back a bad prompt is a Git revert — no redeployment of the underlying model deployment or agent infrastructure is required, which is precisely why keeping prompts out of hardcoded application strings matters operationally, not just for cleanliness.

Common confusion

Versioning a prompt in Git controls *what text is sent to the model*; it says nothing on its own about *whether a given version is actually better*. Git-based prompt management and automated evaluation are complementary practices — one gives you a safe way to change and roll back prompts, the other gives you evidence for whether a specific change was an improvement.

Back to AI-300 Overview Practice Questions