Skip to main content

Building an AI Agent for Azure Policy Assistant - Part 2

· 6 min read
Hasan Gural

Welcome back, Friend!

In the previous article, we covered setting up the necessary resources for the AI Agent for Azure Policy Governance Assistant. Now, we will proceed with the final steps to complete and test the AI Agent:

  • Create the AI Agent using the Azure AI Foundry UI or PowerShell.
  • Develop and provide an example KQL query for retrieving compliance data.
  • Execute the KQL query and upload the results to the AI Agent’s Knowledge.
  • Test the AI Agent to ensure it accurately responds to policy compliance queries.

Create the AI Agent​

Creating an AI Agent is a straightforward process. You can set it up using either the Azure AI Foundry UI or PowerShell.

First, I will quickly walk you through how to create an AI Agent using the Azure AI Foundry Portal. Then, as part of automating the solution, I will show you how to create the AI Agent using PowerShell for a more scalable and repeatable approach.

Create an AI Agent using the Azure AI Foundry Portal​

Go to the Azure AI Foundry portal and click on the Agents tab. Then, click on the New Agents button to create a new AI Agent. It will generate a new AI Agent with a unique ID. See the screenshot below for reference:

Create AI Agent

I will definitely rename the AI Agent to something more meaningful. You can do this by clicking on the Agent Name and entering a new name for the AI Agent. This helps with better identification and ensures the AI Agent is properly named for the Governance Assistant. Additionally, we can configure the Deployment Model to enforce a specific model for processing queries efficiently.

Rename AI Agent

Create an AI Agent using PowerShell​

I know that sometimes it's easier to create this in the UI, especially for the first time. However, the main goal here is to share a script block that consolidates all the developed scripts into a single automated script. This will allow you to quickly generate a demo environment as part of the article I’ve shared.

You're also more than welcome to improve the script and customize it further to fit your specific needs!


## Create AI Agent in Azure
$SubscriptionId    = "SUBSCRIPTION_ID"
$ResourceGroupName = "RESOURCE_GROUP_NAME"
$WorkspaceName     = "AML_WORKSPACE_NAME"
$ApiVersion        = "2024-12-01-preview"
$AgentName         = "Azure Policy Governance Assistant"
$Model             = "gpt-4o-mini"

# Construct the REST API URL
$ApiUrl = "https://ai.azure.com/api/swedencentral/agents/v1.0/subscriptions/$SubscriptionId/resourceGroups/$ResourceGroupName/providers/Microsoft.MachineLearningServices/workspaces/$accountName/assistants?api-version=$ApiVersion"

# Define the AI Agent JSON payload
$AgentConfig = @{

    name         = $AgentName
    tools        = @()
    model        = $Model
    instructions = "You are designed to analyze Azure governance and policy compliance data.`nYour primary focus is on identifying non-compliant resources and policy exemptions at both the subscription and resource group levels."

} | ConvertTo-Json -Depth 3


# Get Access Token using the correct audience
$AccessToken = (Get-AzAccessToken -ResourceUrl "https://management.azure.com").Token

# Headers for the API request
$Headers = @{
    "Authorization" = "Bearer $AccessToken"
    "Content-Type"  = "application/json"
}

Write-Output "Creating AI Agent in Azure..."
$Response = Invoke-RestMethod -Uri $ApiUrl -Method POST -Headers $Headers -Body $AgentConfig
Write-Output "AI Agent Created Successfully!" -ForegroundColor Green

Output:​

Create AI Agent using PowerShell

We have successfully created the AI Agent using the Azure AI Foundry Portal and PowerShell. Now, let's move on to the next step: developing and providing an example KQL query for retrieving compliance data.

policyresources
| where type == "microsoft.policyinsights/policystates"
| extend 
    complianceState = tostring(properties.complianceState),
    scope = tostring(properties.scope),
    subscriptionId = tostring(split(properties.resourceId, "/")[2]),  // Extract Subscription ID from resourceId
    resourceId = tostring(properties.resourceId),
    policyAssignmentId = tostring(properties.policyAssignmentId),
    rawPolicyDefinitionId = tostring(properties.policyDefinitionId),
    policyDefGuid = tostring(split(tolower(tostring(properties.policyDefinitionId)), "/")[4]),
    scopeType = iif(properties.scope contains "/resourceGroups/", "ResourceGroup", "Subscription")
| where complianceState == "NonCompliant"
| join kind=inner (
    policyresources
    | where type =~ "microsoft.authorization/policydefinitions/versions"
    | extend 
        defId = tostring(id),
        policyDefGuid = tostring(split(tolower(id), "/")[4]),
        policyDisplayName = tostring(properties.displayName),
        policyDescription = tostring(properties.description)
    | project policyDefGuid, policyDisplayName, policyDescription
    )
    on policyDefGuid
| join kind=inner (
    resourcecontainers
    | where type == "microsoft.resources/subscriptions"
    | extend subscriptionId = tostring(split(id, "/")[2])
    | project subscriptionId, subscriptionName = name
    )
    on subscriptionId
| summarize resourceIds = make_list(resourceId)
    by subscriptionName, subscriptionId, policyDisplayName, policyDescription, policyAssignmentId, scopeType
| project subscriptionName, subscriptionId, policyDisplayName, policyDescription, policyAssignmentId, scopeType, resourceIds, complianceState = "NonCompliant"

The KQL query is now ready to fetch the compliance data. Below, you can see screenshots of the compliance data structure. This will help you determine how we can extend the KQL queries to retrieve more relevant insights and feed additional knowledge into the AI Agent over time. Once we have the compliance data, we can dynamically upload the results to the AI Agent's Knowledge Base. This ensures that the AI Agent remains up to date with the latest Azure Policy compliance data.

Compliance Data

By leveraging the Machine Learning Workspace API, we can automate the process of uploading compliance data to the AI Agent’s Knowledge, making governance insights more accurate and efficient. 


# Paste the KQL query into the $queries variable
$queries = @" 
............. KQL Query .............
"@

$FilePath = "AzurePolicyCompliance.json"
$getData  = Search-AzGraph -Query $queries
$getData | ConvertTo-Json -Depth 10 | Out-File -FilePath $FilePath

# Prepare Headers
$Headers = @{
    "Authorization"   = "Bearer $AccessToken"
    "Accept-Encoding" = "gzip, deflate, br, zstd"
}

$FileObject = Get-Item -Path $FilePath

Write-Host "Uploading file as multipart/form-data..." -ForegroundColor Cyan
$Response = Invoke-RestMethod -Uri $ApiUrl -Method Post -Headers $Headers -Form @{
    "file"    = $FileObject
    "purpose" = "assistants"
}

Write-Host "File Uploaded Successfully!" -ForegroundColor Green

$Response

Upload Compliance Data to AI Agent

Seems like we have successfully uploaded the compliance data to the AI Agent's Knowledge Base. To confirm, you can check the Knowledge Base section in the Azure AI Foundry portal.

AI Agent Knowledge Base

So we're good to go! Now, let's test the AI Agent to ensure it accurately responds to policy compliance queries.

Test the AI Agent​

To test the AI Agent, follow these steps:

  1. Go to the AI Foundry Portal
  2. Navigate to → Playgrounds → New Playground.
  3. Select the Agent Playground option.
  4. You are now in the AI Agent environment we have built.
  5. Start testing by asking the following questions:
  • [Analyze the compliance status of all resources in the Visual Studio Enterprise - MVP subscription.]

Test AI Agent

Here we are at the end of the article. I hope you enjoyed the journey of building an AI Agent for Azure Policy Governance Assistant. If you have any questions or feedback, please feel free to reach out to me.