Containerized Activities in Durable Workflows - Part 3

This is the third part in a series about how we can build a serverless workflow using Azure Durable Functions, but implement some of the activities in that workflow using containers with Azure Container Instances. Today we'll look at creating an Event Grid subscription and an Azure Function triggered by an event.

Current table of contents:

• Part 1: Introduction - What are we building and why do we need it?
• Part 2: Creating infrastructure with the Azure CLI
• Part 3 (this post): Creating an Event Grid Subscription
• Part 4: Creating Container Instances with the Azure .NET SDK
• Part 5: Creating ACI Instances in a Durable Functions Workflow

Azure Event Grid is a service that acts as a centralized hub that can route events raised by any Azure services (or your own custom events) to any number of subscribers. Many Azure services already publish events to Event Grid, so if you want to know when a deployment completes or when a new blob is uploaded to a storage container, you can use Event Grid to be notified. It uses a push model rather than polling like you'd need to with Service Bus, so Event Grid will call you when an interesting event happens (and can retry with back-off if necessary).

Creating an Event Grid Triggered Azure Function

To support Event Grid triggered functions in Azure Functions, we need to reference the Microsoft.Azure.WebJobs.Extensions.EventGrid NuGet package. Simply add this line to your .csproj file.

<PackageReference Include="Microsoft.Azure.WebJobs.Extensions.EventGrid" 
    Version="2.0.0" />

Then creating an Event Grid triggered function is really easy. You just need to apply the EventGridTrigger attribute to an EventGridEvent parameter and it, and it will be called whenever an event fires. In this example, I'm simply logging out interesting information about the events received. In my case I'm particularly interested in events with the Microsoft.ContainerInstance resource provider. Subscriptions can be filtered so you only receive event types you are interested in.

[FunctionName("AciMonitor")]
public static void Run([EventGridTrigger]EventGridEvent eventGridEvent, 
                       ILogger log)
{
    log.LogInformation($"EVENT: {eventGridEvent.EventType}-{eventGridEvent.Subject}-{eventGridEvent.Topic}");
    log.LogInformation(eventGridEvent.Data.ToString());
    
    // some example properties on data:
    // "resourceProvider": "Microsoft.ContainerInstance"
    // "status": "Succeeded"
    // "resourceUri": "/subscriptions/my-sub-id/resourceGroups/DurableFunctionsAciContainers/providers/Microsoft.ContainerInstance/containerGroups/markacitest1",
    dynamic data = eventGridEvent.Data;
    if (data.operationName == "Microsoft.ContainerInstance/containerGroups/delete")
    {
        log.LogInformation($"Deleted container group {data.resourceUri} with status {data.status}");
    }
    else if (data.operationName == "Microsoft.ContainerInstance/containerGroups/write")
    {
        log.LogInformation($"Created or updated container group {data.resourceUri} with status {data.status}");
    }
}

Generate the endpoint address

Now you might be wondering how we connect up an Event Grid subscription to this function. Well, we simply need to create an Event Grid subscription, specifying what we are interested in receiving, and giving it the URL of this function. Where it gets complicated is that the URL needs to include a secret code that is astonishingly hard to automate the retrieval of. Hopefully this is something the Azure Functions team will improve in the future (it does seem like some work is being done in this area)

We need to get several different keys, so we'll use some PowerShell functions I discuss in this article

First a function that can get the Kudu credentials for a Function App

function getKuduCreds($appName, $resourceGroup)
{
    $user = az webapp deployment list-publishing-profiles `
            -n $appName -g $resourceGroup `
            --query "[?publishMethod=='MSDeploy'].userName" -o tsv

    $pass = az webapp deployment list-publishing-profiles `
            -n $appName -g $resourceGroup `
            --query "[?publishMethod=='MSDeploy'].userPWD" -o tsv

    $pair = "$($user):$($pass)"
    $bytes = [System.Text.Encoding]::ASCII.GetBytes($pair)
    $encodedCreds = [System.Convert]::ToBase64String($bytes)
    return $encodedCreds
}

Second, we need a function that can get the "master" function key given the Kudu creds

function getMasterFunctionKey([string]$appName, [string]$encodedCreds)
{
    $jwt = Invoke-RestMethod `
            -Uri "https://$appName.scm.azurewebsites.net/api/functions/admin/token" `
            -Headers @{Authorization=("Basic {0}" -f $encodedCreds)} -Method GET

    $keys = Invoke-RestMethod -Method GET `
            -Headers @{Authorization=("Bearer {0}" -f $jwt)} `
            -Uri "https://$appName.azurewebsites.net/admin/host/systemkeys/_master" 

    # n.b. Key Management API documentation currently doesn't explain how to get master key correctly
    # https://github.com/Azure/azure-functions-host/wiki/Key-management-API
    # https://$appName.azurewebsites.net/admin/host/keys/_master = does NOT return master key
    # https://$appName.azurewebsites.net/admin/host/systemkeys/_master = does return master key

    return $keys.value
}

Now let's use these functions to get the master key for our Function App:

$kuduCreds = getKuduCreds $functionAppName $resourceGroup
$masterKey = getMasterFunctionKey $functionAppName $kuduCreds

But we're still not done! Now we need to get yet another key - this one is the Event Grid "extension key". We can use the master key we just retrieved to get the extension key:

$extensionKeyUri = "https://$functionAppName.azurewebsites.net/admin/host/systemkeys/eventgrid_extension?code=$masterKey"
$extensionKey = (Invoke-RestMethod -Method GET -Uri $extensionKeyUri).value

Next, we generate the URL of the webhook that Event Grid will use to trigger our Azure Function. You can read about the format of that URL here, but basically we need the name of our function ("AciMonitor" in our case) and the extension key that we just retrieved:

$functionName = "AciMonitor"
$functionUrl = "https://$hostName/runtime/webhooks/EventGrid?functionName=$functionName" + "&code=$extensionKey" 

This function URL is the one that Event Grid will use to report events, but it also uses it to confirm the subscription. That's needed because Event Grid is a push model - it calls your API, and so it needs to check that the endpoint it is calling really does want to receive events. The Azure Functions Event Grid extension already knows how to respond to the validation handshake so you don't need to do anything additional on the Function App side.

However, there is one more pitfall here, and I think it's to do with using the Azure CLI from PowerShell. And that is that we need to escape the & character in the URL to successfully pass it to the Azure CLI command that creates the Event Grid subscription. We do that like this:

$functionUrlEscaped = $functionUrl.Replace("&", "^^^&")

Create the Event Grid Subscription

Finally, we're ready to actually create our subscription. We need to ensure we've registered the Microsoft.EventGrid provider for this subscription, which we can do with the following command:

az provider register -n "Microsoft.EventGrid" 

Now we just need to get hold of the subscription id and the resource id of the resource group we are monitoring for events, and we can create an Event Grid subscription with az eventgrid event-subscription create method, passing in the (escaped) endpoint URL we just constructed. I've asked for all event types in this example, as I want to explore what events are available, but you can filter it down to just the event types you're interested in if you want

$subscriptionId = az account show --query id -o tsv
$resourceId = "/subscriptions/$subscriptionId/resourcegroups/$aciResourceGroup"
az eventgrid event-subscription create --name "AciEvents" `
    --source-resource-id $resourceId `
    --endpoint-type "WebHook" --included-event-types "All" `
    --endpoint $functionUrlEscaped

Note that for this command to execute successfully our Azure Function App needs to be running so it can respond to the validation handshake.

By the way, the Event Grid subscription does not live in a resource group, so if you want to delete it, then you can use the following command:

az eventgrid event-subscription delete --name "AciEvents" `
    --source-resource-id $resourceId

Summary

In this post we saw how we can create an Event Grid subscription that sends events about a particular resource group to an Azure Function. There were a lot of hoops to jump through to get the URL we needed to subscribe - I'm hoping that becomes much easier to automate in the future. But next up in part 4, let's learn how we can use the fluent Azure C# SDK to create ACI container groups from inside our Azure Function App.