Developing an Azure Function Isolated Process with Service Bus Trigger, Batch Processing, and Blob Storage in .NET 8.0

In this blog post, we'll walk through the process of creating an Azure Function isolated process app using .NET 8.0. Our function will use a Service Bus trigger to process messages in batches and store the results in Azure Blob Storage. This approach is particularly useful for high-volume data processing scenarios where you want to optimize performance and reduce the number of write operations to your storage.

Prerequisites

Before we begin, make sure you have the following:

1. An Azure account with an active subscription
2. Azure Functions Core Tools version 4.x
3. .NET 8.0 SDK installed
4. Visual Studio Code with the Azure Functions extension
5. Azure CLI installed

Step 1: Set up Azure Resources

First, let's create the necessary Azure resources. You can do this using the Azure portal or Azure CLI. Here's an example using Azure CLI:

bash
# Set variables
resourceGroup="myResourceGroup"
location="eastus"
serviceBusNamespace="myServiceBusNamespace"
serviceBusQueue="myQueue"
storageAccount="mystorageaccount"
functionApp="myIsolatedFunctionApp"

# Create Resource Group
az group create --name $resourceGroup --location $location

# Create Service Bus namespace and queue
az servicebus namespace create --name $serviceBusNamespace --resource-group $resourceGroup --location $location
az servicebus queue create --name $serviceBusQueue --namespace-name $serviceBusNamespace --resource-group $resourceGroup

# Create Storage account
az storage account create --name $storageAccount --resource-group $resourceGroup --location $location --sku Standard_LRS

# Create Function App (Isolated process)
az functionapp create --name $functionApp --storage-account $storageAccount --consumption-plan-location $location --resource-group $resourceGroup --runtime dotnet-isolated --runtime-version 8.0 --functions-version 4

Step 2: Create the Function App Project

Now, let's create a new Function App project:

1. Open a terminal and navigate to your desired project directory
2. Run the following command to create a new Function App project:

bash
func init IsolatedServiceBusBatchProcessor --worker-runtime dotnet-isolated --target-framework net8.0
cd IsolatedServiceBusBatchProcessor

3. Add a new function to the project:

bash
func new --name ProcessMessages --template "Azure Service Bus Queue trigger"

Step 3: Update Project File

Update your .csproj file to include the necessary package references and ensure it's targeting .NET 8.0:

xml
<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <AzureFunctionsVersion>v4</AzureFunctionsVersion>
    <OutputType>Exe</OutputType>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.Azure.Functions.Worker" Version="1.20.0" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.ServiceBus" Version="5.14.1" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Sdk" Version="1.16.2" />
    <PackageReference Include="Azure.Storage.Blobs" Version="12.19.1" />
  </ItemGroup>
  <ItemGroup>
    <None Update="host.json">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    </None>
    <None Update="local.settings.json">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
      <CopyToPublishDirectory>Never</CopyToPublishDirectory>
    </None>
  </ItemGroup>
</Project>

Step 4: Implement the Function

Replace the content of your ProcessMessages.cs file with the following code:

csharp
using System;
using System.Text;
using System.Threading.Tasks;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Extensions.Logging;
using Azure.Storage.Blobs;

namespace IsolatedServiceBusBatchProcessor
{
    public class ProcessMessages
    {
        private readonly ILogger _logger;
        private readonly BlobServiceClient _blobServiceClient;

        public ProcessMessages(ILoggerFactory loggerFactory, BlobServiceClient blobServiceClient)
        {
            _logger = loggerFactory.CreateLogger<ProcessMessages>();
            _blobServiceClient = blobServiceClient;
        }

        [Function("ProcessMessages")]
        public async Task Run([ServiceBusTrigger("myqueue", Connection = "ServiceBusConnection")] string[] messages)
        {
            _logger.LogInformation($"ServiceBus queue trigger function processed {messages.Length} messages");

            var processedMessages = new List<string>();

            foreach (var message in messages)
            {
                // Process the message (in this example, we're just adding a timestamp)
                var processedMessage = $"{DateTime.UtcNow:yyyy-MM-dd HH:mm:ss}: {message}";
                processedMessages.Add(processedMessage);
            }

            // Combine all processed messages into a single string
            var batchContent = string.Join(Environment.NewLine, processedMessages);

            // Generate a unique file name
            var fileName = $"batch-{Guid.NewGuid()}.txt";

            // Get a reference to the container
            var containerClient = _blobServiceClient.GetBlobContainerClient("processed-messages");

            // Create the container if it doesn't exist
            await containerClient.CreateIfNotExistsAsync();

            // Get a reference to the blob
            var blobClient = containerClient.GetBlobClient(fileName);

            // Upload the batch content to the blob
            using (var ms = new MemoryStream(Encoding.UTF8.GetBytes(batchContent)))
            {
                await blobClient.UploadAsync(ms, overwrite: true);
            }

            _logger.LogInformation($"Batch uploaded to blob storage: {fileName}");
        }
    }
}

This code does the following:

1. Processes each message in the batch by adding a timestamp.
2. Combines all processed messages into a single string.
3. Generates a unique file name for the batch.
4. Creates a blob container if it doesn't exist.
5. Uploads the batch content to a new blob in the container.

Step 5: Configure Dependency Injection

In the isolated process model, we need to configure dependency injection in the Program.cs file. Replace the content of Program.cs with the following:

csharp
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.DependencyInjection;
using Azure.Storage.Blobs;

var host = new HostBuilder()
    .ConfigureFunctionsWorkerDefaults()
    .ConfigureServices(services =>
    {
        services.AddSingleton(sp =>
        {
            var configuration = sp.GetRequiredService<IConfiguration>();
            var storageConnectionString = configuration["AzureWebJobsStorage"];
            return new BlobServiceClient(storageConnectionString);
        });
    })
    .Build();

host.Run();

This code adds the BlobServiceClient to the dependency injection container, using the connection string from the AzureWebJobsStorage app setting.

Step 6: Configure Application Settings

Make sure your local.settings.json file includes the following settings:

json
{
  "IsEncrypted": false,
  "Values": {
    "AzureWebJobsStorage": "YOUR_STORAGE_ACCOUNT_CONNECTION_STRING",
    "FUNCTIONS_WORKER_RUNTIME": "dotnet-isolated",
    "ServiceBusConnection": "YOUR_SERVICE_BUS_CONNECTION_STRING"
  }
}

Replace YOUR_STORAGE_ACCOUNT_CONNECTION_STRING and YOUR_SERVICE_BUS_CONNECTION_STRING with the actual connection strings for your Azure Storage account and Service Bus namespace.

Step 7: Test and Deploy

You can now test your function locally using the Azure Functions Core Tools:

bash
func start

To deploy your function to Azure, you can use the Azure Functions extension in Visual Studio Code or the Azure CLI:

bash
func azure functionapp publish $functionApp

Conclusion

In this blog post, we've created an Azure Function isolated process app using .NET 8.0 that uses a Service Bus trigger to process messages in batches and store the results in Azure Blob Storage. This approach can significantly improve performance for high-volume data processing scenarios by reducing the number of write operations to storage.

Some key takeaways:

1. The isolated process model in Azure Functions provides better performance and scalability for .NET applications.
2. Using batch processing with Service Bus can improve the efficiency of your Azure Functions.
3. The BlobServiceClient provides an easy way to interact with Azure Blob Storage.
4. Dependency injection in isolated process Azure Functions allows for better separation of concerns and testability.

Remember to monitor your function's performance and adjust the batch size and other parameters as needed for your specific use case.

Happy coding!