Introducing Azure DevOps ID Token Refresh and Terraform Task Version 5

We are excited to share some recent updates that improve the experience of using Workload identity federation (OpenID Connect) with Azure DevOps and Terraform on Microsoft Azure.

Many working parts have come together to make this possible and we’ll share those here.

We are also very pleased to announce version 5 of the Microsoft DevLabs Terraform Task, which supports ID Token refresh by default.

What is ID Token Refresh?

Workload identity federation requires an ID Token issued from the identity provider, in our case Azure DevOps. This ID Token has a short lifespan of ~5 minutes by design. It is immediately exchanged for an access token, which has a longer lifespan and is used to authenticate to Microsoft Azure. You can read more here.

However, there are many cases where you can have breaks or long running jobs that may need to request a new access token. In this scenario a new ID Token is required as the old one will have expired and cannot be used. ID Token Refresh is the capability of the running task to request a new ID Token and exchange it for an access token when it knows the previous token has expired.

An error you may see when a token times out without ID Token Refresh is AADSTS700024: Client assertion is not within its valid time range.

An example scenario is running Terraform init using CLI args for the backend. These would get cached in the plan file and then if there is a delay in approving the apply, it would result in a timeout error since the ID Token was cached in the plan file. Terraform test for integration testing is another scenario where you would historically see token time outs.

You can read more about the SDK implementation here.

What has changed to support ID Token Refresh?

In Terraform, the provider is responsible for the authentication to Azure. As such it is the providers responsibility to refresh credentials when it needs them. Therefore every Azure provider had to be updated to support ID Token Refresh. This includes the following providers:

• azurerm
• azapi
• azuread

There is also the azurerm backend for remote state, which is part of the core Terraform CLI, which also needed to be updated to support ID Token Refresh.

Announcing Microsoft DevLabs Terraform Task Version 5

We are very pleased to announce a new version of the Terraform Tasks that uses ID Token Refresh by default when working with a Workload identity federation service connection. This task abstracts away the need for you to understand the settings, but we share those later in the article. You can find the updated task here.

We have also updated the documentation for the task with YAML based examples.

Here is an example of using the new task with ID Token Refresh:

- task: TerraformTask@5
  displayName: Run Terraform Init
  inputs:
    provider: 'azurerm'
    command: 'init'
    backendServiceArm: 'your-service-connection'
    backendAzureRmStorageAccountName: 'your-stg-name'
    backendAzureRmContainerName: 'your-container-name'
    backendAzureRmKey: 'state.tfstate'

- task: TerraformTask@5
  name: terraformPlan
  displayName: Run Terraform Plan
  inputs:
    provider: 'azurerm'
    command: 'plan'
    commandOptions: '-out tfplan'
    environmentServiceNameAzureRM: 'your-service-connection'

# Only runs if the 'terraformPlan' task has detected changes the in state.
- task: TerraformTask@5
  displayName: Run Terraform Apply
  condition: and(succeeded(), eq(variables['terraformPlan.changesPresent'], 'true'))
  inputs:
    provider: 'azurerm'
    command: 'apply'
    commandOptions: 'tfplan'
    environmentServiceNameAzureRM: 'your-service-connection'

How is ID Token Refresh configured?

This section covers using ID Token Refresh with the Azure CLI task in case you are not using the Microsoft DevLabs Terraform task.

There are three settings with common environment variables implemented across the providers and backend that are required:

• ARM_OIDC_AZURE_SERVICE_CONNECTION_ID: The service connection ID
• ARM_OIDC_REQUEST_URL: The API endpoint used to request the ID Token (this uses SYSTEM_OIDCREQUESTURI by default)
• ARM_OIDC_REQUEST_TOKEN: The token used to request the ID Token (this is the System.AccessToken in Azure DevOps)

ARM_OIDC_AZURE_SERVICE_CONNECTION_ID

Each of the providers and the backend now support a variable for supplying the Azure DevOps Service Connection ID. This is required for ID Token Refresh to know which service connection to request an ID Token for with the relevant subject claims.

ARM_OIDC_REQUEST_URL

Azure DevOps exposes an endpoint that allows an API call to generate an ID Token with the subject claims needed for Workload identity federation. This endpoint uses the service connection context as supplied by the service connection ID. This endpoint is required for ID token refresh to request a new ID Token on expiry.

Azure DevOps exposes this URL in an environment variable called SYSTEM_OIDCREQUESTURI. Each of the providers and backend look for this environment variable and use it by default. This means we don’t actually need to supply ARM_OIDC_REQUEST_URL explicitly, as you can see in the example below.

ARM_OIDC_REQUEST_TOKEN

The ID Token request URL requires authentication with a token. This token is automatically generated in the context of the Azure DevOps pipeline. The token can be found in the built in variable called System.AccessToken. System.AccessToken is not set as an environment variable by default per the docs, so you need to explicitly set that to the ARM_OIDC_REQUEST_TOKEN environment variable.

NOTE: We plan to support setting the environment variable SYSTEM_ACCESSTOKEN as an alternative in future iterations.

Example usage

Here is a basic example of configuring it with the Azure CLI task:

- task: AzureCLI@2
    displayName: 'Terraform'
    inputs:
      azureSubscription: my-service-connection
      scriptType: pscore
      scriptLocation: inlineScript
      inlineScript: |
        $subscriptionId = $(az account show --query id -o tsv)

        $env:ARM_TENANT_ID = $env:AZURESUBSCRIPTION_TENANT_ID
        $env:ARM_SUBSCRIPTION_ID = $subscriptionId
        $env:ARM_CLIENT_ID =  $env:AZURESUBSCRIPTION_CLIENT_ID
        $env:ARM_OIDC_AZURE_SERVICE_CONNECTION_ID = $env:AZURESUBSCRIPTION_SERVICE_CONNECTION_ID
        $env:ARM_USE_OIDC = "true"

        terraform init `
          -backend-config="storage_account_name=my-storage-accont" `
          -backend-config="container_name=my-container" `
          -backend-config="key=terraform.tfstate" `
          -backend-config="use_azuread_auth=true"

        terraform plan -out tfplan
        terraform apply tfplan
    env:
      ARM_OIDC_REQUEST_TOKEN: $(System.AccessToken)

For more comprehensive examples, see the templates in our code sample .

What next?

Please try it out and give us feedback.

We are already working on some improvements to reduce the number of environment variables you need to supply, but please let us know what else you’d like to see.

Recognition

This has been a huge effort across multiple teams. In particular we’d like to thank:

• Zhaoting Weng (Microsoft) for updating all the providers and backend
• Eric van Wijk (Microsoft) and Grzegorz Gurgul (Microsoft) for guidance and updating the Azure CLI task
• Manuel Ericstam (Solidify) for guidance and publishing the Terraform Task

The post Introducing Azure DevOps ID Token Refresh and Terraform Task Version 5 appeared first on Azure DevOps Blog.