Deploy Terraform using Azure DevOps YAML Pipelines

HashiCorp Terraform is a popular tool for managing infrastructure as code (IaC). By defining your IaC using Terraform, you can use version control with your infrastructure configuration and also automate infrastructure deployment in a consistent and repeatable way. Azure DevOps Pipelines can be used to setup YAML pipelines to instrument the Terraform infrastructure deployments using the traditional Terraform Plan and Apply workflow. This article will show you how to set up two separate YAML pipelines in Azure DevOps to implement the Plan and Apply workflow for your Terraform infrastructure deployments.

[toc]

How the Terraform Plan and Apply Workflow will be set up in Azure DevOps

With the usual Terraform workflow, the plan command is first run, then the apply command is run. This is the normal Terraform workflow process. However, here’s a process flow outline of the steps necessary for the Terraform workflow along with the places where the YAML pipelines shown in this article fit in.

1. The DevOps Engineer or Site Reliability Engineer (SRE) commits Terraform code changes to the git repository within Azure DevOps.
2. The Terraform Plan (defined in this article) will trigger automatically to run the terraform plan command and generate a Terraform Plan (.tfplan) file that will get stored in the build artifacts for the pipeline.
3. The DevOps Engineer or SRE will then perform the manual task of inspecting the Azure DevOps Pipeline output and review the generated Terraform plan.
   
4. Once the proposed Terraform plan of infrastructure changes to make has been approved by the DevOps Engineer or SRE, they will manually trigger the Terraform Apply pipeline to run.
5. The Terraform Apply pipeline will pull in the Terraform plan file (.tfplan) stored within the build artifacts of the most recent run of the Terraform Plan pipeline, then it will execute the plan and make all the necessary infrastructure changes.
   
6. When ever Terraform configuration changes are needed to update / modify the infrastructure deployment, this process repeats again.

This Terraform workflow of the plan and apply steps could be implemented using a single Azure DevOps YAML Pipeline. However, then the pipeline would automatically apply after performing the plan step. This would not be ideal as it’s very important to inspect the Terraform plan before applying. If you skip this review step then it’s possible for unintended or breaking changes to be made to the infrastructure deployment. It’s best practice to always review the Terraform plan before applying it.

Now, let’s look at the YAML code for the Azure DevOps Pipelines to implement multi-pipeline strategy for managing HashiCorp Terraform deployments using Azure DevOps!

Define Terraform Plan Azure DevOps YAML Pipeline

The following is the YAML for the Azure DevOps Pipeline for performing the Terraform Plan step of the workflow:

name: "Terraform Plan"

trigger:
  branches:
    include:
      - main

pool:
  vmImage: 'ubuntu-latest'

steps:

- task: TerraformInstaller@0
  displayName: "Install Terraform"
  inputs:
    terraformVersion: '1.3.9'
    terraformDownloadLocation: 'https://releases.hashicorp.com/terraform'

- script: |
    terraform init
    terraform plan -out=terraform.tfplan
  displayName: 'Terraform Plan'
  
- task: PublishBuildArtifacts@1
  inputs:
    pathtoPublish: '$(Build.SourcesDirectory)/terraform.tfplan'
    artifactName: 'terraformPlan'
    publishLocation: 'Container'

To implement these pipelines according to what’s laid out in this article, be sure to name this first YAML pipeline Terraform Plan. This name is important, as the next YAML pipeline will reference it by name.

Notice the trigger defined on this pipeline. It is configured to automatically trigger this pipeline on pushes to the main branch of the repository. This will help to automatically generate a Terraform Plan each time changes are pushed to the branch. Since the terraform plan command doesn’t make any environment changes, this is perfectly save to do. However, you may want to configure a different trigger if that fits better with your teams project release goals.

Also notice that the Terraform Plan file is named terraform.tfplan and the artifact name is set to terraformPlan. These are important values, as they will be used in the Terraform apply pipeline later to reference the correct artifacts when downloading / loading the plan file in that step of the Terraform workflow.

This Terraform Plan YAML pipeline contains the following steps to build the Terraform plan and save it in the build artifacts for the pipeline:

1. The pool.vmImage for the YAML defines the use of Ubuntu Linux for the operating system of the build agent. This is done with the value of ubuntu-latest to specify the latest Ubuntu image version available.
2. The TerraformInstaller@0 task is used to download and setup HashiCorp Terraform for use on the Azure DevOps build agent machine. It is also specifying the Terraform version required for the Terraform code in the repository. Be sure to set this to the version you’re standardizing your Terraform deployments on.
3. The script task is used to call both the terraform init and terraform plan commands to generate the Terraform plan file (.tfplan) for the infrastructure configuration in the repository.
4. The PublishBuildArtifacts@1 task is used to publish the Terraform plan file (.tfplan) to the build artifacts for this pipeline. This is important, as this is how the Terraform apply pipeline will be able to access the previously generated plan from this pipeline.

Once the Terraform Plan YAML pipeline is set up in your Azure DevOps project, the next item to set up is the Terraform Apply pipeline.

Define Terraform Apply Azure DevOps YAML Pipeline

The following is the YAML for the Azure DevOps Pipeline for performing the Terraform Apply step of the workflow:

name: "Terraform Apply"

trigger: none

pool:
  vmImage: 'ubuntu-latest'

steps:
- task: DownloadPipelineArtifact@2
  displayName: 'Download Terraform Plan'
  inputs:
    buildType: specific
    buildVersionToDownload: 'latest'
    project: 'TFYAML' # replace with the name of your Azure DevOps Project
    definition: 'Terraform Plan'
    artifactName: 'terraformPlan'
    path: '$(Pipeline.Workspace)'

- task: TerraformInstaller@0
  displayName: "Install Terraform"
  inputs:
    terraformVersion: '1.3.9'
    terraformDownloadLocation: 'https://releases.hashicorp.com/terraform'

- script: |
    terraform init
    terraform apply $(Pipeline.Workspace)/terraform.tfplan
  displayName: 'Terraform Apply'

The Terraform Apply pipeline is setup similarly to the Terraform Plan pipeline, but has a couple differences necessary for applying the Terraform infrastructure changes based on the previously generated Terraform plan.

This Terraform Apply YAML pipeline contains the following steps to pull in the previously generated Terraform Plan file (.tfplan) and then use terraform apply command to make the planned changes to the infrastructure deployment:

1. The DownloadPipelineArtifact@2 task is used to download the Terraform Plan file (.tfplan) stored in the build artifacts of the Terraform Plan pipeline so it can be used within this pipeline.
   Be sure the following arguments of the DownloadPipelineArtifact@2 task are set correctly as follows:
   • project: This needs to be set to the name of the Azure DevOps Project where the Terraform Plan pipeline is run. In my example, it’s TFYAML but will be different for your configuration of Azure DevOps.
   • definition: This is the name of the Terraform Plan pipeline. If you named your instance of the pipeline differently, then this must be set to the name you used..
   • artifactName: The Terraform Plan pipeline code example stores the plan file (.tfplan) in a build artifact named terraformPlan. If you change the code to use a different artifact name, then this will need to be set accordingly.
2. The TerraformInstaller@0 task is used to download and setup HashiCorp Terraform for use on the Azure DevOps build agent machine. It is also specifying the Terraform version required for the Terraform code in the repository. Be sure to set this to the Terraform version in this pipeline to the same version used in the Terraform Plan pipeline.
3. The script task runs both the terraform init and terraform apply commands to perform the proposed changes from the Terraform plan and modify the infrastructure deployment to match.

Happy Terraforming! I hope this article helps you get your Azure DevOps Pipelines set up correctly to be able to perform Terraform infrastructure as code (IaC) deployments efficiently.