Use a Terraform template in Gitlab Pipeline

Overview

In GitLab, templates can be used in pipelines to simplify the process of creating a pipeline. Templates are essentially predefined scripts that can be reused across multiple pipelines, making it easier to maintain and manage pipelines across an organization.

To use templates in a GitLab pipeline, follow these steps:

Define a template: First, you need to create a YAML file with the template definition. This file should contain the script or job that you want to reuse in your pipeline.
Configure the pipeline: Once you have defined the template, you can configure your pipeline to use it. To do this, you need to include the template reference in your pipeline definition. You can do this by using the include keyword and specifying the path to the template file.

Terraform Template

In order to use a template for performing basic Terraform steps, lets first create a templates project. In the repository create a new file called terraform.yml and add the following code:

variables:
  TERRAFORM_IMAGE_VERSION: "0.13.5"
  TERRAFORM_FOLDER: "./"
  TERRAFORM_JSON_PLAN_FILE: tfplan.json
  TERRAFORM_PLAN: plan.tfplan
  TERRAFORM_PLAN_EXTRA_VARS: ""
  TERRAFORM_WORKSPACE: ${INVENTORY_CODE}_${ENVIRONMENT}_${REGION}

.terraform_image:
  image:
    name: hashicorp/terraform:$TERRAFORM_IMAGE_VERSION
    entrypoint:
      - '/usr/bin/env'
      - 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'

.terraform_cache:
  cache:
    paths:
      - .terraform

.terraform_init:
  before_script:
    - cd $TERRAFORM_FOLDER
    - terraform --version
    - terraform init
    - terraform workspace select $TERRAFORM_WORKSPACE || terraform workspace new $TERRAFORM_WORKSPACE

.terraform_base:
  extends:
    - .terraform_image
    - .terraform_cache
    - .terraform_init

.terraform_validate:
  extends: .terraform_base
  script:
    - terraform validate
  variables:
    TERRAFORM_WORKSPACE: "default"

.terraform_plan:
  extends: .terraform_base
  script:
    - terraform plan $TERRAFORM_PLAN_EXTRA_VARS -var-file=$ENVIRONMENT/$REGION/terraform.tfvars -out=$TERRAFORM_WORKSPACE-$TERRAFORM_PLAN
    - terraform show --json $TERRAFORM_WORKSPACE-$TERRAFORM_PLAN > $TERRAFORM_WORKSPACE-$TERRAFORM_JSON_PLAN_FILE
  artifacts:
    paths:
      - $TERRAFORM_FOLDER/$TERRAFORM_WORKSPACE-$TERRAFORM_PLAN
    reports:
      terraform: $TERRAFORM_FOLDER/$TERRAFORM_WORKSPACE-$TERRAFORM_JSON_PLAN_FILE
  when: manual

.terraform_apply:
  extends: .terraform_base
  environment:
    name: $ENVIRONMENT-$REGION
  script:
    - terraform apply -input=false $TERRAFORM_WORKSPACE-$TERRAFORM_PLAN
  when: manual

.terraform_destroy:
  extends: .terraform_base
  environment:
    name: $ENVIRONMENT-$REGION
  script:
    - terraform destroy -var-file=$ENVIRONMENT/$REGION/terraform.tfvars -auto-approve
  when: manual

Variables

In the above code a number of variables are going to be used and here is an explanation of what they need to be:

TERRAFORM_IMAGE_VERSION	This is the version of then Terraform docker container that will be used when performing the Terraform functions.
TERRAFORM_FOLDER	The location of where the terraform folder is going to be located, wrt the pipeline.
TERRAFORM_JSON_PLAN_FILE	The name of the JSON plan file
TERRAFORM_PLAN	The name of the Terraform plan file
TERRAFORM_PLAN_EXTRA_VARS	Any additional Terraform vars
TERRAFORM_WORKSPACE	The name of the Terraform workspace
INVENTORY_CODE	A prefix for your Terraform managed resources
ENVIRONMENT	The environment for your resources. Eg: prod, dev, staging, etc.
REGION	The region where your resources will be provisioned. Eg: ap-southeast-2.

Template Jobs

The above template defines the following jobs:

terraform_image	The docker hub Terraform image provided by Hashicorp. (Link)
terraform_cache	The .terraform cache artifact
terraform_init	The steps to perform terraform init
terraform_base	The steps to setup terraform in the pipeline.
terraform_validate	Validate your terraform code
terraform_plan	Perform a Terraform plan and output to the TERRAFORM_PLAN file.
terraform_apply	Provision your Terraform managed resources.
terraform_destroy	Destroy all Terraform managed resources.

What is extends

The extends keyword is used to inherit from the template and then override or add to its properties as needed. For example, you might want to inherit from a template that defines a set of environment variables, but then add a new script to run a specific command in your pipeline.

In the above script in .terraform_apply step we are extending the .terraform_base and then adding the apply script. The .terraform_base in turn extends the .terraform_image, .terraform_cache and .terraform_init steps.

Gitlab Pipeline

Now its time to use the above template in your specific pipeline.

Include the template

The first step in your pipeline is to include the above template. You can do so in the following manner:

include:
  - project: 'templates'
    ref: 'master'
    file:
      - 'terraform.yml'

So in the above snippet we are importing the terraform.yml file from the templates project.

Use the template

In order to use the template, it is like using any step in a pipeline. Add the following code to your gitlab pipeline:

# Default output file for Terraform plan
variables:
  INVENTORY_CODE: "my-resources"
  TERRAFORM_WORKSPACE: ${INVENTORY_CODE}_${ENVIRONMENT}_${REGION}_${OWNER}

### TEMPLATE ###
.terraform_plan: &terraform_plan
  extends: .terraform_base
  environment:
    name: $ENVIRONMENT-$REGION
    action: prepare
  script:
    - terraform plan -var-file=$ENVIRONMENT/$REGION/$OWNER/terraform.tfvars -out=$TERRAFORM_WORKSPACE-$TERRAFORM_PLAN
    - terraform show --json $TERRAFORM_WORKSPACE-$TERRAFORM_PLAN > $TERRAFORM_WORKSPACE-$TERRAFORM_JSON_PLAN_FILE
  when: manual
  artifacts:
    paths:
      - $TERRAFORM_FOLDER/$TERRAFORM_WORKSPACE-$TERRAFORM_PLAN
    reports:
      terraform: $TERRAFORM_FOLDER/$TERRAFORM_WORKSPACE-$TERRAFORM_JSON_PLAN_FILE
stages:
  - validate
  - build
  - deploy
  - destroy

### Terraform Validate ###
validate:
  stage: validate
  extends: .terraform_validate

### AU1 - DEV - AP-SOUTHEAST-2 ###
au1:demo:ap-southeast-2:plan:
  stage: build
  <<: *terraform_plan
  variables:
    ENVIRONMENT: "dev"
    REGION: "ap-southeast-2"
    OWNER: "au1"

Lets look at the .terraform_plan: &terraform_plan job.

Over here we are exending the terraform_base job from the templates.yml file. And then we are adding additional scripts to do a terraform plan. This is an example where we are not using the terraform_plan job from the templates.yml file in order to show that there are some occasions where you can use your own code.

We then call the terraform_plan job in the au1:demo:ap-southeast-2:plan: step and provide the values to the necessary variables.