Testing your Terraform project on CircleCI

At Brightfame we run our Terraform tests using Terratest. I described this in more detail last week in my post - Testing your Terraform project with Terratest. Today I want to explain how you can easily test your Terraform code on CircleCI. CircleCI is great for running your continuous integration tasks in the cloud and not worrying about having to manage the infrastructure yourself. It is also particularly great for smaller teams and early stage startups as they offer a free plan limited to one container and one concurrent job. Recently they released CircleCI 2.0 which includes many new features and more powerful Docker support. We’ll use CircleCI to easily run our Terraform test suite.

Note: I wrote this post on a Mac but it can be easily adapted to other systems.

Getting Started

In order to complete this tutorial you’ll need Git, Terraform and Packer installed. Please follow the details described in my Bootstrapping Terraform and Packer on your Mac post. If you want to test the steps locally you should have Docker installed. Additionally you will require an AWS account with access to the IAM service and CircleCI configured for your GitHub account.

Workflow

For this example I’d like to reuse my Rolling Deploys on AWS using Terraform example project. Last week I added some tests using Terratest and pushed them to the terratest branch. We’ll use CircleCI to execute the tests every time we push to the circleci branch.

Checking out the example repository

First of all we need need to check out my example project and switch to the terratest branch:

$ git clone https://github.com/robmorgan/terraform-rolling-deploys.git
$ cd terraform-rolling-deploys
$ git checkout terratest
Branch terratest set up to track remote branch terratest from origin.
Switched to a new branch 'terratest'

Then we need to create a new branch to store the CircleCI configuration:

$ git checkout -b circleci

You should then create a new GitHub repository and configure Git to use this as your new origin. We’ll need this later so CircleCI can run the tests.

Now we’ll need to write a CircleCI configuration file.

The CircleCI configuration file

CircleCI uses a configuration file (.circleci/config.yml) to describe the build behaviour and workflow. Let’s create a new configuration file to tell CircleCI what steps to perform when building our project:

$ mkdir -p .circleci
$ touch .circleci/config.yml

And paste in the following YAML code:

# Golang CircleCI 2.0 configuration file
#
# Check https://circleci.com/docs/2.0/language-go/ for more details
version: 2
jobs:
  build:
    # only build the circleci branch for now
    branches:
      only:
        - circleci
    docker:
      # specify the version
      - image: circleci/golang:1.10
    working_directory: /go/src/github.com/robmorgan/terraform-rolling-deploys
    steps:
      - checkout
      - run: curl -L -s https://github.com/golang/dep/releases/download/v0.4.1/dep-linux-amd64 -o /go/bin/dep && chmod +x /go/bin/dep
      - run: curl -L -s https://releases.hashicorp.com/packer/1.2.3/packer_1.2.3_linux_amd64.zip -o /go/bin/packer.zip && unzip /go/bin/packer.zip -d /go/bin && chmod +x /go/bin/packer
      - run: curl -L -s https://releases.hashicorp.com/terraform/0.11.7/terraform_0.11.7_linux_amd64.zip -o /go/bin/terraform.zip && unzip /go/bin/terraform.zip -d /go/bin && chmod +x /go/bin/terraform
      - run: dep ensure -vendor-only
      - run: make test

As Terratest is written using the Go Programming Language I chose to use the prebuilt CircleCI Go Docker image. I also extended this image to include a few tools including the latest versions of Packer and Terraform. Finally I run the tests using the test target in the Makefile.

In the next step I’ll briefly explain how to test the environment locally using Docker.

Testing Locally

Note: This step is optional and can be skipped to save time.

As CircleCI runs builds inside Docker containers you can easily test your workflow locally using their official images:

$ docker pull circleci/golang:1.10
$ docker run -ti -v $PWD:/go/src/github.com/robmorgan/terraform-rolling-deploys circleci/golang:1.10 bash
$ export AWS_REGION=us-east-1
$ export AWS_ACCESS_KEY_ID=foo
$ export AWS_SECRET_ACCESS_KEY=bar
$ mkdir -p /go/src/github.com/robmorgan/terraform-rolling-deploys
$ curl -L -s https://github.com/golang/dep/releases/download/v0.4.1/dep-linux-amd64 -o /go/bin/dep && chmod +x /go/bin/dep
$ curl -L -s https://releases.hashicorp.com/packer/1.2.3/packer_1.2.3_linux_amd64.zip -o /go/bin/packer.zip && unzip /go/bin/packer.zip -d /go/bin && chmod +x /go/bin/packer
$ curl -L -s https://releases.hashicorp.com/terraform/0.11.7/terraform_0.11.7_linux_amd64.zip -o /go/bin/terraform.zip && unzip /go/bin/terraform.zip -d /go/bin && chmod +x /go/bin/terraform
$ dep ensure -vendor-only
$ make test

I generally just mimic the commands described in the configuration file (.circleci/config.yml).

Note: You will need to add your SSH public keys 🔑 and probably configure them using ssh-agent for Terraform to work correctly.

Now let’s create an AWS IAM key pair so Terratest can create real resources using Packer and Terraform.

Creating the AWS IAM Key Pair

In order to run the tests successfully on CircleCI we will need to create an AWS IAM key pair so we can create real resources.

Head over to the IAM console to create a new user:

I am using the region us-east-1. I chose the username circleci for my key pair. Select the 'Programmatic access' access type.

Now create or select an appropriate AWS IAM policy.

Note: I’ve attached the PowerUserAccess policy (full access to creating AWS resources, but does not allow the management of IAM users and groups) in order to complete this tutorial, however you may want to create an account with lesser privileges.

Then click ‘Next: Review’ and finally click ‘Create user’.

Be sure to use the ‘Download .csv’ function to save your credentials locally. We’ll need to use them in a later step.

Generate a Private Key Pair

A SSH key pair is required so Terraform can provision our instances. Open a Bash console locally, then run:

$ ssh-keygen -t rsa -b 4096 -C "deployer@robs-laptop.local" -N '' -f ssh_keys/deployer.pem
$ mv ssh_keys/deployer.pem.pub ssh_keys/deployer.pub

This will create an SSH key pair we will use with CircleCI, Terraform and AWS to provision the Bastion instance used in the example project. Keep the key pair handy as we’ll need them in the next step.

Configuring CircleCI

It’s time to configure CircleCI to build your GitHub project. Ensure you have committed the CircleCI configuration file and pushed your branch to the new GitHub origin. Then visit the CircleCI dashboard and add your project using the ‘Add Projects’ button. Make sure you click the ‘Start building’ button.

CircleCI will probably try to build your project right away and fail. We will need to configure it with the AWS credentials and SSH key pair.

Let’s add the AWS IAM credentials first. Head to the CircleCI settings for your chosen repository and click the ‘Environment Variables’ link under ‘Build Settings’.

We will need to configure the following environment variables:

1. AWS_REGION=us-east-1
2. AWS_ACCESS_KEY_ID=foo
3. AWS_SECRET_ACCESS_KEY=bar

Note: Be sure to replace the AWS values with your own credentials.

Now we need to add the private key from the SSH key pair we generated.

You can copy this to your clipboard with the following command:

$ cat ssh_keys/deployer.pem | pbcopy

And paste it into the CircleCI ‘Add an SSH Key’ modal. Ensure the Hostname field is empty and click ‘Add SSH Key’. This step is described in more detail in the ‘Adding an SSH Key to CircleCI’ documentation.

Now its time to run our first build.

Building your project

If CircleCI hasn’t already triggered a successful build for your project, head over to their dashboard and trigger one using the ‘rebuild’ feature or add a commit locally and push it to GitHub. My example project takes about about 12 minutes to run a complete build. If anything goes wrong, be sure to dig into the output and see if you missed a step. Hopefully all of the steps pass and CircleCI will successfully complete your first build.

You can see my public CircleCI build status here: https://circleci.com/gh/robmorgan/terraform-rolling-deploys.

Caveats

Go's testing package has a default timeout of 10 minutes so we run our tests using the -timeout 30m flag to increase this. CircleCI also likes to kill builds after 10 minutes with no log activity. If you have long running tests like we do internally at Brightfame then you may have to work around this. Please refer to the Terratest README.md for more information.

That concludes this tutorial!

Conclusion

There we have a basic overview of how to use CircleCI to run your Terraform tests using Terratest. I have pushed an example branch with all of the relevant code to my GitHub repository here: https://github.com/robmorgan/terraform-rolling-deploys/tree/circleci.

More information about CircleCI can be found in their documentation.

If you have any questions, comments or feedback then write to me on Twitter.

Happy testing!