Testing your Terraform project with Terratest

Cover Image for Testing your Terraform project with Terratest
Rob Morgan
Rob Morgan

Early last week Gruntwork open-sourced “Terratest” - their internal tool for testing infrastructure code. You can read more about it on their blog post here: Open sourcing Terratest: a swiss army knife for testing infrastructure code. I decided to take a closer look and add tests to my Rolling Deploys on AWS using Terraform project. I hope you can follow along and adapt the workflow to suit your own projects.

Note: I wrote this post on a Mac but it can be easily adapted to other systems.

Getting Started

First things first you’ll need a recent version of the Go language installed. If you're on a Mac you can install one easily using Homebrew:

$ brew update
$ brew install go

Next install the dep dependency management tool so we can add Terratest to the project.

$ brew install dep

If you wish to install Terratest globally then you can use the go get command:

$ go get -v github.com/gruntwork-io/terratest

You’ll also need Git, Terraform and Packer installed. Please follow the details described in my Bootstrapping Terraform and Packer on your Mac post.

Cloning the Repository

Now let's clone my example repository so we can add tests to it:

$ git clone https://github.com/robmorgan/terraform-rolling-deploys.git

Terratest Workflow

As mentioned on the Gruntwork blog post the basic workflow for using Terratest is as follows:

  1. Write tests using Go’s built-in package testing by creating files that end in _test.go.
  2. Use Terratest to execute Terraform and Packer which will in turn create real infrastructure in a real environment.
  3. Use the helper methods in Terratest to validate that the infrastructure was created and works correctly.
  4. Clean up and Destroy everything at the end.

Adding Terratest to your Project

If you didn’t install Terratest globally then the preferred way is by using the dep tool:

dep init
dep ensure -add github.com/gruntwork-io/terratest

Note: If your project is not checked out under your $GOPATH then dep may throw an error.

I got an error about not having any Go files in the root of my project so I created a dummy one:

$ echo "package main" > main.go

Writing Tests

As my project was initially written two years ago, I first had to upgrade it to work with the latest version of Terraform. I added a new init target to the Makefile and locked the AWS provider to the latest version.

My project includes both Terraform and Packer code it makes sense to test both areas. I came up with the following test sequence:

  1. Run Packer to build an AMI.
  2. Create the infrastructure using the Terraform apply command.
  3. Validate that the Autoscaling group has been created and the web servers are responding correctly.
  4. Deregister and cleanup the AMI.
  5. Destroy the infrastructure using the Terraform destroy command.

After browsing the various Terratest examples I found I could reuse most of the code in the terraform-redeploy-example project.

Testing the Packer Code

It makes sense to test the Packer code first as it executes prior to Terraform. Here is the Packer template I use in my project:

{
  "variables": {
    "aws_access_key": "{{env `AWS_ACCESS_KEY_ID`}}",
    "aws_secret_key": "{{env `AWS_SECRET_ACCESS_KEY`}}",
    "ssh_username": "ubuntu"
  },
  "builders": [
    {
      "type": "amazon-ebs",
      "access_key": "{{user `aws_access_key`}}",
      "secret_key": "{{user `aws_secret_key` }}",
      "region": "eu-west-1",
      "source_ami": "ami-8d16ccfe",
      "instance_type": "c3.large",
      "ssh_username": "{{user `ssh_username`}}",
      "ami_name": "packer-base {{timestamp}}",
      "associate_public_ip_address": true
    }
  ],
  "provisioners": [
    {
      "type": "shell",
      "execute_command": "echo {{user `ssh_username`}} | {{ .Vars }} sudo -E -S sh '{{ .Path }}'",
      "inline": [
        "mkdir -p /ops",
        "chmod a+w /ops"
      ]
    },
    {
      "type": "file",
      "source": "{{template_dir}}/../../ssh_keys",
      "destination": "/ops"
    },
    {
      "type": "shell",
      "execute_command": "echo {{user `ssh_username`}} | {{ .Vars }} sudo -E -S sh '{{ .Path }}'",
      "inline": [
        "cat /ops/ssh_keys/* >> /home/ubuntu/.ssh/authorized_keys"
      ]
    },
    {
      "type": "shell",
      "scripts": [
        "{{template_dir}}/../scripts/base.sh"
      ],
      "execute_command": "chmod +x {{ .Path }}; {{ .Vars }} sudo -E '{{ .Path }}'"
    },
    {
      "type": "shell",
      "scripts": [
        "{{template_dir}}/../scripts/docker.sh"
      ],
      "execute_command": "chmod +x {{ .Path }}; {{ .Vars }} sudo -E '{{ .Path }}'"
    },
    {
      "type": "shell",
      "execute_command": "echo {{user `ssh_username`}} | {{ .Vars }} sudo -E -S sh '{{ .Path }}'",
      "inline": [
        "docker pull nginx"
      ]
    },
    {
      "type": "shell",
      "scripts": [
        "{{template_dir}}/../scripts/cleanup.sh"
      ],
      "execute_command": "chmod +x {{ .Path }}; {{ .Vars }} sudo -E '{{ .Path }}'"
    }
  ]
}

You can also find it directly in the GitHub repository. I rewrote the paths to use the {{template_dir}} variable as it plays better with Terratest.

I then created a new test file under the test directory:

$ mkdir -p test
$ touch test/terraform_packer_test.go

And added the following code:

package test

import (
	"fmt"
	"testing"
	"time"

	"github.com/gruntwork-io/terratest/modules/aws"
	"github.com/gruntwork-io/terratest/modules/http-helper"
	"github.com/gruntwork-io/terratest/modules/packer"
	"github.com/gruntwork-io/terratest/modules/retry"
	"github.com/gruntwork-io/terratest/modules/terraform"
	"github.com/gruntwork-io/terratest/modules/test-structure"
)

// The test is broken into "stages" so you can skip stages by setting environment variables (e.g., skip stage
// "deploy_initial" by setting the environment variable "SKIP_deploy_initial=true"), which speeds up iteration when
// running this test over and over again locally.
func TestTerraformPackerExample(t *testing.T) {
	t.Parallel()

	// Use eu-west-1 for now, but in future we should pick a random AWS region
	awsRegion := "eu-west-1"

	// The folder where we have our Terraform code
	workingDir := "../"

	// Build the AMI for the web app
	test_structure.RunTestStage(t, "build_ami", func() {
		buildAmi(t, awsRegion, workingDir)
	})

	// At the end of the test, delete the AMI
	defer test_structure.RunTestStage(t, "cleanup_ami", func() {
		deleteAmi(t, awsRegion, workingDir)
	})
}

// Build the AMI
func buildAmi(t *testing.T, awsRegion string, workingDir string) {
	packerOptions := &packer.Options{
		// The path to where the Packer template is located
		Template: "../packer/aws/app-server.json",
	}

	// Save the Packer Options so future test stages can use them
	test_structure.SavePackerOptions(t, workingDir, packerOptions)

	// Build the AMI
	amiId := packer.BuildAmi(t, packerOptions)

	// Save the AMI ID so future test stages can use them
	test_structure.SaveAmiId(t, workingDir, amiId)
}

// Delete the AMI
func deleteAmi(t *testing.T, awsRegion string, workingDir string) {
	// Load the AMI ID and Packer Options saved by the earlier build_ami stage
	amiId := test_structure.LoadAmiId(t, workingDir)

	aws.DeleteAmi(t, awsRegion, amiId)
}

This code allows Terratest to build an AMI and save it locally in a configuration file to be used by later steps. Finally when the test finishes Terratest deregisters the AMI.

Testing the Terraform Code

Now we need to extend the tests to cover the Terraform code. I simply copied and modified some additional methods from the example project. This included running the Terraform apply command and verifying the ELB was returning a HTTP 200 response.

The test file now looks like:

package test

import (
	"fmt"
	"testing"
	"time"

	"github.com/gruntwork-io/terratest/modules/aws"
	"github.com/gruntwork-io/terratest/modules/http-helper"
	"github.com/gruntwork-io/terratest/modules/packer"
	"github.com/gruntwork-io/terratest/modules/retry"
	"github.com/gruntwork-io/terratest/modules/terraform"
	"github.com/gruntwork-io/terratest/modules/test-structure"
)

// The test is broken into "stages" so you can skip stages by setting environment variables (e.g., skip stage
// "deploy_initial" by setting the environment variable "SKIP_deploy_initial=true"), which speeds up iteration when
// running this test over and over again locally.
func TestTerraformPackerExample(t *testing.T) {
	t.Parallel()

	// Use eu-west-1 for now, but in future we should pick a random AWS region
	awsRegion := "eu-west-1"

	// The folder where we have our Terraform code
	workingDir := "../"

	// Build the AMI for the web app
	test_structure.RunTestStage(t, "build_ami", func() {
		buildAmi(t, awsRegion, workingDir)
	})

	// At the end of the test, delete the AMI
	defer test_structure.RunTestStage(t, "cleanup_ami", func() {
		deleteAmi(t, awsRegion, workingDir)
	})

	// At the end of the test, undeploy the web app using Terraform
	defer test_structure.RunTestStage(t, "cleanup_terraform", func() {
		undeployUsingTerraform(t, workingDir)
	})

	// Deploy the web app using Terraform
	test_structure.RunTestStage(t, "deploy_terraform", func() {
		deployUsingTerraform(t, awsRegion, workingDir)
	})

	// Validate that the ASG deployed and is responding to HTTP requests
	test_structure.RunTestStage(t, "validate", func() {
		validateAsgRunningWebServer(t, workingDir)
	})
}

// Build the AMI
func buildAmi(t *testing.T, awsRegion string, workingDir string) {
	packerOptions := &packer.Options{
		// The path to where the Packer template is located
		Template: "../packer/aws/app-server.json",
	}

	// Save the Packer Options so future test stages can use them
	test_structure.SavePackerOptions(t, workingDir, packerOptions)

	// Build the AMI
	amiId := packer.BuildAmi(t, packerOptions)

	// Save the AMI ID so future test stages can use them
	test_structure.SaveAmiId(t, workingDir, amiId)
}

// Delete the AMI
func deleteAmi(t *testing.T, awsRegion string, workingDir string) {
	// Load the AMI ID and Packer Options saved by the earlier build_ami stage
	amiId := test_structure.LoadAmiId(t, workingDir)

	aws.DeleteAmi(t, awsRegion, amiId)
}

// Deploy the terraform-packer-example using Terraform
func deployUsingTerraform(t *testing.T, awsRegion string, workingDir string) {
	// Load the AMI ID saved by the earlier build_ami stage
	amiId := test_structure.LoadAmiId(t, workingDir)

	terraformOptions := &terraform.Options{
		// The path to where our Terraform code is located
		TerraformDir: workingDir,

		// Variables to pass to our Terraform code using -var options
		Vars: map[string]interface{}{
			"ami": amiId,
		},
	}

	// Save the Terraform Options struct, instance name, and instance text so future test stages can use it
	test_structure.SaveTerraformOptions(t, workingDir, terraformOptions)

	// This will run `terraform init` and `terraform apply` and fail the test if there are any errors
	terraform.InitAndApply(t, terraformOptions)
}

// Undeploy the terraform-packer-example using Terraform
func undeployUsingTerraform(t *testing.T, workingDir string) {
	// Load the Terraform Options saved by the earlier deploy_terraform stage
	terraformOptions := test_structure.LoadTerraformOptions(t, workingDir)

	terraform.Destroy(t, terraformOptions)
}

// Validate the ASG has been deployed and is working
func validateAsgRunningWebServer(t *testing.T, workingDir string) {
	// Load the Terraform Options saved by the earlier deploy_terraform stage
	terraformOptions := test_structure.LoadTerraformOptions(t, workingDir)

	// Run `terraform output` to get the value of an output variable
	url := terraform.Output(t, terraformOptions, "url")

	// It can take a few minutes for the ASG and ELB to boot up, so retry a few times
	//	maxRetries := 30
	timeBetweenRetries := 5 * time.Second

	// Verify that the ELB returns a proper response
	elbChecks := retry.DoInBackgroundUntilStopped(t, fmt.Sprintf("Check URL %s", url), timeBetweenRetries, func() {
		http_helper.HttpGetWithCustomValidation(t, url, func(statusCode int, body string) bool {
			return statusCode == 200
		})
	})

	// Stop checking the ELB
	elbChecks.Done()
}

Now it's time to execute the test suite.

Executing the Test Suite

I added a new target to the Makefile which simply proxies to the go test command. You can execute the tests by running:

$ make test

This will take approximately 15 minutes to run the entire test suite.

Note: By default Go imposes a 10 minute timeout on running tests, so I’ve increased this value to 30m in the Makefile using the -timeout parameter to play it safe.

Terratest also includes functionality to skip stages by specify environment variables. This is a really handy feature when writing tests as you can skip time-expensive stages like rebuilding the AMI each time. When I was debugging the tests I simply set SKIP_build_ami=true before running the test suite:

$ SKIP_build_ami=true make test

Note: You may need to manually build an AMI or ensure your test suite doesn’t remove the one in .test-data/AMI.json before skipping a stage.

When Terratest finishes running the tests correctly it will return the following output:

PASS
ok      github.com/robmorgan/terraform-rolling-deploys/test     800.570s

Wrap Up

Now I have a simple, yet effective test suite for my Rolling Deploys on Terraform project. You can find all of my commits in the terratest branch. In the future I could expand the tests to build a new AMI and redeploy it to the Autoscaling group using Terraform. Terratest could validate that the ELB continues to serve HTTP 200 responses without any downtime.

For more information you might want to check out the best practices guide available in the Gruntwork Terratest repository or write me to on Twitter.

Happy testing!