Opening The Door To Continous Deployment with CloudFormation

I recently wrote an article called Hello Little Lambda which explained what Serverless Lambdas are and how to create them using the Amazon Web Services console.

While it’s great that humans can create that a new Lambdas can be made using the AWS console, there are some significant advantages that can be gained from using code to create your Serverless Lambdas for you. When your Lambdas are created by code, they will always be created in the same predictable way, making their configuration less error prone.

Using code to create Serverless Lambdas (or any other piece of infrastructure such as a database or a message queue) is called the principle of “Infrastructure as Code” which is often abbreviated to IaC. IaC has already been proven in the most demanding environments at companies like Amazon, Netflix, Google, Facebook and Etsy where systems simply must work and be highly reliable.

Using code to create Serverless Lambdas opens the door to Continuous Delivery. Continuous Delivery is really exciting because one of the key findings in the 2019 Accelerate State of DevOps report is that “Delivering software quickly, reliably and safely is at the heart of technology transformation and organisational performance”.

As with everything in software development, there is always a trade off. The trade off for IaC is that it will certainly take longer to create your Lambdas this way. Code and templates will need to be written to create them, as well as also writing the code that executes when the Lambdas run. It will be harder to implement IaC at first, at least until you have gained some experience in writing the templates that create Lambdas. However the upfront cost of more time required to create Lambdas pays for itself many times over, saving the cost of humans having to repeatedly create and configure infrastructure manually. IaC is an investment for the future. There are far more advantages to IaC than there are disadvantages and it is a core principle of DevOps practices. IaC thinking allows a single person to literally create hundreds of Lambda functions that are automatically configured and properly set up within seconds using a single command.

So how is a Lambda created from code?

In the Amazon Web Services landscape, a service called CloudFormation is used to create infrastructure and deploy it in an orderly predictable fashion.

Lets start with the basics and cover some CloudFormation terminology as it can be a little confusing if you are new to working with AWS.

• Each piece of infrastructure created by CloudFormation is called a resource.
• CloudFormation uses templates to deploy resources.
• All the resources which are contained within the same template are known as a stack.
• A stack is simply a collection or group of resources that are all deployed by the same template.
• When it comes to Serverless Lambdas, an extension to CloudFormation templates called SAM templates is used to create them.
• SAM stands for Serverless Application Model.

Any resources which can be declared in a CloudFormation template, can also be declared in a SAM template. SAM builds on top of CloudFormation adding a few more resource types that are not present in CloudFormation such as AWS::Serverless::Function which we use to define Lambda functions.

A SAM template is essentially a configuration file where all your Serverless resources are declared. SAM templates can be written in either YAML or JSON. Once your resources are declared in a template, CloudFormation can be invoked using the command line to execute the template.

When a SAM template is executed, it is automatically transformed into a CloudFormation template and it is then that CloudFormation template which is executed and that creates and deploys the resources.

So SAM Template -> transforms into -> CloudFormation Template -> CloudFormation uses the transformed template to deploy resources.

If you ever need to read the full documentation for SAM Templates, the docs can be found here on Github

Hello Lambda As Code

Let’s recreate the same Hello Lambda function described in my previous post, as code, using a SAM template, without making a single mouse click in the AWS console.

You will need to install the AWS CLI

All the SAM templates I have encountered so far have been written in YAML. Best practice for other deployment configuration tools (such as Kubernetes) is to write config files in YAML as they tend to be slightly more user friendly. I am unsure how common JSON SAM templates are so for this example we will be sticking with YAML.

Open up your favourite IDE (I use Visual Studio Code) and create a new file named samTemplate.yaml.

The first thing we are going to write in our SAM template is

AWSTemplateFormatVersion: "2010-09-09"

This line identifies the capabilities of the template. The latest template version is 2010-09-09 and at the time of writing this article, this is currently the only valid value. This line is optional, however if not included, the template will assume the latest version is to be used, which may catch you out if it changes in the future.

The next line to add to our SAM template is

Transform: 'AWS::Serverless-2016-10-31’

What this line does is it tells our SAM Template to use serverless transform. Serverless transform is what transforms the SAM Template into a CloudFormation template when it is executed. The CloudFormation template it transforms into is what is used to deploy the infrastructure.

It’s a good idea to add a description to the SAM template to explain it’s purpose. We can do that by adding the line

Description: Deploys a CloudFormation Stack for Hello Lambda

Your SAM template should now look something like this.

Next, we need to start defining our resources in our SAM template. For our example we are going to define a new Lambda function. Each resource is defined in a section named Resources: where it is given a unique name and its type and properties are defined. A typical resources section for a Serverless Lambda function in Go would look like this.

When we first created a Lambda using the console, we had to upload our compiled Go binary as a zip file. This time around, we have our compiled binary on our local machine and have specified the local path in our SAM template using the CodeUri property. However, templates expect a CodeUri to be a path to an s3 bucket in the format “s3://bucket/key” which contains the compiled Go binary.

If we were to try and deploy this template, as it is, using the AWS CLI with the command

aws cloudformation deploy --template-file samTemplate.yaml --stack-name HelloLambda --region eu-west-1

We would see the following error…

Failed to create the changeset: Waiter ChangeSetCreateComplete failed: Waiter encountered a terminal failure state Status: FAILED. Reason: Transform AWS::Serverless-2016-10-31 failed with: Invalid Serverless Application Specification document. Number of errors found: 1. Resource with id [HelloLambda] is invalid. 'CodeUri' is not a valid S3 Uri of the form "s3://bucket/key" with optional versionId query parameter.

The way to solve this problem is to “package” our compiled binary.

CloudFormation Packaging

What packaging does is it takes your compiled binary from the local filepath specified in your template, uploads it into an S3 bucket for you, then makes a copy of your template where all references to local files are replaced with references to the new S3 locations where it has uploaded the compiled binaries. This new template which is generated is the template which we will tell CloudFormation to deploy for us.

So the next thing we need to create our Serverless Lambda is an S3 bucket which we will use as a deployment bucket. This bucket will hold compiled Go binaries that we intend to deploy.

To avoid using the was console to create this bucket, we can create it using the CLI instead. This task only ever needs to be done once so that every template has a place to store compiled binaries that it intends to deploy.

Keep in mind that S3 bucket names are in a namespace that is shared by all AWS users, so naming your bucket a common name like “deployment-bucket” is unlikely to work. You may need to prefix your bucket name with your name or your companies name to make it’s name totally unique.

Once you have decided on a name, use the following command to create your bucket.

aws s3api create-bucket --bucket rosie-deployment-bucket --region eu-west-1 --create-bucket-configuration LocationConstraint=eu-west-1

So now we have a bucket, we can package our compiled binaries into that bucket.

I am using the same compiled and zipped binary I created in my previous article. To generate the binary you will need to build your Go code for Linux and zip up the built executable.

The package command is

aws cloudformation package --template samTemplate.yaml --s3-bucket rosie-deployment-bucket --output-template-file packaged-samTemplate.yaml

Try it. If you see an error like this…

Unable to upload artifact deployment.zip referenced by CodeUri parameter of HelloLambda resource.
An error occurred (AccessDenied) when calling the PutObject operation: Access Denied

The chances are that your AWS profile is missing S3 access. When invoking CloudFormation via the CLI (without telling it to use a specific access role) it will automatically use the permissions of the current profile instead. So you will need to check in Identity Access Management that your AWS profile has S3 access.

Once you have successfully packaged your template using this command, you should see a new template file called packageSamTemplate.yaml is created. Opening this file it will look something like

You can see that the CodeUri has been changed in the packaged template to the name of the S3 bucket where the compiled binary was uploaded.

Deploying a CloudFormation Template

After running the package command, the CLI will be super helpful and tell you the next command to run to deploy your template

You should see something like…

Successfully packaged artifacts and wrote output template to file packagedSamTemplate.yaml.
Execute the following command to deploy the packaged template
aws cloudformation deploy --template-file /pathToFile/packagedSamTemplate.yaml --stack-name <YOUR STACK NAME>

A few things are missing from this command though, you need to tell it which region you want to deploy to and you also have to specify capabilities, the capabilities tell it whether you are using default or named permission roles.

The command to deploy your template will be

aws cloudformation deploy --template-file /pathToFile/packagedSamTemplate.yaml --stack-name HelloLambdaStack --region eu-west-1 --capabilities CAPABILITY_IAM

Successful output will look like this…

Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - HelloLambdaStack

Now log into the console and navigate to the CloudFormation service dashboard

You should see that a stack called HelloLambdaStack now exists.

If you click on the stack and then click on Resources, you will be able to see the resources that make up this stack.

Two resources have been created, our Lambda function named HelloLambda and a IAM permission role for our Lambda function.

Now if you log into the console and select Lambda, you should see we have a new Lambda function called HelloLambdaStack-HelloLambda

This lambda does exactly the same thing as the lambda function we created in Hello Little Lambda however rather than creating it in the console, we created it using the principles of infrastructure as code on the command line.

We can test it in the same way, by clicking on it then clicking Test in the top right, sending it a test event, then looking in the CloudFormation logs.

So now we can do something cool. I want you to go back to the CloudFormation console, select your HelloLambdaStack and then click the Delete button. You will get a warning message, but go ahead and click the Delete stack button.

It’s status will change to DELETE_IN_PROGRESS, what’s happening now is AWS is deleting the group of resources we defined in our template, the Lambda function (and its automatically generated permission role). It might take a minute, but once it’s deleted, navigate back to the Lambda service console, guess what it’s all gone! Do not panic!

Return to the terminal window and press the up arrow key to get the last executed command back in the terminal window. Then press enter to run that command again.

CloudFormation will re-create your stack in a few seconds, ta-da your Lambda function is back.

This is the benefit of having your infrastructure as code. Now we never have to manually create infrastructure resources again. We can write reusable templates and let CloudFormation do all the hard work of creating all our resources for us.

This is one of the foundations of Continuous Deployment and now we know how to write infrastructure as code we are a step closer to creating our very first Continous Deployment pipeline.

Imagine how beautiful it’s going to be when once our code is written, committed and pushed if it compiles itself, uploads itself, packages itself and deploys itself within minutes. So many hours will be saved that we can focus on the stuff that that matters. Getting code into our production environment will no longer be a difficult time consuming task.

This was quite a long post, so well done for making it to the end! I promise I will write again soon about Serverless development.

• go
• serverless
• lambda