Configure S3 bucket as Terraform backend [Step-by-Step]

 

Introduction - Configure AWS S3 bucket as Terraform backend

When using Terraform for IAC, it needs to keep track of the infrastructure it's creating. It does this by means of a state file. This file is just a simple JSON file (though its extension is .tfstate) that defines the infrastructure that has been created by Terraform and any proposed changes. By default, this file is stored on the local machine where terraform commands are run and should be checked into git or any other source code management system used.

But because this is just a simple file, it can actually be edited by anyone that has access to it and this might cause unwanted behaviours in the state of your infrastructure. Also, this is not ideal for collaboration as git conflicts may arise if multiple developers are modifying their own local copy of the file. As a result of this, Terraform introduce multiple online storage locations for this file. Some of them include; An AWS S3 bucket, Terraform cloud, etc.

In this article, we will be looking at how we can configure an S3 bucket as our backend.

 

Pre-requisites

Now, in order to follow up with this tutorial, here are a few things you need to get set up in your local environment.

• Terraform: This is our IAAC tool of choice so you need to install it in your local environment.
• An AWS account: Since we are using an AWS S3 bucket for our backend, you need to have an AWS account with permissions to create an S3 bucket, edit bucket policies and create a dynamodb table.
• The AWS CLI: Terraform needs the AWS CLI installed in order to make API calls. Follow these instructions to install it as well as configure the CLI with your access key and secret key.
• AWS S3 Bucket: You should have some background knowledge on AWS S3 Bucket.

Alright, that's all you need to get started so let's get into it

 

Step 1: Create AWS S3 bucket

Now the first step here is to create the S3 bucket we will be using to store our state. Search for Buckets in the AWS Console and click on Buckets

 

Next click on Create Bucket to create a new S3 Bucket and follow the wizard to create a new S3 bucket.

 

In my case, I will be creating a bucket called golinuxbucket1. Make sure the name is unique.

 

Make sure versioning is turned on for the bucket.

 

Enable KSM encryption for the bucket as well. Then click on create bucket.

 

Step 2: Modify AWS S3 bucket policy

Now, click on the newly created bucket, the Permissions tab, and edit the bucket policy.

 

Add the following bucket policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "<your_user_arn>"
            },
            "Action": "s3:ListBucket",
            "Resource": "<your_bucket_arn>"
        },
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "<your_user_arn>"
            },
            "Action": [
                "s3:GetObject",
                "s3:PutObject"
            ],
            "Resource": "<your_bucket_arn>/*"
        }
    ]
}

 

Make sure to replace <your_user_arn> and <your_bucket_arn> with the appropriate values. You can get your user arn from the command line by running aws sts get-caller-identity

root@ubuntu:~/terraform-resources# aws sts get-caller-identity
{
    "UserId": "311590943723",
    "Account": "311590943723",
    "Arn": "arn:aws:iam::311590943723:root"
}

 

Your Bucket ARN can be found in the properties tab of the S3 bucket.

 

Alright save the policy:

 

Step 3: Create DynamoDB table

Now, when using an S3 bucket as a backend, it improves collaboration as multiple team members can modify the state file without causing any infrastructure/git conflicts. However, what happens when 2 or more people attempt to modify the state file at the same time🤔 Well, this is where state locking comes in. State locking essentially prevents write operations to your state file while another write operation is ongoing. I won't be going in depth into this though but you can read more about it in the official documentation. In our case, we will be using a dynamodb table to lock our state.

Search for dynamo in the AWS console:

 

Head over to the dynamodb console and create a new table.

 

Use any table name of your choice but set the partition key to LockID. Leave the remaining settings as default and click create table.

Alright, Our AWS account has been setup. Lets go modify our terraform file to use this backend.

 

Step 4: Configure Terraform to point to this backend

Now that the infrastructure has been setup, let's modify our Terraform configuration to point to this backend. In your main.tf file, add the following code preferably at the top of the file.

terraform {
  backend "s3" {
    bucket         = "<your_bucket_name>"
    key            = "terraform.tfstate"
    region         = "<your_aws_region>"
    dynamodb_table = "<your_dynamo_dbtable_name>"
  }
}

Replace the following variables with appropriate values:

• <your bucket name>: The S3 Bucket which we created above i.e. golinuxbucket1
• <your aws region>: You can get this information from the AWS Console. You can check AWS regions and Availability zones to get your region name.
• <your_dynamo_dbtable_name>: The DynamoDB table we created i.e. golinuxtable

 

Step-5: Initialize Terraform

Even though you may have initialize your environment, but whenever there is a change in configuration of the backend then it recommended to re-run the terraform init command

 

Step-6: Apply Terraform changes

Now run terraform apply and wait for the infrastructure to get created.

root@ubuntu:~/terraform-resources# terraform apply
Acquiring state lock. This may take a few moments...

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no
differences, so no changes are needed.
Releasing state lock. This may take a few moments...

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Now since we have not added any resource details to create an new infrastructure hence nothing was deployed.

But Terraform should automatically recognize the new backend and store the state there. Now head over to the S3 bucket you created and you should see your state file there.

 

So we have successfully configured AWS S3 Bucket as the backend for Terraform based deployments.

 

Conclusion

We have finally come to the end of this tutorial. We took a look at how to configure terraform to use a remote storage. In this case, AWS S3. I hope you learned a lot. Bye for now✌️