In this blog post, I’m going to look into a VCS-integrated (Version Control System) Amazon EKS deployment using Hashicorp’s Terraform Cloud. Once deployed, I will also be deploying Palo Alto Networks’ Prisma Cloud Defender agents into the cluster. In future series, I will be looking at tuning Palo Alto Prisma to monitor and protect Kubernetes clusters using real-time container protection and firewall capabilities.

 

For this deployment, I’ll be using our VCS to maintain the infrastructure code for the Amazon EKS cluster. Terraform Cloud supports integrations with many of the leading VCS, including Gitlab, GitHub, Bitbucket and Azure DevOps Services. In my case, the VCS I’ll be using is Gitlab Enterprise.

 

The prerequisites required to follow along with this lab include:

 

 

Step 1: Create a new application in Gitlab

 

 

Step 2: Add a VCS Provider in Terraform Cloud

 

Step 3: Update the Gitlab Callback URI

 

 

Step 4: Request access for Terraform Cloud

 

 

Creating a Gitlab Project & Pushing the Code to the Repo

At this stage, I need to create a project to link the Terraform Cloud workspace to and populate it with code so that I don’t generate any errors in Terraform (see below). Here’s how to create the project:

 

 

 

Image 6a: Creating the new Gitlab project

 

Creating a Terraform Cloud Workspace

Once the VCS provider and Terraform Cloud have been integrated, a project workspace must exist in Terraform Cloud so that code commits can trigger a Terraform plan run. To create a Terraform workspace connected to our Gitlab Enterprise, I perform the following steps:

 

 

 

Image 7: Creating a new Workspace – Connecting the VCS Provider

 

 

Image 8: Creating a new Workspace – Choosing a Repo to Connect to

 

 

Image 9: Creating a new Workspace – Configuring the Settings

 

Note: If the Gitlab repository you created and are attempting to link to is empty, Terraform Cloud will generate the following error when attempting to link a workspace to a repo:

 

 

Image 9a: Terraform Cloud workspace creation error (due to user’s Gitlab repo being empty)

 

Deploying an EKS Cluster using Terraform Cloud & Gitlab

Warning: Following this guide will create objects in your AWS account that will increase your AWS bill.

At this stage, using our Gitlab Enterprise/Terraform integration, we’re ready to deploy Amazon EKS Kubernetes Infrastructure as Code using Terraform Cloud.

 

The Infrastructure as Code, AWS EKS Terraform scripts we are working with are listed below and can be found at Optiv’s Github repository here. I will briefly go into each script and explain its purpose. It should be noted that these scripts are intended for educational purposes only and should only be used as a tool for further development.

 

These scripts include provisioning for the following resources:

 

 

eks-cluster.tf – This script provisions the following cluster resources:

 

 

eks-worker-nodes.tf - This script provisions the following cluster resources:

 

 

outputs.tf – This script generates the correct values for kubectl’s .config file, specifically the server and certificate-authority-data. This data is required to connect to the EKS cluster. This data can be obtained from Terraform Cloud once the terraform run has completed. We will revisit this section once the Terraform plan has been applied.

 

providers.tf – This script contains the values for the AWS Access ID and AWS Secret for the user tasked with creating the infrastructure. While these values can be hard-coded into script, this method is discouraged. A more secure and recommended method is to create environment variables for both “aws_access_id” and “aws_secret_id” within Terraform Cloud. This step will be covered in more detail later in this series.

 

variables.tf – Although this functionality can be centrally managed and used in Terraform Cloud (Settings > Environment Variables), this script is for variables that users may want to dynamically call within their code.

 

vpc.tf – This script generates VPC resources required for the cluster, including VPCs, subnets, internet gateways and routing tables.

 

workstation-external-ip.tf – This script is not required and is only provided as an example to easily fetch the external IP of your local workstation to configure inbound EC2 Security Group access to the Kubernetes cluster.

 

Prior to our deployment, I’m going to modify some variable values in some of the scripts to change what is being deployed. Other values can be modified as needed.

 

Changing instance type and default SSH key to be used for the instance

 

In this case, I wanted to change the type of instance to be launched in the cluster (t2.micro) and also wanted to use an existing AWS SSH keypair for my EKS cluster nodes. The SSH key value is set using the “key_name” variable, which is highlighted below. Alternatively, “key_name” can be omitted and a new AWS keypair will be automatically generated.

 

resource "aws_launch_configuration" "demo" {
associate_public_ip_address	= true
iam_instance_profile	= "${aws_iam_instance_profile.demo-node.name}"
image_id	= "${data.aws_ami.eks-worker.id}"
instance_type	= "t2.micro"
name_prefix	= "terraform-eks-demo"
security_groups	= ["${aws_security_group.demo-node.id}"]
user_data_base64	= "${base64encode(local.demo-node-userdata)}"
key_name	= "xxxxx"
lifecycle {
create_before_destroy	= true
}
}

 

 

Change the initial deployment and autoscaling options for EKS Cluster

 

In this case, I wanted to change the values of my cluster, including “desired_capacity,” “max_size” and “min_size.”

 

resource "aws_autoscaling_group" "demo" {
desired_capacity	= 3
launch_configuration	= "${aws_launch_configuration.demo.id}"
max_size	= 3
min_size	= 1
name	= "terraform-eks-demo"
vpc_zone_identifier	= "${aws_subnet.demo[*].id}"
tag {
key	= "Name"
value	= "terraform-eks-demo"
propagate_at_launch	= true
}
tag {
key	= "kubernetes.io/cluster/${var.cluster-name}"
value	= "owned"
propagate_at_launch	= true
}
}

 

 

Image 9b: Adding the AWS credentials to Terraform environment variables

 

Once a user has successfully committed a file in Gitlab, the integration with Terraform takes over. Switch to Terraform Cloud and select the workspace linked to the repository in Gitlab. You’ll see that Terraform knows about the change in the code and as a result a plan run is queued and ready to apply. For this exercise, I’m using the workspace “eks-terraform01” in Terraform.

 

Image 10: Terraform Cloud plan run that is requiring confirmation

 

From here, users can review the changes that will be happening and the cost associated with the plan run. Full logs are available showing the plan that has been applied.

 

Image 11: Pending Terraform plan run

 

Image 12: A run of a Terraform plan run while resources are being provisioned (this process can take 10-15 minutes to complete).

 

Terraform plan as well as Terraform apply work by comparing a diff of the state file they have with the contents of the source code for a particular branch that is being committed or merged. If using the free, open-source version of Terraform (OSS), ensure that safe handling of the tfstate file (e.g. encrypted at rest) is observed. Another mistake often made by OSS users that we want to avoid is to make sure to never commit the tfstate file to VCS – a very real error that happens all the time with users of Terraform OSS, hence one benefit of the SaaS offering (both Cloud and Enterprise).

 

Image 13: Terraform Apply run after successfully completing.

 

The result of the dynamic output from outputs.tf, includes the server and certificate-data information needed to connect to the cluster (see below). Applying a Terraform plan takes around 10-15 minutes, depending on the amount of resources being provisioned.

 

Configuring AWS Authenticator and kubectl

 

At this time, we move onto connecting to the cluster using the program kubectl. For kubectl to connect and interface properly with an AWS EKS cluster, we also need to install and configure the AWS CLI client with the aws-iam-authenticator component. Instructions on how to install the following components can be found below:

 

 

Once the AWS authenticator and kubectl have been installed, we need to update our kubectl config file, which is typically found at ~/.kube/config. We primarily need to focus on the cluster endpoint URL and the contents of certificate-data. These values can either be found in the Terraform run logs or from within the AWS console, specifically in the EKS section of the UI. To get these values from Terraform Cloud, do the following:

 

 

Alternatively, we can get the information from the AWS console. To get this information from the AWS console, do the following:

 

 

The data from the API Server Endpoint URL is needed for kubectl config’s “server” parameter and the data from the Certificate Authority field is needed for kubectl config’s “certificate-authority-data” field.

 

Image 14: AWS Console – EKS Cluster Information

 

After populating kubectl’s config with the required information, we should be able to run a Kubernetes command to test the AWS auth and connectivity. To do this, we issue the following command:

 

kubectl get svc

 

If everything is working correctly, we should be able to connect to the API endpoint on the EKS master and receive the following response (the internal IP has been masked):