Deploying AKS with Terraform and Azure DevOps

11 min readAug 20, 2019

This is the first article of a multi-part series focused on AKS:

Deploying AKS with Terraform and Azure DevOps
Building and deploying a sample application with Azure DevOps and Azure Container Registry and AKS
Performing a blue/green deployment to update the application
Testing new features with Canary testing
Exploiting the AKS and containers logs with Azure Monitor Logs

12 factors app applies to IaC as well

In this article we’re going to go through the steps to setup a CI/CD pipeline to deploy a new AKS with Azure DevOps through different steps while trying to adhere to the 12 factor apps guidelines:

Version everything: Application Code, Infra As Code, CI/CD pipeline
Disposability: Fast startups and graceful shutdowns (of the application and the infrastructure)
Dev/Prod parity
Logs: Treat logs as event streams

Source

The code is located on my github : https://github.com/nyuen/AKS-Articles

git clone https://github.com/nyuen/AKS-Articles.git

Requirements

An Azure subscription
An Azure DevOps organization + Git Repository: https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/create-organization?view=azure-devops
The Azure CLI (or the Azure Cloud shell), Terraform, and Kubectl

Preview features

I’m using a few preview features for AKS including the VM scaleSets and multi node pools. Follow the documentation below to enable these new features on your Azure Subscription

Azure/AKS

Please be aware, enabling preview features takes effect at the Azure subscription level. Do not install preview…

github.com

az extension add --name aks-previewaz feature register -n MultiAgentpoolPreview --namespace 
Microsoft.ContainerServiceaz feature register -n VMSSPreview --namespace Microsoft.ContainerService#Then refresh your registration of the AKS resource provider:az provider register -n Microsoft.ContainerService

Terraform 0.12 and Visual Studio Code

I’ve decided to use Terraform 0.12 to create the infrastructure on Azure since it now supports first-class expression syntax and iteration constructs among a bunch of new features.

Before we dive in the Terraform code, here are a few tools I use to easily edit my Terraform code: Visual Studio Code for Mac + some handy extensions.

Azure Terraform Extension for VS Code: Allows you to run Terraform command from VSCode locally or directly on the Azure Cloud Shell

Terraform for VS Code: Supports code completion, formatting, linting, syntax highlighting, definition…

1. Creating the Infra as Code with Terraform

I’ve setup a Git Repository containing the supporting Terraform files to deploy an AKS cluster including Advanced networking with Azure CNI which enables integration with existing Azure resources within a VNet.

To correctly maintain the state of our Terraform deployment we will require a few resources upfront:

A storage account to act as a backend for Terraform: A “backend” in Terraform determines how state is loaded and how an operation such as apply is executed. Having a remote backend such as an Azure Storage Account has several benefits including working with multiple team members or performing long running remote operations (e.g Azure Pipeline for instance)
An Azure KeyVault to store secrets and sensitive information, we will store a SAS token for the storage account in the Keyvault
A Service Principal (required for AKS)
An SSH Key (required for the Linux VM / AKS nodes)

I’ve created an Azure Resource Group called AksTerraform-RG containing the two above mentioned Azure resources. We need to perform a few more steps to store sensitive data in KeyVault and minimize the risk of leaking a secret in a configuration file.

#creating the resource Group
az group create -n AksTerraform-RG -l eastus2
Location    Name
----------  ---------------
eastus2     AksTerraform-RG#creating the storage account
az storage account create -n <YOUR_STORAGE_ACCOUNT_NAME> -g AksTerraform-RG -l eastus2#creating a tfstate container
az storage container create -n tfstate --account-name <YOUR_STORAGE_ACCOUNT_NAME>#creating the KeyVault
az keyvault create -n <YOUR_KV_NAME> -g AksTerraform-RG -l eastus2
Location    Name            ResourceGroup
----------  --------------  ---------------
eastus2     <YOUR_KV_NAME>  AksTerraform-RG#Creating a SAS Token for the storage account, storing in KeyVault
az storage container generate-sas --account-name <YOUR_STORAGE_ACCOUNT_NAME> --expiry 2020-01-01 --name tfstate --permissions dlrw -o json | xargs az keyvault secret set --vault-name <YOUR_KV_NAME> --name TerraformSASToken --value#creating a Service Principal for AKS and Azure DevOps
az ad sp create-for-rbac -n "AksTerraformSPN"#creating an ssh key if you don't already have one
ssh-keygen  -f ~/.ssh/id_rsa_terraform#store the public key in Azure KeyVault
az keyvault secret set --vault-name <YOUR_KV_NAME> --name LinuxSSHPubKey -f ~/.ssh/id_rsa_terraform.pub > /dev/null#store the service principal id in Azure KeyVault
az keyvault secret set --vault-name <YOUR_KV_NAME> --name spn-id --value <SPN_ID> /dev/null#store the service principal secret in Azure KeyVault
az keyvault secret set --vault-name <YOUR_KV_NAME> --name spn-secret --value <SPN_SECRET> /dev/null

Make sure that you keep track of the generated AppID, Password and Tenant.

You should now have a Resource Group containing a Storage Account + container and a KeyVault

The KeyVault should contain four secrets:

You also need to grant Get access to the secret for the SPN we created

Terraform files

Let’s now have a look at the Terraform files required to create the infrastructure (Vnet + Subnet, AKS Cluster, Azure Container Registry)

I’ve decided to split each resource in separate terraform files, the variable definitions are also isolated in dedicated files prefixed with var-. The code is located on my github at https://github.com/nyuen/AKS-Articles

acr.tf & var-acr.tf : creates the Azure Container registry
aks_cluster.tf & var-aks.tf: creates the AKS cluster with Advanced Networking (CNI + Azure)
aks_vnet.tf & var-vnet.tf: creates the Vnet and Subnet for the AKS cluster
data_source.tf : Links the KeyVault data (the four secrets created)
backend.tf : defines the Azure storage account as a backend for the Terraform State
You will need to create an aks_conf.tfvars: this file contains input variable file that you will need to update with your own parameters. *tfvars file should not be stored in a source repository as it often contains sensitive information.

Make sure to create a aks_conf.tfvars to input your configuration with your:

resource group name (I chose a different resource group name to create the cluster)
cluster name
azure region
kubernetes version
KeyVault reference (KeyVault resource group and name)

To know which version of AKS is available on Azure you can use the following command:

az aks get-versions --location EastUS2

KubernetesVersion    Upgrades
-------------------  ------------------------
1.14.5               None available
1.13.9               1.14.5
1.12.8               1.13.9
1.12.7               1.12.8, 1.13.9
1.11.10              1.12.7, 1.12.8
1.11.9               1.11.10, 1.12.7, 1.12.8
1.10.13              1.11.9, 1.11.10
1.10.12              1.10.13, 1.11.9, 1.11.10

Unfortunately the Terraform Backend resource does not support variables, we will need to pass the parameter in the init method to correctly configure the backend in our CI/CD pipeline

terraform init \
    -backend-config="resource_group_name=<RG_containing_your_storage_account>" \
    -backend-config="storage_account_name=<YOUR_SA_NAME>" \
    -backend-config="container_name=tfstate"

We can now verify that the Terraform files are valid with the command line below

terraform validate -var-file=aks_conf.tfvarsSuccess! The configuration is valid.

2. Configuring Azure DevOps

Now that we have all the resource available to create an infrastructure with Terraform it’s time to automate the process. I’m using Azure Pipelines and the newly released YAML multi-stage definition which allows both CI and CD stages to be defined as a single YAML file. You can find more info on this new feature in the Azure DevOps documentation: https://www.azuredevopslabs.com/labs/azuredevops/yaml/

Let’s first start by creating a new Project on Azure DevOps, I’ve called this project AKS-AzureDevOps, to do so login on dev.azure.com

Installing a Terraform extension

I decided to use a MarketPlace extension for Azure DevOps : https://marketplace.visualstudio.com/items?itemName=charleszipp.azure-pipelines-tasks-terraform

This extension will simplify the use of Terraform in the pipeline by wrapping commands and installing the Terraform CLI on the Agent.

Setting up a basic YAML pipeline

Let’s go straight to Azure Pipeline to create our deployment.

My repo is on a public GitHub, therefore I’ve selected GitHub YAML

Azure Pipeline will automatically create a new YAML file based on the content of my repository, in this specific scenario I don’t have much code so the pipeline will be fairly basic to start with

Clicking save and run will commit the basic pipeline to my repository and run a first pipeline

Obviously the first run is successful considering that so the pipeline is very basic right now.

Connecting Azure DevOps with Azure (with Service Principal)

Before we start to modify the YAML file to deploy our infrastructure we need to configure Azure DevOps to be able to create resources on Azure using the Service Principal that we generated. Add a service connection on Project Settings->Service Connections->New Service Connection -> Azure Resource Manager

Since we already have a Service Principal created in step 1 we don’t need to let Azure DevOps create a new one, use the Advanced Service Connection Editor instead:

You will be able to manage Azure resources through Azure DevOps once the ARM connection is validated:

Enable the New Multi-stage pipeline

Stages are the major divisions in a pipeline: “build this app”, “run these tests”, and “deploy to pre-production” are good examples of stages. They are a logical boundary in your pipeline at which you can pause the pipeline and perform various checks. Since we are editing our pipeline using the new YAML experience we need to enable the Multi-Stage preview feature in the user settings:

We had to ho through a few steps to get there but Azure Pipeline is now properly configured to deploy our resources on Azure.

3. Deploy the infrastructure with Terraform and Azure Pipelines

Adding the .tfvars file to the Azure Pipelines secure file library

I’ve added the *.tfvars pattern to my .gitignore to make sure secrets or sensitive data are not leaked. You can either store sensitive information in Azure KeyVault or leverage the Secure File Library from Azure DevOps. Below is a screenshot of my aks_conf.tfvars :

Here is the code of the aks_conf.tfvars

azure_region = "EastUS"
kubernetes_version = "1.14.5"
resource_group = "<YOUR_VALUE>"
acr_name = "<YOUR_VALUE>"
keyvault_rg = "<YOUR_VALUE>"
keyvault_name = "<YOUR_VALUE>"

The secure library is accessible on the right hand side of the Azure Pipelines menu, you’ll need to allow access to the secure file to your pipeline:

Modifying the Pipeline YAML file

It’s now time to update the pipeline to perform the following Terraform tasks using the ARM connection:

Terraform init
Terraform validate
Terraform plan
Terraform apply

We are going to split these commands in two stages : Validation and Deployment, here is the structure of our CI/CD pipeline

The Validation phase will perform Terraform init, validate while the Deployment phase will perform the init, plan and apply commands

Let’s have a look at these two phases on azure-pipelines.yaml :

Phase one is all about testing if the terraform files are valid through the terraform validate command

let’s break down the stage:

We first copy the content of the terraform folder as a Pipeline Artifact using the publish feature of Azure Pipelines
We install Terraform on the host using the Installation task from the Terraform Extension
the init phase fetches the required Azure provider and links the deployment to the Backend (Azure Storage account)
The validate phase performs a high level syntax check on the Terraform files

Next let’s move to the deployment phase, I’m using the new deployment job provided by the Multi-stage pipeline experience, a deployment job is a special type of job t(a collection of steps to be run sequentially against the environment).Using deployment job provides some benefits:

Deployment history: You get end-to-end deployment history across pipelines down to a specific resource and status of the deployments for auditing.
Apply deployment strategy: Define how your application is rolled-out

The deployment stage is also located in the YAML file:

The deployment job is configured with a few parameters:

deployment: the name of the job within the current stage
strategy: as of August 2019, only RunOnce is supported
environment : the name of the targeted environment, it is also possible to add a resource name (example dev.kubernetes)
deploy: contains the steps required to perform the deployment in the targeted environment

the next tasks within the deploy steps are again leveraging the Terraform extension:

init: assume that we have a new agent from the Microsoft hosted pool
plan: outputs a terraform plan to terraform_aks_out
apply: to deploy the resources to Azure

Note that the working directory is set to $(Pipeline.Workspace)/terraform_out/ which maps to the artifact that we generated in the first step. By default, every artifacts generated in previous stages are automatically copied to the next stage, you can specify a specific artifact or none using the -download directive in yaml.

After running the pipeline you should have two successful stages : validation and Dev (as per the environment name set in the deployment job)

You can then navigate to the Environments tab and check the output of the deployment:

On the Azure side we now have our resources up and running including:

an AKS cluster with 3 nodes using VMSS, Advanced networking with Azure CNI
an Azure Container registry
a VNet + Subnet

The nodes (Azure IaaS Vms) are located on a dedicated Resource Group

Navigating to the VNet on the initial Resource-Group, we can see that the subnet is already provisioned with 300 devices. This is the expected behaviour when using Azure CNI, we configured the cluster to support 100 pods per node and we have a total of 3 nodes : 3 x 100 = 300. The subnet is provisioned to secured the IPs upfront

The storage account for the backend now contains the deployment state of our resources

Conclusion

In this article we’ve configured Azure to support AKS preview features, setup an Azure DevOps project and a Multi-stage pipeline in YAML to deploy an AKS cluster on Azure. The deployment process is secured (KeyVault and Azure Pipelines secret files) and repeatable (CI/CD + Azure Backend for Terraform).

The next article will focus on deploying a sample Kubernetes application to this AKS instance + ACR using a dedicated CI/CD multi-stage pipeline.