When we create EKS cluster based on offical developer guide or terraform module, we will get default Auto Scaling Groups(ASG) to manage the ec2 worker nodes in EKS cluster, we could adjust the instance count in the ASG to adjust the number of ec2 worker nodes in the EKS cluster, but when ASG is created we could not change the instance type of worker nodes's availability zone(AZ). I have a scenario that I will need to provision very large ec2 instances for a specific job and terminiate the giant ec2 instances after job complete. with the power of EKS and ASG, I could easily achieve the goal.

eks-on-asg

Auto Scaling Launch Configuration

When an EKS cluster is created, there will be at least 1 Auto Scaling Launch Configuration created for the cluster to manage the ec2 worker nodes, we could re-use this Launch Configuration by copy and modify pattern. when copy an exist Launch Configuration we could re-use most of the key information which already worked for EKS cluster and just change the Instance Type to what we want. some key configurations are

  • AMI ID: AWS is using optimised AMI for different verison of kubernetes and each region has its own AMI,you could find the right AMI you could use in this matrix
  • Iam Instance Profile: use to controle aws permission of the worker nodes, normally just re-use it
  • Key name: the key user could use to connect to ec2 instance, normally just re-use it and it's very rare to connect ec2 instnace in EKS env.
  • Instance Type: the ec2 instance type you want, normally we need to change the value to meet our needs

Auto Scaling Groups

After the Auto Scaling Launch Configuration is in place, we could create Auto Scaling Groups based on the Auto Scaling Launch Configuration, there are some config fields we need to pay attention to

  • Desired Capacity: how many ec2 instance you want in this ASG
  • Min and Max: to control the ec2 instances limit , normally we set Min to 0 and Max to the max ec2 instance number we want.
  • Availability Zone(s): which AZ the ec2 instance need to be provision to, before kubernetes 1.12 , Topology-Aware Volume Provisioning is not supported , and AWS EBS could not be mounted across AZ, so be careful when set multiple AZs in the ASG when you need to provision PV and mount to pod.
  • Subnets: this connect to Availability Zone(s) as well , determine which subnets the ec2 need to be provision to
  • Tags: one special tag need to set on the ASG, the tag format is kubernetes.io/cluster/${clusterName}=owned, and must set Tag New Instance to yes, so that when new ec2 instances are provisioned in this ASG, it will have a tag kubernetes.io/cluster/${clusterName}=owned by default and EKS will use this tag to discover new ec2 work nodes in the cluster

Script to provision ec2 nodes to join EKS

we could use some simple shell command to provision the ec2 nodes in ASG and let the join EKS

  1. use this command to determine how many ec2 instances already in the ASG
aws --profile=vipmind-prod \
    autoscaling describe-auto-scaling-groups \
    --auto-scaling-group-name ${asg_group_name} \
    | jq '.AutoScalingGroups  | .[0] | .Instances | .[] | .InstanceId' \
    | grep 'i-' \
    | wc -l \
    | sed 's/ //g'
  1. use this command to provisoin new ec2 instances in ASG
aws --profile=vipmind-prod \
    autoscaling set-desired-capacity --desired-capacity ${desired_worker_node_count} \
    --auto-scaling-group-name ${asg_group_name} \
    --no-honor-cooldown
  1. use this command to determine how many nodes are avaialble in EKS cluster
kubectl --kubeconfig=${HOME}/.kube/kubeconfig_vipmind-prod \
        get nodes \
        | grep -v 'NAME' \
        | wc -l \
        | sed 's/ //g'
  1. wait all the worker nodes to be ready
not_ready_worker_node_count=1
while [[ ${not_ready_worker_node_count} -gt 0 ]]; do
    echo "${not_ready_worker_node_count} not ready,  sleep 5s and check again"
    sleep 5
    not_ready_worker_node_count=`kubectl \
        --kubeconfig=${HOME}/.kube/kubeconfig_vipmind-prod \
        get nodes \
        | grep 'NotReady' \
        | wc -l \
        | sed 's/ //g'`
done
  1. we could also label new created nodes with some tags, so that we could easily determine the ec2 worker nodes usage
node_names=`aws --profile=vipmind-prod \
    ec2 describe-instances \
    --instance-ids ${instance_ids} \
    --query 'Reservations[*].Instances[*].[PrivateDnsName]' \
    | jq '.[] | .[] | .[]' \
    | sed 's/"//g'`

kubectl --kubeconfig=${HOME}/.kube/kubeconfig_vipmind-prod label nodes ${node_names} vipmind/usage=${node_tag}
kubectl --kubeconfig=${HOME}/.kube/kubeconfig_vipmind-prod label nodes ${node_names} aws/asg-group-name=${asg_group_name}

Script to destroy ec2 nodes in EKS

when the specific job complete, we could terminiated the ec2 work nodes in the cluster with one command

aws --profile=vipmind-prod \
    autoscaling set-desired-capacity --desired-capacity 0 \
    --auto-scaling-group-name ${asg_group_name} \
    --no-honor-cooldown

if for some reason, user do not want to terminate all the nodes in a ASG, user could use command like below to destroy node in ASG, see ref

aws --profile=vipmind-prod \
    autoscaling terminate-instance-in-auto-scaling-group \
    --instance-id ${ec2_instance_id} \
    --should-decrement-desired-capacity