AWS: EKS - 3

EKS - Part 3

Cluster Autoscaler

The Kubernetes Cluster Autoscaler automatically adjusts the number of nodes in your cluster when pods fail to launch due to lack of resources or when nodes in the cluster are underutilized and their pods can be rescheduled onto other nodes in the cluster.

Strategy of auto scaling

  • Stateful application

If you are running a stateful application across multiple Availability Zones that is backed by Amazon EBS volumes and using the Kubernetes Cluster Autoscaler, you should configure multiple node groups, each scoped to a single Availability Zone.

  • Other option

Create a single node group that spans multiple Availability Zones.

Single managed node group

Create an Amazon EKS cluster with a single managed node group

eksctl create cluster --name pg-smng --version 1.15 --managed --asg-access

Node group per AZ

Create a cluster with a dedicated managed node group for each Availability Zone

eksctl create cluster --name pg-ngaz --version 1.15 --without-nodegroup

For each Availability Zone in your cluster, use the following eksctl command to create a node group.

eksctl create nodegroup \
        --cluster pg-ngaz \
        --node-zones ap-southeast-2a \
        --name ap-southeast-2a \
        --asg-access \
        --node-type t3.medium \
        --nodes-min 1 --nodes 1 \
        --nodes-max 3 --managed
  • Node Group IAM Policy

The Cluster Autoscaler requires the following IAM permissions to make calls to AWS APIs on your behalf. The tool eksctl automatically provides and attaches to your worker node IAM roles, when it creates the node groups.

    "Version": "2012-10-17",
    "Statement": [
            "Action": [
            "Resource": "*",
            "Effect": "Allow"

Deploy Autoscaler

Deploy the Cluster Autoscaler to your cluster with the following command.

kubectl apply -f

Add the annotation to the deployment with the following command.

kubectl -n kube-system annotate deployment.apps/cluster-autoscaler"false"

Edit the Cluster Autoscaler deployment with the following command.

kubectl -n kube-system edit deployment.apps/cluster-autoscaler  

Edit the cluster-autoscaler container command to replace with your cluster’s name, and add the following options.


Final change will look like below

  - command:
    - ./cluster-autoscaler
    - --v=4
    - --stderrthreshold=info
    - --cloud-provider=aws
    - --skip-nodes-with-local-storage=false
    - --expander=least-waste
    - --balance-similar-node-groups
    - --skip-nodes-with-system-pods=false

Set the Cluster Autoscaler image tag

kubectl -n kube-system set image deployment.apps/cluster-autoscaler

Log Autoscaler

kubectl -n kube-system logs -f deployment.apps/cluster-autoscaler

Add scale policy

Horizontal Pod Autoscaler

The Kubernetes Horizontal Pod Autoscaler automatically scales the number of pods in a deployment, replication controller, or replica set based on that resource’s CPU utilization. This can help your applications scale out to meet increased demand or scale in when resources are not needed, thus freeing up your worker nodes for other applications. When you set a target CPU utilization percentage, the Horizontal Pod Autoscaler scales your application in or out to try to meet that target.

The Horizontal Pod Autoscaler is a standard API resource in Kubernetes that simply requires that a metrics source (such as the Kubernetes metrics server) is installed on your Amazon EKS cluster to work.

Install metrics-server with curl and jq

DOWNLOAD_URL=$(curl -Ls "" | jq -r .tarball_url)
DOWNLOAD_VERSION=$(grep -o '[^/v]*$' <<< $DOWNLOAD_URL)
curl -Ls $DOWNLOAD_URL -o metrics-server-$DOWNLOAD_VERSION.tar.gz
mkdir metrics-server-$DOWNLOAD_VERSION
tar -xzf metrics-server-$DOWNLOAD_VERSION.tar.gz --directory metrics-server-$DOWNLOAD_VERSION --strip-components 1
kubectl apply -f metrics-server-$DOWNLOAD_VERSION/deploy/1.8+/

Horizontal Autoscale Test

  • Install httpd pod

    kubectl run httpd \
    --generator=run-pod/v1 \
    --image=httpd --requests=cpu=100m \
    --limits=cpu=200m --expose --port=80
  • Run benchmark test

    kubectl run apache-bench \
        --generator=run-pod/v1 \
        -i --tty --rm --image=httpd \
        -- ab -n 900000 \
        -c 9999 http://httpd.default.svc.cluster.local/

Vertical Pod Autoscaler

The Kubernetes Vertical Pod Autoscaler automatically adjusts the CPU and memory reservations for your pods to help “right size” your applications. This adjustment can improve cluster resource utilization and free up CPU and memory for other pods. This topic helps you to deploy the Vertical Pod Autoscaler to your cluster and verify that it is working.

Deploy Vertical Autoscaler

Open a terminal window and navigate to a directory where you would like to download the Vertical Pod Autoscaler source code.

  • Clone the kubernetes/autoscaler GitHub repository.

    git clone
  • Change to the vertical-pod-autoscaler directory.

    cd autoscaler/vertical-pod-autoscaler/
  • (Optional) If you have already deployed another version of the Vertical Pod Autoscaler, remove it with the following command.

  • Deploy the Vertical Pod Autoscaler to your cluster with the following command.

  • Check Vertical Autoscaler pods

    kubectl get pods -n kube-system | grep vpa

Test Vertical Autoscaler

  • Deploy the hamster.yaml Vertical Pod Autoscaler example with the following command.

    kubectl apply -f examples/hamster.yaml
  • Get the pods from the hamster example application.

    kubectl get pods -l app=hamster
    # Output:
    hamster-c7d89d6db-rglf5   1/1     Running   0          48s
    hamster-c7d89d6db-znvz5   1/1     Running   0          48s
  • Describe one of the pods to view its CPU and memory reservation.

    kubectl describe pod hamster-c7d89d6db-rglf5
  • Describe the hamster-vpa resource to view the new recommendation.

    kubectl describe vpa/hamster-vpa
    # Output
        Last Transition Time:  2020-02-11T13:31:48Z
        Status:                True
        Type:                  RecommendationProvided
        Container Recommendations:
        Container Name:  hamster
        Lower Bound:
            Cpu:     530m
            Memory:  262144k
            Cpu:     587m
            Memory:  262144k
        Uncapped Target:
            Cpu:     587m
            Memory:  262144k
        Upper Bound:
            Cpu:     1
            Memory:  500Mi