AWS: EKS - 5

EKS - Part 5

Metrics Server

The Kubernetes metrics server is an aggregator of resource usage data in your cluster, and it is not deployed by default in Amazon EKS clusters. The metrics server is commonly used by other Kubernetes add ons, such as the Horizontal Pod Autoscaler or the Kubernetes Dashboard.

  • Deploy the metrics server

    kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.6/components.yaml

  • Verify that the metrics-server deployment

    kubectl get deployment metrics-server -n kube-system

Prometheus

The Kubernetes API server exposes a number of metrics that are useful for monitoring and analysis. These metrics are exposed internally through a metrics endpoint that refers to the /metrics HTTP API. Like other endpoints, this endpoint is exposed on the Amazon EKS control plane.

Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud. Since its inception in 2012, many companies and organizations have adopted Prometheus, and the project has a very active developer and user community. It is now a standalone open source project and maintained independently of any company. To emphasize this, and to clarify the project’s governance structure, Prometheus joined the Cloud Native Computing Foundation in 2016 as the second hosted project, after Kubernetes.

Deploying Prometheus

  • Create a Prometheus namespace.

    kubectl create namespace prometheus

  • Install helm

    ## MAC
brew install helm

## Linux
curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 > get_helm.sh
chmod 700 get_helm.sh
./get_helm.sh

## Add stable repo to helm
helm repo add stable https://kubernetes-charts.storage.googleapis.com/

  • Deploy Prometheus.

    helm install prometheus stable/prometheus \
        --namespace prometheus \
        --set alertmanager.persistentVolume.storageClass="gp2",server.persistentVolume.storageClass="gp2"

  • Verify that all of the pods in the prometheus namespace are in the READY state.

    kubectl get pods -n prometheus

Grafana

Set the storage class to gp2, admin password, configuring the datasource to point to Prometheus and creating an external load balancer for the service.

kubectl create namespace grafana
helm install grafana stable/grafana \
        --namespace grafana \
        --set persistence.storageClassName="gp2" \
        --set adminPassword='grafana' \
        --set datasources."datasources\.yaml".apiVersion=1 \
        --set datasources."datasources\.yaml".datasources[0]    name=Prometheus \
        --set datasources."datasources\.yaml".datasources[0]    type=prometheus \
        --set datasources."datasources\.yaml".datasources[0]    url=http://prometheus-server.prometheus.svc.cluster.local \
        --set datasources."datasources\.yaml".datasources[0]    access=proxy \
        --set datasources."datasources\.yaml".datasources[0]    isDefault=true \
        --set service.type=LoadBalancer

Get your ‘admin’ user password

kubectl get secret --namespace grafana grafana \
    -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

The Grafana server can be accessed via port 80 on the following DNS name from within your cluster: grafana.grafana.svc.cluster.local

Get the Grafana URL to visit by running these commands in the same shell:

 export SERVICE_IP=$(kubectl get svc --namespace grafana grafana -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
 http://$SERVICE_IP:80

Import dashboard

  • Cluster Monitoring Dashboard

    • Click ’+’ button on left panel and select ‘Import’.
    • Enter 3119 dashboard id under Grafana.com Dashboard.
    • Click ‘Load’.
    • Select ‘Prometheus’ as the endpoint under prometheus data sources drop down.
    • Click ‘Import’.

  • Pods Monitoring Dashboard

    • Click ’+’ button on left panel and select ‘Import’.
    • Enter 6417 dashboard id under Grafana.com Dashboard.
    • Click ‘Load’.
    • Enter Kubernetes Pods Monitoring as the Dashboard name.
    • Click change to set the Unique identifier (uid).
    • Select ‘Prometheus’ as the endpoint under prometheus data sources drop down.s
    • Click ‘Import’.

Kubernetes Dashboard

  • Deploy the Kubernetes Metrics Server

  • Deploy the Dashboard

    kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta8/aio/deploy/recommended.yaml

  • Create an eks-admin Service Account and Cluster Role Binding

cat<<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: eks-admin
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: eks-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: eks-admin
  namespace: kube-system
EOF

  • Get login token

    kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep eks-admin | awk '{print $1}')

  • Start the kubectl proxy

    kubectl proxy

  • Access dashboard via browser

    http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/#!/login.

  • Use token from above to login

  • Expose the dashboard to public

cat<<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"creationTimestamp":"2020-04-12T12:34:10Z","labels":{"k8s-app":"kubernetes-dashboard"},"name":"kubernetes-dashboard","namespace":"kubernetes-dashboard","resourceVersion":"380715","selfLink":"/api/v1/namespaces/kubernetes-dashboard/services/kubernetes-dashboard","uid":"31489c5f-2dff-4b88-9a36-46b248bf9ce2"},"spec":{"externalTrafficPolicy":"Cluster","ports":[{"port":80,"protocol":"TCP","targetPort":8443}],"selector":{"k8s-app":"kubernetes-dashboard"},"sessionAffinity":"None","type":"LoadBalancer"},"status":{"loadBalancer":{}}}
  creationTimestamp: "2020-04-12T13:23:52Z"
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
  resourceVersion: "402339"
  selfLink: /api/v1/namespaces/kubernetes-dashboard/services/kubernetes-dashboard
  uid: d29bab71-d159-4c33-9ba1-00f05138ecb6
spec:
  externalTrafficPolicy: Cluster
  ports:
  - nodePort: 30556
    port: 443
    protocol: TCP
    targetPort: 8443
  selector:
    k8s-app: kubernetes-dashboard
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:{}