Monitoring Kubernetes Control Plane with Prometheus

Monitoring Basics

Nowadays it is easy to get lost in a sea of data. Each system can produce tons of telemetry
data. But how do you make sense of the data and use it effectively to keep track of a
system’s health and availability

Setting Up Prometheus to Monitor Kubernetes

Prometheus can monitor applications running inside Kubernetes and the Kubernetes
control plane. As with any monitoring system, there are many deployment considerations.
This chapter answers the following questions:

Monitoring Mixins

Monitoring Mixins (Mixins) are customizable monitoring packages that can bundle:

• Prometheus alerting rules
• Prometheus recording rules
• Grafana dashboards.

Many people don’t use Mixins—they copy-paste alerting rules, find dashboards from the internet, and manually deploy everything to Prometheus and Grafana. 

Writing Alerting Rules

Writing good alerts is a challenging task. Unfortunately, many folks create alerts that on-call people learn to ignore. Some of the problems include the following:

• Creating unactionable alerts 
• Creating too many alerts that all fire together
• Creating flaky alerts that fire often and quickly self-close

Kubernetes Metrics and Stability

This chapter covers Kubernetes metrics, and goes into more detail on Kubernetes service level indicators and service level objectives.

Kubernetes Architecture Overview

This chapter covers Kubernetes architecture and it's design principles that helps to build a resilient system.

Monitoring the Kubernetes API Server

This chapter covers the inside workings of the Kubernetes API Server. Learn how  it works, the internals of various algorithms and how to configure it to make the Kubernetes API Server production-ready. Additionally, we explore SLOs, common issues and most important metrics.

Monitoring etcd

etcd is a strongly consistent, distributed key–value store that reliably stores data. Kubernetes uses it as the database to store all the cluster resources, such as Deployments and Services. In Kubernetes, all requests to etcd go through the Kubernetes API server, which connects to etcd via the gRPC API.

Monitoring the Kubernetes Controller Manager

The Kubernetes controller manager is a daemon that embeds the core Kubernetes controllers. The controllers watch the cluster’s shared state through the Kubernetes API server and command changes to move the current state toward the desired state.

Monitoring the Kubernetes Scheduler

The Kubernetes Scheduler is a service that assigns Pods to Nodes so that Kubelet can run them. The Scheduler watches for pending Pods with no Node assigned and puts them into a scheduling queue. Then it filters suitable Nodes according to various constraints and available resources. At first glance, this task might seem trivial, but there are numerous factors to consider...

Monitoring Kubelet

The Kubelet is a Node daemon that registers itself with the Kubernetes API server and manages Pods for that Node. It’s important to know that Kubelet works in terms of Pods rather than Deployments or containers. It watches the Pods assigned to it and ensures that they are
healthy and running. There are many things Kubelet needs to do to manage and run Pods.

Monitoring Kube Proxy

The Kube Proxy is a service that runs on every Node and manages Node-level networking rules and routes. These routes make the Kubernetes Service object live—routing and load- balancing traffic between the Pods.