The Definitive Guide to Kubernetes Monitoring with kubectl top

As Kubernetes grows exponentially across organizations of all sizes, one tool has rapidly emerged as a “first responder” for troubleshooting performance issues – the kubectl top command.

With its quick aggregated views into pod and node resource usage, kubectl top enables rapid detection and diagnosis of problems like:

• Identifying “noisy neighbor” pods disrupting other workloads
• Spotting overloaded or unhealthy cluster nodes
• Reviewing usage metrics to right-size infrastructure
• Pinpointing containers hitting resource bottlenecks

However, as usage spreads, many teams hit limitations in what raw top metrics provide. Questions come up like:

• How do the metrics work under the hood?
• What other signals indicate resource saturation?
• How can metrics compare across clusters or times?
• How to configure retention and access controls?

This comprehensive guide aims to unpack everything the Kubernetes Metrics Server and kubectl top provide while laying out best practices for monitoring cluster health. We’ll cover:

• Metrics Server internals
• Advanced troubleshooting workflows
• Long term monitoring alternatives
• Cluster monitoring architecture patterns

So whether you’re just getting started with kubectl top or looking to improve existing usage for large scale production clusters, read on!

Inside the Metrics Server

The kubelet agents on each node expose base resource usage metrics through the Metrics Server, an aggregated API stored in memory. This simple pipeline makes metrics instantly accessible without complex setup.

The Metrics Server collects data like:

• Node CPU/Memory
• Pods CPU/Memory/Filesystem/Network Usage
• Timeseries data at 10-15s intervals

This provides tremendous insight into real-time health, albeit with ephemeral storage lacking long term trends.

Some key design decisions and tradeoffs:

All-In-One Deployment

The Metrics Server runs as a single cluster-wide deployment, collecting metrics from all nodes in one place. This provides high availability by replicating metrics across nodes.

In Memory Metrics

Storing metrics in memory without persistence yields peak responsiveness for “hot data”, critical for detecting transient spikes. Disk IOPS can become a bottleneck for large clusters under load.

HTTP Transport

Pull-based HTTP transports keep things simple avoiding head-of-line blocking that can plague native gRPC. Standardization also eases extensibility.

Autoscaling Sources

Horizontal pod autoscalers specifically leverage the Metrics Server since it provides container-level CPU/memory data without custom configuration.

Now that we understand the basics of how metrics work, let’s walk through some real world troubleshooting examples.

Troubleshooting Workflows

While great for at-a-glance spot checks, kubectl top opens up exponentially more powerful troubleshooting workflows when drilling down into specific metrics. Let’s explore some patterns.

Catching Throttled Containers

Kubernetes can throttle containers exceeding their resource limits. This impacts latency sensitive services. The throttling metric tracks throttling:

$ kubectl top pod <pod_name> --containers

NAME                 CPU              THROTTLING
auth                 100m             50m

Here 50% of cycles were throttled, indicating the container hit CPU limits.

We can track it down further with node metrics:

$ kubectl top nodes

NAME                       CPU(cores)      CPU% 
node01                    713m              71%
node02                    500m              50% 
node03                    920m              92%

node01 is heavily saturated. We should rebalance pods or add capacity.

Without throttling metrics we might miss constrained containers degrading latency.

Diagnosing Frequency of Evictions

The evictions metric tracks container terminations due to resource exhaustion. Unexpected spikes point to underprovisioned quotas:

$ kubectl describe nodes

Name:           node02
Evictions:      4

4 evictions on one node hints at resource scarcity. We dig deeper into namespace quotas:

$ kubectl describe quota computing-quota -n processing 

CPU Limit:          4
CPU Used:           6
Requests exceeding limits!

The processing team hit ceiling, triggering evictions! We expand their CPU quota to resolve, avoiding further disruption.

This workflow demonstrates how metrics at both cluster and pod levels combine for root cause identification.

Reviewing Pressure Condition Events

So how do know which metrics indicate risk of issues before problems start? Pressure condition events help predict saturation headroom across nodes and pods.

The node conditions status fields indicate resource pressure:

$ kubectl describe node node03

Conditions:
  Type             Status  LastHeartbeatTime                 Reason                       Message
  ----             ------  -----------------                 ------                       -------
  MemoryPressure   True    Thu, 17 Mar 2022 15:01:53 +0530   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     True    Thu, 17 Mar 2022 15:01:53 +0530   KubeletHasDiskPressure       kubelet has disk pressure
  PIDPressure      False   Thu, 17 Mar 2022 15:01:53 +0530   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Thu, 17 Mar 2022 15:02:03 +0530   KubeletReady                 kubelet is posting ready status

Here we see both MemoryPressure and DiskPressure at true warning node storage is nearly exhausted, risking workload stability.

Examining pod conditions can detect per container resource constraints:

$ kubectl get pods webapp -o jsonpath=‘{.status.conditions[*].message}‘

memory resource requirements exceeds available memory

This Message field indicates the webapp container was OOM killed due to insufficient memory, something pod top metrics would not expose directly.

Layering predictive pressure alerts with real-time metrics enables avoiding outages through early remediation.

Now that we’ve covered the fundamentals of tapping into kubectl top for troubleshooting, let’s explore how to monitor Kubernetes for the long haul.

Long Term Monitoring

While extremely useful for transient troubleshooting, kubectl top metrics only serve as an ephemeral view lacking historical context. Their in-memory design prevents permanent storage for analysis across time periods.

For long term visibility, monitoring systems like Prometheus store precise timeseries data on disk for extensive graphing and analytics:

Prometheus pulls metrics via exporters, allowing custom application-level metrics. And omniversal scraping provides a unified global query view across all endpoints.

These capabilities enable deeper historical monitoring with dimension not readily available from ephemeral top metrics, including:

Fine Grained Data at Scale

Prometheus samples metrics every 15 seconds allowing analysis of workload changes missed by kubectl top. This granularity scales across clusters with 10,000+ nodes.

Federated Views Across Regions

Global queries can roll up metrics across multiple Kubernetes clusters in separate regions to aggregate usage for consumption planning.

Dashboarding and SLI Analytics

Long term retention enables derived visualizations, alerts, and SLO burn rate calculations unavailable ephemeral data.

Application Performance Monitoring

Custom exporters expose app metrics like RPC rates, error budgets, and latency percentiles for business intelligence.

So Prometheus hypercharges historical visibility. But hitting query limits at planet scale, solutions like Thanos shard and compact data into a distributed store while retaining full Prometheus query semantics.

Now with long term monitoring covered, how do we bring this together into a best practice monitoring architecture?

Cluster Monitoring Design Patterns

While many tools can gather metrics, production grade monitoring requires careful multi-layered design considering:

Storage Planning

• Short term (in-memory metrics for hours)
• Medium term (durable metrics for weeks)
• Long term (compacted metrics for years)

Data Transport

• Push vs Pull
• Buffering & Retry logic
• Multi-tenancy & RBAC

Query Responsiveness

• Caching
• Federated queries
• Parallel scanning (e.g Cloud Bigtable)

Cost Management

• Auto tiering hot/cold metrics
• Right sizing clusters
• Scaling down non-prod

Domain Expertise

• Cluster health (Kubernetes expertise)
• Business logic (App domain knowledge)

Balancing these facets helps build reliable, performant and cost effective visibility. Here is an example setup:

• Metrics Server for in-memory transient autoscaling/troubleshooting using kubectl top
• Prometheus for medium term durable metrics with Kubernetes SLI/SLO analysis
• Thanos long term compacted storage for historical capacity planning
• Logging pipeline for indexing application messages & events
• UI Dashboards to visualize clusters, apps, networks holistically

These open source building blocks provide battle tested foundations for each monitoring layer.

Now we have blueprints combining real-time and time-series visibility at planet scale. Let’s round out our recommendations for effectively leveraging kubectl top.

Best Practices Summary

We’ve covered extensive ground on maximizing Kubernetes visibility through metrics so let’s summarize key lessons into concise best practices:

Set Resource Requests and Limits

Requests/Limits enable scheduling predictability and prevent overprovisioning.

Profile Apps Under Load

Identify real capacity needs through load tests to avoid guesswork.

Version Control Config

All alerting and dashboard rules should have peer reviewed config.

Embrace Conventions

Shared vocabularies, tags and idioms ease understanding across teams.

Validate Monitoring Early

Smoke test signal collection, dashboards, and alerts in dev clusters.

Right Size Retention Windows

Balance storage costs against data resolution needed for different analysis.

Federate Across Regions

Unify metrics from disparate prod clusters under global queries.

Automate Anomaly Detection

Static thresholds get outdated fast. Adaptive detection resists drift.

While only scratching the surface, these tips will help focus tooling investments by aligning monitoring to overarching organizational objectives around capacity planning, disaster recovery and application health.

Conclusion

Kubectl top delivers invaluable visibility by exposing resource metrics instantly through the Kubernetes Metrics Pipeline. While limited to transient in-memory data, these interactive troubleshooting workflows uncover container throttling, node pressure, and other signals identifying instability risks missed by static dashboards.

Combining such tactical visibility with durable long term monitoring lays a foundation for resilient, efficient Kubernetes infrastructure ready to scale dynamically to meet business demands. The metrics roadmap points toward an exciting future with deeper visibility and smarter autonomous healing on the horizon.

So grab kubectl top and go exploring – the insights uncovered may just surprise you!