Kubernetes Learning Week Series 8
CNI and Network Namespaces — A Complete Guide — Part 1
This article is the first part of a comprehensive guide on container networking using CNI (Container Network Interface) and network namespaces. It introduces the concept of network namespaces and demonstrates how to create and connect two namespaces using iproute2 and veth devices. The guide is divided into five parts, each focusing on different aspects of container networking, including creating CNI plugins and configuring container runtimes.
Key Points:
Introduction to network namespaces and their role in container networking, using tools like Containerd and CRI-O.
Step-by-step guide to creating and connecting two network namespaces using iproute2 and veth devices.
Explanation of how to configure IP addresses and routing to enable communication between namespaces.
Conclusion highlighting the scalability issues of the current approach and a preview of the next section, which focuses on using Bridge interfaces.
Benchmark Results of Kubernetes Network Plugins (CNI) on 40 Gbit/s Networks [2024]
This article presents a comprehensive benchmark of Kubernetes network plugins (CNI) on 40 Gbit/s networks, comparing various CNIs based on performance, efficiency, and resource consumption. It covers testing methodologies, hardware setups, and insights from the results, offering recommendations for selecting a CNI based on specific use cases.
Key Points:
The benchmark tests used Kubernetes 1.26 and Ubuntu 22.04 with CNIs updated as of January 2024.
Seven CNIs were selected for benchmarking based on criteria like active maintenance and support for network policies.
A total of 21 CNI variants were tested, including configurations using WireGuard and eBPF.
The test setup involved three Supermicro servers connected via a 40 Gbit switch, with minimal system tuning to reflect real-world conditions.
The benchmark protocol included multiple tests to assess TCP and UDP bandwidth across different scenarios.
Transitioning from 10 Gbit to 40 Gbit networks posed challenges, such as limitations of single-threaded measurement tools like iperf3.
The statexec tool was used for monitoring, capturing system metrics with minimal resource overhead.
Benchmark results are available as open-source resources for detailed analysis.
Encryption in CNIs generally reduced CPU usage but led to lower performance and bandwidth.
Memory usage was bandwidth-independent, and efficiency was measured using a “CPU efficiency” metric.
The performance differences between CNIs were minor, with eBPF showing slight advantages in TCP multi-stream scenarios.
UDP performance was generally poor, raising concerns about protocols like HTTP/3.
Calico VPP required extensive tuning to achieve optimal performance.
WireGuard outperformed IPsec in efficiency but lacked key rotation functionality.
Recommendations include using Kube-router for low-resource clusters and Cilium for standard clusters.
Calico VPP is suggested for highly tuned clusters with extensive hardware and software optimizations.
Allocatable Resources in Kubernetes Nodes
Pods deployed in a Kubernetes cluster consume memory, CPU, and storage resources. However, not all resources on a Node are available for running Pods. The operating system and kubelet also require memory and CPU, and these additional resource needs must be accounted for. When examining a single node, available resources can be categorized as follows:
Resources required to run the operating system and system daemons (e.g., SSH, systemd, etc.).
Resources needed for Kubernetes agents, such as kubelet, container runtime, and node problem detectors.
Resources available for Pods.
As you might expect, all these allocations are customizable.
Key Points:
Not all CPU and memory on Kubernetes nodes are available for Pods; the OS and Kubernetes agents also consume resources.
Resource allocation within Kubernetes nodes is customizable. Major services like GKE, AKS, and EKS have their own allocation rules.
GKE reserves a specific percentage of memory and CPU based on the node’s total resources.
Larger instances may be more efficient in terms of allocatable resources.
EKS uses code logic to determine reserved resources and limits based on the number of Pods per instance. AWS provides a script to calculate the maximum number of Pods for each EC2 instance.
AKS reserves memory based on a fixed amount per Pod or a percentage of total memory, whichever is lower.
Larger nodes aren’t always cost-effective; consider factors such as overhead, HA strategies, and auto-scaling.
Graceful Shutdown of Pods in Kubernetes
This article provides a detailed explanation of handling graceful shutdowns in Kubernetes. It highlights the importance of preventing connection disruptions when Pods start or stop and discusses techniques for managing long-running tasks and connections gracefully. The article explores the processes involved in creating and deleting Pods, the roles of Kubernetes components like kubelet, and how to effectively manage endpoint updates to avoid downtime.
Key Points:
Discusses how to prevent connection disruptions during Pod startup or shutdown in Kubernetes.
Explains the Pod creation process, including the roles of kubelet, Container Runtime Interface (CRI), Container Network Interface (CNI), and Container Storage Interface (CSI).
Covers how Kubernetes handles endpoints and their critical role in routing traffic.
Highlights the Pod deletion process and the race conditions that can occur if endpoints are not updated promptly.
Recommends using application-level preStop hooks or await mechanisms to ensure graceful shutdowns and avoid race conditions.
Discusses the impact of prolonged shutdowns on cluster autoscaling and suggests keeping shutdown times short.
Proposes using rainbow deployments for applications with long-running tasks to avoid disruptions during updates.
The Fun of Kubernetes Authorization Auditing - Multiple authz Plugins
https://raesene.github.io/blog/2024/04/22/Fun-with-Kubernetes-Authz/
This article explores the challenges of auditing permissions in Kubernetes, particularly when multiple authorization mechanisms are in use. It highlights the difficulties of auditing permissions in clusters using both RBAC and webhook authorizers, pointing out that not all permissions can be viewed through standard tools like kubectl auth can-i --list. The article underscores the need for manual review of each authorization system to ensure comprehensive auditing.
Key Points:
Kubernetes’ flexible security model supports multiple authentication and authorization methods, leading to complexity in auditing permissions.
Kubernetes authorizers are queried sequentially, and effective permissions depend on their responses.
The kubectl auth can-i command is helpful for reviewing permissions but may provide incomplete outputs in clusters with multiple authorizers.
Webhook authorizers do not support listing permissions, so their permissions will not appear in kubectl auth can-i --list.
Effective auditing requires reviewing each authorization system in the cluster, as automated tools may not cover all systems.
K8S Backup Solutions: Velero Implementation Guide
https://weng-albert.medium.com/k8s-backup-solution-velero-implementation-guide-en-ec06cb6b50dd
This article provides a detailed guide to implementing Velero as a backup solution for Kubernetes environments. It highlights Velero’s capabilities, such as backing up and migrating Kubernetes resources and persistent volumes, and explains the deployment process, including integration with Minio for storage. The article also covers the backup and restore process, the use of Custom Resource Definitions (CRDs), and configuring the storage location for backups and volume snapshots. Additionally, it discusses how Velero synchronizes object storage data with Kubernetes and offers step-by-step instructions for deploying and verifying Velero.
Key Points:
Velero is used for backing up and migrating Kubernetes resources and persistent volumes, supporting both on-premises and cloud environments.
Key features include backup and restore capabilities, cluster resource transfer, and production environment replication. It uses CRDs to manage operations and stores backups in object storage.
The backup process involves creating a CRD object called “Backup,” collecting data, and uploading it to object storage. Persistent volume snapshots can be customized with parameters.
Velero allows specifying backup retention periods and provides mechanisms to delete expired backups. Synchronization ensures that backup resources in object storage are reflected in Kubernetes.
Deployment steps include creating a Minio bucket, installing the Velero client, configuring Minio authentication, deploying the Velero server, and verifying the setup.
Backup and restore operations involve creating test backups, verifying results, deleting namespaces, and restoring from backups.
Essential Kubernetes Autoscaling Guide: Monitoring Karpenter with Prometheus
This article discusses the importance of Kubernetes autoscaling and monitoring Karpenter (an open-source project by AWS) using Prometheus. It highlights Karpenter’s role in efficient resource management and the necessity of monitoring its performance to optimize Kubernetes clusters. The article provides real-world examples of issues encountered with Karpenter, emphasizing the need for specific monitoring and alert setups using Prometheus and Grafana to detect and resolve potential problems promptly.
Key Points:
Kubernetes autoscaling is critical for efficient resource management, and Karpenter can dynamically provision and decommission nodes based on demand.
Monitoring Karpenter with Prometheus is essential for gaining deeper insights into Kubernetes autoscaling performance.
A sample issue involves pending Pods caused by an error in the AMI, underscoring the importance of monitoring and quick response.
Setting up Prometheus monitoring includes enabling serviceMonitor in the Karpenter Helm chart and using Grafana for visualization.
Alerts in Prometheus can help detect issues such as unregistered nodes, resource limits, and cloud provider errors.
Proper monitoring and alerts enhance the ability to proactively address anomalies in Kubernetes infrastructure.