Kubernetes Learning Week Series 14
Container Interference Detection and Mitigation: Part 1
https://zouyee.medium.com/container-interference-detection-and-mitigation-part-1-977d4d99acc8
This article discusses the issue of “neighbor interference” in cloud infrastructure, where resource contention between applications on the same compute node leads to performance degradation. It explores various optimization strategies to mitigate this issue, such as resource requests and limits, affinity and anti-affinity rules, isolation policies, priority and preemption, NUMA-aware scheduling, and dedicated hardware. The article also introduces the technical background of performance metrics, such as CPI (Cycles Per Instruction) and LLC (Last Level Cache), to understand and measure performance interference.
Key Points
The “neighbor interference” problem is a common issue in cloud infrastructure, where resource contention between applications on the same compute node leads to performance degradation.
Kubernetes provides features such as the CPU Manager, Device Plugin Manager, and Topology Manager, but these do not directly solve the “neighbor interference” problem.
Optimization strategies to mitigate “neighbor interference” include resource requests and limits, affinity and anti-affinity rules, isolation policies, priority and preemption, NUMA-aware scheduling, and dedicated hardware.
CPI (Cycles Per Instruction) is a key metric for evaluating the efficiency of compute-intensive tasks and can be used to identify performance interference caused by resource contention.
LLC (Last Level Cache) is a critical resource for job performance, and interference in sharing these resources can lead to significant performance degradation.
Intel’s RDT (Resource Director Technology) provides monitoring and allocation capabilities for cache and memory bandwidth to help address performance interference.
Methods like CPI2 and LIBRA use CPI as a performance proxy and adjust resource allocation accordingly to manage performance interference.
PARTIES utilizes stress testing to determine the optimal latency target for applications and adjusts resources accordingly to ensure the quality of service (QoS) for all applications.
Comparison of Kubernetes Network Solutions
https://machinezone.github.io/research/networking-solutions-for-kubernetes/
This article compares the performance of different Kubernetes network solutions, including Docker’s --net=host, Flannel, and IPvlan, analyzing them primarily in terms of latency, throughput, and setup cost. The results show that IPvlan has the lowest latency and highest maximum throughput, while the combination of Flannel with host-gw performs well in terms of performance characteristics and is easy to set up.
Key Points
Compared network solutions: Docker’s --net=host, Flannel, and IPvlan.
Load testing scenarios: Measuring latency and throughput based on varying request rates and response sizes.
Results: IPvlan demonstrated the lowest latency and highest maximum throughput, while Flannel with host-gw performed well and was chosen as the preferred option.
Test environment setup: Configured NUMA, IRQ, RPS, XPS, and RFS in detail to achieve high performance.
Potential Interview Questions
What are the key performance differences among Kubernetes network solutions?
What factors were considered in the load testing scenarios?
Why was Flannel with host-gw chosen as the network solution?
How was the test environment configured to achieve high performance?
Migration of HPA to KEDA
https://yasintahaerol.medium.com/migrating-hpa-to-keda-13e946ee29ee
This article discusses the process of migrating from the Horizontal Pod Autoscaler (HPA) to Kubernetes Event-Driven Autoscaling (KEDA) and provides a solution for automating the migration process.
Key Points
The author’s team decided to migrate their environment to KEDA because they found HPA insufficient for their needs. The variety of KEDA’s scalers and the ability to create custom scalers were particularly appealing.
The author raises the challenge that while using new scalers is straightforward, they needed a method to move existing HPAs to KEDA.
The author describes a solution where they developed an Operator that automatically creates a ScaledObject when a specific annotation is added to an HPA. This reduces the burden on developers and avoids potential issues related to HPA deletion.
Solution Overview
The author developed an Operator to automate the migration process:
By adding the annotation transfer-hpa: true, the Operator detects the event and creates a ScaledObject.
This reduces the workload for developers, minimizes downtime, and prevents Kubernetes from scaling down to the minimum replica count when HPA is deleted.
Kubernetes Pod Resource Sizing
This article discusses a method for accurately defining CPU and memory resources for Kubernetes Pods by analyzing application requirements, setting up a test environment, and using load testing tools.
Key Points
• Define application requirements, such as response time, throughput, error rate, and resource utilization.
• Set up a Kubernetes cluster and monitoring tools to collect performance metrics.
• Choose a load testing tool (e.g., K6) to define load tests and thresholds.
• Deploy the application with initial resource settings, run load tests, gradually increase the load, and adjust resources until the target is met.
• Apply stress beyond the target load to determine the point at which the application starts to break.
By defining application requirements, setting up a realistic test environment, and using effective load testing tools, resources can be iteratively adjusted to meet performance goals. This systematic approach also helps identify bottlenecks and better plan for production scaling.