Console
Start free

Manage process ID limits

Version 1.16 (latest)keyboard_arrow_down

Overview

A Process ID (PID) limit is a Kubernetes resource constraint on nodes and pods to prevent excessive process creation, which can impact node stability. Apigee Hybrid supports the Kubernetes feature to set process ID limits. This document provides instructions on how to set these limits, with recommendations for values for Apigee services on specific platforms.

When Apigee hybrid users manage their own clusters, setting PID limits in Kubernetes can improve system stability, security, and resource management. This is also consistent with Kubernetes best practices.

Definitions of process ID limits

Process ID limits include Node PID limits and Pod PID limits.

Node PID limits include Kube-reserved PIDs and system-reserved PIDs. The total number of allocatable PIDs is the Kernel max minus the kube-reserved PIDs minus the system-reserved PIDs minus the eviction threshold PIDs:

Kernel max ID limits
    - Kube reserved
    - System reserved
    - Eviction threshold
= Allocatable

The pod PID limit can be configured for the node and shared across all pods within the node.

Prepare to manage the process ID limits

These procedures uses the following environment variables:

export PROJECT_ID=MY_PROJECT_ID
export CLUSTER_NAME=MY_CLUSTER_NAME
export LOCATION=MY_CLUSTER_LOCATION
export APIGEE_NAMESPACE=MY_APIGEE_NAMESPACE # Default: apigee

Verify access

Before configuring the process ID limit, ensure that you have access to edit Kubernetes clusters.

The following steps are for installations on GKE. For other platforms, see the documentation for your platform.

  1. Check if you have roles/container.clusterAdmin in the IAM policy: 
    gcloud projects get-iam-policy ${PROJECT_ID}  \
 --flatten="bindings[].members" \
 --format='table(bindings.role)' \
 --filter="bindings.members:your_account_email"
  2. If you do not have the access, add the role to your account: 
    gcloud projects add-iam-policy-binding ${PROJECT_ID} \
 --member user:your_account_email \
 --role roles/container.clusterAdmin

Verify Existing PID Limits

Verify whether the node has existing PID limits before configuring new limits.

  1. Get a node from the cluster to check values. You will need to check nodes from both the apigee-data and apigee-runtime node pools: 
    kubectl get nodes -n ${APIGEE_NAMESPACE}

    Your output should look similar to the following:

    NAME                                      STATUS   ROLES    AGE    VERSION
gke-my-hybrid-apigee-data-0a1b2c3d-efgh      Ready       2d8h   v1.31.5-gke.1169000
gke-my-hybrid-apigee-runtime-1b2c3d4e-fghi   Ready       2d8h   v1.31.5-gke.1169000
  2. Export the node name from the output of the previous step. Run this the next step first for an apigee-data node and again for an apigee-runtime node:

    Code

    export NODE_NAME=MY_NODE_NAME

    Example

    export NODE_NAME="gke-my-hybrid-apigee-data-0a1b2c3d-efgh"
  3. Check the node PID limits. Use the following commands to check the reserved values. If the value is null, the value is not configured: 
    kubectl get --raw "/api/v1/nodes/${NODE_NAME}/proxy/configz" | jq '.kubeletconfig.kubeReserved'
kubectl get --raw "/api/v1/nodes/${NODE_NAME}/proxy/configz" | jq '.kubeletconfig.systemReserved'
kubectl get --raw "/api/v1/nodes/${NODE_NAME}/proxy/configz" | jq '.kubeletconfig.evictionHard'
  4. Check the pod PID limit. Use the following command to check the existing pod PID limit. If the returned value is -1 or empty, the limit is not set: 
    kubectl get --raw "/api/v1/nodes/${NODE_NAME}/proxy/configz" | jq '.kubeletconfig.podPidsLimit'

Manage the process ID limits

Manage the Node PID limits

For installations on GKE, the infrastructure resources of Kubernetes nodes are managed internally and you do not need to configure them. The current capacity and allocatable resources can be viewed following Node allocatable resources in the Google Kubernetes Engine documentation.

For non-GKE platforms, refer to corresponding Kubernetes documentation for the platform. If the cluster/node is user-managed (as opposed to Fully-managed), the kube-reserved PID limit and system-reserved PID limit can be configured with Kubelet. See Node PID limits in the Kubernetes documentation.

Tooling

This procedure uses Kubelet to manage the process ID limit. Kubelet is the agent that runs on pods and containers that ensures they are running according to their PodSpec. If you need to install Kubelet, follow the instructions in the Kubernetes documentation: Installing kubeadm, kubelet and kubectl.

Procedure

  1. Create a Kubelet configuration file called kubelet-config.yaml. 
    apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
kubeReserved:
 pid: PID_VALUE # Example: 1000

    See more info on the configurations: Kube Reserved in the Kubernetes documentation.

  2. Apply the Kubelet configuration: 
    kubelet --config PATH_TO_KUBELET_CONFIG_YAML

Manage the Pod PID limits

Choosing the Limit

If the PID limit is set too low, it can prevent a pod from starting up. If it is set too high, it can fail to detect resource misbehaviors. When selecting the appropriate limit, it is important to consider the previous behavior of the node and any service-specific requirements.

GKE has a required range for the value: [1024, 4194304]. On GKE Platforms you can see the status of your Kubernetes service accounts in the Google Cloud console Metrics Explorer. Select Kubernetes Node - PID usage metrics and apply filters. This metrics shows the recent usage of process IDs and can be referenced when choosing a PID limit.

Metrics Explorer

On non-GKE platforms, different monitoring options may be available. Please refer to the corresponding platform's Kubernetes documentation to check the metrics.

Process ID requirements for Apigee pods

Apigee Hybrid uses 2 node pools, apigee-data and apigee-runtime. Since some of the Apigee components are deployed in both node pools, the Pod PID limit shall be the same for the two node pools. The Pod PID limit should also be higher than the maximum number of PIDs required across all Apigee pods. The required Apigee Pod PID limit is 1000, which is below the minimum required number for GKE platforms.

Some platforms enforce a minimum value requirement for the number of the Pod PIDs limit, in which case the min value requirement is selected.

Platform Min Pod PID Limit
GKE on Google Cloud 1024
GKE on AWS 1024
GKE on Azure 1024
Google Distributed Cloud (software only) on VMware 1024
Google Distributed Cloud (software only) on bare metal 1024
EKS 1000
AKS 1000
OpenShift 1000
Rancher Kubernetes Engine (RKE) 1000

Procedure

The procedures for managing the Pod PID limits are different for GKE and non-GKE platforms.

GKE platform

GKE platform that supports PID limit update includes:

The Pod PID limit is controlled by Node system configurations. GKE has a required range for the value: [1024, 4194304]. For more information, see NodeKubeletConfig.

  1. Create a node system configuration called node-config.yaml with designated pod PID limit with the following contents: 
    kubeletConfig:
 podPidsLimit: POD_PID_VALUE # Example: 1024
  2. Apply the configuration to both the apigee apigee-data and apigee-runtime node pools. When you apply the configuration, the node will start rollout with one of the zero-downtime Node upgrade strategies. 
    gcloud container OPTIONAL_HOST_PLATFORM node-pools update NODE_POOL_NAME \
   --cluster CLUSTER_NAME \
   --region CLUSTER_LOCATION \
   --system-config-from-file=node-config.yaml \
   --project PROJECT_ID

Non-GKE platforms

For non-GKE platforms, the Pod PID limit is controlled by Kubelet. The limit is set by the podPidsLimit field in the Kubelet configuration file.

  1. Create a Kubelet configuration file called kubelet-config.yaml with the following contents: 
    apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
podPidsLimit: POD_PID_VALUE # Example: 1024
  2. Apply the configuration. Setting the podPidsLimit requires rebooting the affected node, which could incur downtime: 
    kubelet --config PATH_TO_KUBELET_CONFIG_YAML
  3. Verify the configuration. See Verify Existing PID Limits for instructions.

Pod PID limits configuration commands and tooling recommendations are different across various platforms. Refer to documentation of respective platforms for detailed commands. Below are some documentation links of non-GKE platforms for reference. Note that they are subject to changes:

Platform Documentation
EKS Customize managed nodes with launch templates
AKS Customize node configuration for Azure Kubernetes Service (AKS) node pools
OpenShift Risks of setting higher process ID limits for Red Hat OpenShift Service on AWS pods
Rancher Kubernetes Engine (RKE) Access a Cluster with Kubectl and kubeconfig

Troubleshooting process ID limits

Pod stuck in Pending status with a FailedScheduling error

When the Node or Pod PIDs limit evicts or restricts a pod from starting up, the pod will be stuck in Pending status and fail with FailedScheduling error.

  1. Get the Node column: 
    kubectl get pods -n ${APIGEE_NAMESPACE} ${POD_NAME} -o wide
  2. Check if there's a PIDPressure condition: 
    kubectl describe node -n apigee ${NODE_NAME} | grep PIDPressure
  3. Alternatively, check ApigeeDeployment of the corresponding pod. Get the ApigeeDeployment from the result that has the same prefix as the erroring pod. 
    kubectl get ApigeeDeployment -n ${APIGEE_NAMESPACE}
  4. Check if the recent Events has PID-related error messages: 
    kubectl describe ApigeeDeployment -n ${APIGEE_NAMESPACE} ${APIGEE_DEPLOYMENT_NAME}
  5. If the cause is confirmed to be PID limit, follow the steps in Manage the Node PID limits to update the PID limit to a higher value.

Invalid podPidsLimit

When setting the limit for GKE, if the podPidsLimit is beyond limit, error will prompt: 

ERROR: (gcloud.container.node-pools.update) ResponseError: code=400, message=Invalid podPidsLimit : value must be 1024 <= podPidsLimit <= 4194304.

Update the podPidsLimit value to within the required range.