1 pods are unevictable from node eksctl

2023. 6. 16. 17:20Dev/EKS

728x90

How to solve "1 pods are unevictable from node in eksctl".

I tried to delete the AWS EKS cluster with the eksctl command.

Problem: eksctl delete fail (pending, stuck, block)

eksctl delete cluster -f host.yaml
2023-06-16 16:41:12 [ℹ]  deleting EKS cluster "host"
2023-06-16 16:41:12 [ℹ]  will drain 0 unmanaged nodegroup(s) in cluster "host"
2023-06-16 16:41:12 [ℹ]  starting parallel draining, max in-flight of 1
2023-06-16 16:41:13 [ℹ]  cordon node "ip-192-168-25-101.ap-northeast-2.compute.internal"
2023-06-16 16:41:13 [ℹ]  cordon node "ip-192-168-43-78.ap-northeast-2.compute.internal"
2023-06-16 16:41:13 [ℹ]  cordon node "ip-192-168-76-53.ap-northeast-2.compute.internal"
2023-06-16 16:42:25 [!]  1 pods are unevictable from node ip-192-168-25-101.ap-northeast-2.compute.internal
2023-06-16 16:43:26 [!]  1 pods are unevictable from node ip-192-168-25-101.ap-northeast-2.compute.internal
2023-06-16 16:44:26 [!]  1 pods are unevictable from node ip-192-168-25-101.ap-northeast-2.compute.internal
2023-06-16 16:45:27 [!]  1 pods are unevictable from node ip-192-168-25-101.ap-northeast-2.compute.internal
2023-06-16 16:46:28 [!]  1 pods are unevictable from node ip-192-168-25-101.ap-northeast-2.compute.internal
2023-06-16 16:47:28 [!]  1 pods are unevictable from node ip-192-168-25-101.ap-northeast-2.compute.internal
2023-06-16 16:48:29 [!]  1 pods are unevictable from node ip-192-168-25-101.ap-northeast-2.compute.internal
2023-06-16 16:49:29 [!]  1 pods are unevictable from node ip-192-168-25-101.ap-northeast-2.compute.internal
2023-06-16 16:50:30 [!]  1 pods are unevictable from node ip-192-168-25-101.ap-northeast-2.compute.internal

(Interrupt and terminate a pending eksctl command.) ctrl + c

Cause: ebs-csi-controller's pdb (poddisruptionbudget)

2023-06-16 17:04:30 [!]  1 pods are unevictable from node ip-192-168-25-101.ap-northeast-2.compute.internal
2023-06-16 17:05:32 [!]  1 pods are unevictable from node ip-192-168-25-101.ap-northeast-2.compute.internal
2023-06-16 17:06:32 [!]  1 pods are unevictable from node ip-192-168-25-101.ap-northeast-2.compute.internal
2023-06-16 17:07:33 [!]  1 pods are unevictable from node ip-192-168-25-101.ap-northeast-2.compute.internal
^C

k get pdb -A
NAMESPACE     NAME                 MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
kube-system   ebs-csi-controller   N/A             1                 0                     5h32m

 

Solution:  delete pdb

k get pdb -A
NAMESPACE     NAME                 MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
kube-system   ebs-csi-controller   N/A             1                 0                     5h32m

k delete pdb ebs-csi-controller -n kube-system
poddisruptionbudget.policy "ebs-csi-controller" deleted

 

Result: eksctl delete works well

eksctl delete cluster -f host.yaml
2023-06-16 17:08:52 [ℹ]  deleting EKS cluster "host"
2023-06-16 17:08:53 [ℹ]  will drain 0 unmanaged nodegroup(s) in cluster "host"
2023-06-16 17:08:53 [ℹ]  starting parallel draining, max in-flight of 1
2023-06-16 17:09:16 [✔]  drained all nodes: [ip-192-168-76-53.ap-northeast-2.compute.internal ip-192-168-25-101.ap-northeast-2.compute.internal ip-192-168-43-78.ap-northeast-2.compute.internal]
2023-06-16 17:09:16 [ℹ]  deleted 0 Fargate profile(s)
2023-06-16 17:09:19 [✔]  kubeconfig has been updated
2023-06-16 17:09:19 [ℹ]  cleaning up AWS load balancers created by Kubernetes objects of Kind Service or Ingress
2023-06-16 17:09:20 [ℹ]
4 sequential tasks: { delete nodegroup "ng-spot", delete IAM OIDC provider,
    2 parallel sub-tasks: {
        delete addon IAM "eksctl-host-addon-aws-ebs-csi-driver" [async],
        delete addon IAM "eksctl-host-addon-vpc-cni" [async],
    }, delete cluster control plane "host" [async]
}
2023-06-16 17:09:20 [ℹ]  will delete stack "eksctl-host-nodegroup-ng-spot"
2023-06-16 17:09:20 [ℹ]  waiting for stack "eksctl-host-nodegroup-ng-spot" to get deleted
2023-06-16 17:09:21 [ℹ]  waiting for CloudFormation stack "eksctl-host-nodegroup-ng-spot"
2023-06-16 17:09:51 [ℹ]  waiting for CloudFormation stack "eksctl-host-nodegroup-ng-spot"
2023-06-16 17:10:42 [ℹ]  waiting for CloudFormation stack "eksctl-host-nodegroup-ng-spot"
2023-06-16 17:12:10 [ℹ]  waiting for CloudFormation stack "eksctl-host-nodegroup-ng-spot"
2023-06-16 17:12:11 [ℹ]  will delete stack "eksctl-host-addon-vpc-cni"
2023-06-16 17:12:11 [ℹ]  will delete stack "eksctl-host-addon-aws-ebs-csi-driver"
2023-06-16 17:12:12 [ℹ]  will delete stack "eksctl-host-cluster"
2023-06-16 17:12:12 [✔]  all cluster resources were deleted