• 1. 迁移Pod
    • 1.1. 设置节点是否可调度
    • 1.2. 执行kubectl drain命令
    • 1.3. 特别说明
  • 2. kubectl drain 流程图
  • 3. TroubleShooting
  • 4. kubectl drain

    1. 迁移Pod

    1.1. 设置节点是否可调度

    确定需要迁移和被迁移的节点，将不允许被迁移的节点设置为不可调度。

    1. # 查看节点
    2. kubectl get nodes
    4. # 设置节点为不可调度
    5. kubectl cordon <NodeName>
    7. # 设置节点为可调度
    8. kubectl uncordon <NodeName>

    1.2. 执行kubectl drain命令

    1. kubectl drain <NodeName> --force --ignore-daemonsets

    示例：

    1. $ kubectl drain bjzw-prek8sredis-99-40 --force --ignore-daemonsets
    2. node "bjzw-prek8sredis-99-40" already cordoned
    3. WARNING: Deleting pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet: kube-proxy-bjzw-prek8sredis-99-40; Ignoring DaemonSet-managed pods: calicoopsmonitor-mfpqs, arachnia-agent-j56n8
    4. pod "pre-test-pro2-r-0-redis-2-8-19-1" evicted
    5. pod "pre-test-hwh1-r-8-redis-2-8-19-2" evicted
    6. pod "pre-eos-hdfs-vector-eos-hdfs-redis-2-8-19-0" evicted

    1.3. 特别说明

    对于statefulset创建的Pod，kubectl drain的说明如下：

    kubectl drain操作会将相应节点上的旧Pod删除，并在可调度节点上面起一个对应的Pod。当旧Pod没有被正常删除的情况下，新Pod不会起来。例如：旧Pod一直处于Terminating状态。

    对应的解决方式是通过重启相应节点的kubelet，或者强制删除该Pod。

    示例：

    1. # 重启发生`Terminating`节点的kubelet
    2. systemctl restart kubelet
    4. # 强制删除`Terminating`状态的Pod
    5. kubectl delete pod <PodName> --namespace=<Namespace> --force --grace-period=0

    2. kubectl drain 流程图

    安全迁移节点 - 图1

    3. TroubleShooting

    1、存在不是通过ReplicationController, ReplicaSet, Job, DaemonSet 或者StatefulSet创建的Pod（即静态pod，通过文件方式创建的），所以需要设置强制执行的参数--force。

    1. $ kubectl drain bjzw-prek8sredis-99-40
    2. node "bjzw-prek8sredis-99-40" already cordoned
    3. error: unable to drain node "bjzw-prek8sredis-99-40", aborting command...
    5. There are pending nodes to be drained:
    6.  bjzw-prek8sredis-99-40
    7. error: DaemonSet-managed pods (use --ignore-daemonsets to ignore): calicoopsmonitor-mfpqs, arachnia-agent-j56n8; pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet (use --force to override): kube-proxy-bjzw-prek8sredis-99-40

    2、存在DaemonSet方式管理的Pod，需要设置--ignore-daemonsets参数忽略报错。

    1. $ kubectl drain bjzw-prek8sredis-99-40 --force
    2. node "bjzw-prek8sredis-99-40" already cordoned
    3. error: unable to drain node "bjzw-prek8sredis-99-40", aborting command...
    5. There are pending nodes to be drained:
    6.  bjzw-prek8sredis-99-40
    7. error: DaemonSet-managed pods (use --ignore-daemonsets to ignore): calicoopsmonitor-mfpqs, arachnia-agent-j56n8

    4. kubectl drain

    1. $ kubectl drain --help
    2. Drain node in preparation for maintenance.
    4. The given node will be marked unschedulable to prevent new pods from arriving. 'drain' evicts the pods if the APIServer
    5. supports eviction (http://kubernetes.io/docs/admin/disruptions/). Otherwise, it will use normal DELETE to delete the
    6. pods. The 'drain' evicts or deletes all pods except mirror pods (which cannot be deleted through the API server).  If
    7. there are DaemonSet-managed pods, drain will not proceed without --ignore-daemonsets, and regardless it will not delete
    8. any DaemonSet-managed pods, because those pods would be immediately replaced by the DaemonSet controller, which ignores
    9. unschedulable markings.  If there are any pods that are neither mirror pods nor managed by ReplicationController,
    10. ReplicaSet, DaemonSet, StatefulSet or Job, then drain will not delete any pods unless you use --force.  --force will
    11. also allow deletion to proceed if the managing resource of one or more pods is missing.
    13. 'drain' waits for graceful termination. You should not operate on the machine until the command completes.
    15. When you are ready to put the node back into service, use kubectl uncordon, which will make the node schedulable again.
    17. ! http://kubernetes.io/images/docs/kubectl_drain.svg
    19. Examples:
    20.   # Drain node "foo", even if there are pods not managed by a ReplicationController, ReplicaSet, Job, DaemonSet or
    21. StatefulSet on it.
    22.   $ kubectl drain foo --force
    24.   # As above, but abort if there are pods not managed by a ReplicationController, ReplicaSet, Job, DaemonSet or
    25. StatefulSet, and use a grace period of 15 minutes.
    26.   $ kubectl drain foo --grace-period=900
    28. Options:
    29.       --delete-local-data=false: Continue even if there are pods using emptyDir (local data that will be deleted when
    30. the node is drained).
    31.       --dry-run=false: If true, only print the object that would be sent, without sending it.
    32.       --force=false: Continue even if there are pods not managed by a ReplicationController, ReplicaSet, Job, DaemonSet
    33. or StatefulSet.
    34.       --grace-period=-1: Period of time in seconds given to each pod to terminate gracefully. If negative, the default
    35. value specified in the pod will be used.
    36.       --ignore-daemonsets=false: Ignore DaemonSet-managed pods.
    37.   -l, --selector='': Selector (label query) to filter on
    38.       --timeout=0s: The length of time to wait before giving up, zero means infinite
    40. Usage:
    41.   kubectl drain NODE [options]
    43. Use "kubectl options" for a list of global command-line options (applies to all commands).

    参考文档：

    • https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/
    • https://kubernetes.io/docs/tasks/run-application/configure-pdb/
    • https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#drain