|
运行环境
-
Ubuntu 20.04 -
DOCKER 19.03.15 -
KUBEASZ 3.1.0 -
K8S_BIN 1.21.0
1 错误1
1.1 事件还原
由于使用 kubeasz（地址：https://github.com/easzlab/kubeasz）
部署二进制的 k8s ，为了便于操作，我提前将 k8s-master1，k8s-master2，k8s-master3，k8s-node1，k8s-node2，k8s-node3 都安装了 docker，这样在执行 ezctl setup k8s-01 03 的时候，发现每台机器都安装了 docker 就不会再执行安装 docker 的操作了，相应的也不会再修改 /etc/docker/daemon.json 文件，直接导致 docker 运行的命名空间和默认的命名空间不一致，所以无法启动 kubelet 服务
执行 [root@k8s-master1 kubeasz]# ./ezctl setup k8s-01 04
报错如下：
TASK [kube-node : 轮询等待kubelet启动] *******************************************************************************************************
changed: [172.18.60.202]
changed: [172.18.60.201]
FAILED - RETRYING: 轮询等待node达到Ready状态 (8 retries left).
FAILED - RETRYING: 轮询等待node达到Ready状态 (8 retries left).
FAILED - RETRYING: 轮询等待node达到Ready状态 (7 retries left).
FAILED - RETRYING: 轮询等待node达到Ready状态 (7 retries left).
FAILED - RETRYING: 轮询等待node达到Ready状态 (6 retries left).
FAILED - RETRYING: 轮询等待node达到Ready状态 (6 retries left).
FAILED - RETRYING: 轮询等待node达到Ready状态 (5 retries left).
FAILED - RETRYING: 轮询等待node达到Ready状态 (5 retries left).
FAILED - RETRYING: 轮询等待node达到Ready状态 (4 retries left).
FAILED - RETRYING: 轮询等待node达到Ready状态 (4 retries left).
FAILED - RETRYING: 轮询等待node达到Ready状态 (3 retries left).
FAILED - RETRYING: 轮询等待node达到Ready状态 (3 retries left).
FAILED - RETRYING: 轮询等待node达到Ready状态 (2 retries left).
FAILED - RETRYING: 轮询等待node达到Ready状态 (2 retries left).
FAILED - RETRYING: 轮询等待node达到Ready状态 (1 retries left).
FAILED - RETRYING: 轮询等待node达到Ready状态 (1 retries left).
TASK [kube-node : 轮询等待node达到Ready状态] ***************************************************************************************************
fatal: [172.18.60.201]: FAILED! => {"attempts": 8, "changed": true, "cmd": "/usr/bin/kubectl get node 172.18.60.201|awk 'NR>1{print $2}'", "delta": "0:00:00.263088", "end": "2021-09-01 12:26:16.691810", "rc": 0, "start": "2021-09-01 12:26:16.428722", "stderr": "Error from server (NotFound): nodes \"172.18.60.201\" not found", "stderr_lines": ["Error from server (NotFound): nodes \"172.18.60.201\" not found"], "stdout": "", "stdout_lines": []}
fatal: [172.18.60.202]: FAILED! => {"attempts": 8, "changed": true, "cmd": "/usr/bin/kubectl get node 172.18.60.202|awk 'NR>1{print $2}'", "delta": "0:00:00.283975", "end": "2021-09-01 12:26:17.056414", "rc": 0, "start": "2021-09-01 12:26:16.772439", "stderr": "Error from server (NotFound): nodes \"172.18.60.202\" not found", "stderr_lines": ["Error from server (NotFound): nodes \"172.18.60.202\" not found"], "stdout": "", "stdout_lines": []}
PLAY RECAP *****************************************************************************************************************************
172.18.60.201 : ok=53 changed=28 unreachable=0 failed=1 skipped=1 rescued=0 ignored=0
172.18.60.202 : ok=50 changed=26 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
[root@k8s-master1 kubeasz]#
使用 journalctl -u kubelet 查看日志
为了便于查看，我们可以将日志重定向到一个文件中
journalctl -u kubelet > /tmp/kubelet.log
发现错误
Sep 02 08:52:33 k8s-master1.example.com kubelet[612230]: E0902 08:52:33.526931 612230 server.go:292] "Failed to run kubelet" err="failed to run Kubelet: misconfiguration: kubelet cgroup driver: \"systemd\" is different from docker cgroup driver: \"cgroupfs\""
Sep 02 08:52:33 k8s-master1.example.com systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Sep 02 08:52:33 k8s-master1.example.com systemd[1]: kubelet.service: Failed with result 'exit-code'.
1.2 解决办法
[root@k8s-master1 ~]# cat /etc/docker/daemon.json
{
"data-root": "/var/lib/docker",
"exec-opts": ["native.cgroupdriver=systemd"],
"registry-mirrors": [
"https://docker.mirrors.ustc.edu.cn",
"http://hub-mirror.c.163.com"
],
"insecure-registries": ["127.0.0.1/8","172.18.8.214","172.18.8.215"],
"max-concurrent-downloads": 10,
"live-restore": true,
"log-driver": "json-file",
"log-level": "warn",
"log-opts": {
"max-size": "50m",
"max-file": "1"
},
"storage-driver": "overlay2"
}
[root@k8s-master1 ~]#
[root@k8s-master1 ~]# docker info|grep "Cgroup Driver"
WARNING: No swap limit support
Cgroup Driver: systemd
[root@k8s-master1 ~]#
注意：Ubuntu18 的 /etc/docker/daemon.json 文件如下
root@k8s-master-1:~# docker info|grep "Cgroup Driver"
Cgroup Driver: cgroupfs
WARNING: No swap limit support
root@k8s-master-1:~# cat /etc/docker/daemon.json
{
"data-root": "/var/lib/docker",
"exec-opts": ["native.cgroupdriver=cgroupfs"],
"insecure-registries": ["127.0.0.1/8","172.18.8.214"],
"max-concurrent-downloads": 10,
"live-restore": true,
"log-driver": "json-file",
"log-level": "warn",
"log-opts": {
"max-size": "50m",
"max-file": "1"
},
"storage-driver": "overlay2"
}
2 错误2
Sep 02 16:31:17 k8s-master2.com kubelet[87623]: W0902 16:31:17.259389 87623 cni.go:204] Error validating CNI config list {"cniVersion":"0.3.1","name":"mynet","plugins":[{"bridge":"mynet0","cniVersion":"0.3.1","hairpinMode":true,"ipMasq":true,"ipam":{"subnet":"10.200.0.0/16","type":"host-local"},"isDefaultGateway":true,"name":"mynet","type":"bridge"}]}: [failed to find plugin "bridge" in path [/opt/cni/bin]]
Sep 02 16:31:17 k8s-master2.com kubelet[87623]: W0902 16:31:17.260166 87623 cni.go:204] Error validating CNI config list {
Sep 02 16:31:17 k8s-master2.com kubelet[87623]: "name": "cbr0",
Sep 02 16:31:17 k8s-master2.com kubelet[87623]: "cniVersion": "0.3.1",
Sep 02 16:31:17 k8s-master2.com kubelet[87623]: "plugins": [
Sep 02 16:31:17 k8s-master2.com kubelet[87623]: {
Sep 02 16:31:17 k8s-master2.com kubelet[87623]: "type": "flannel",
Sep 02 16:31:17 k8s-master2.com kubelet[87623]: "delegate": {
Sep 02 16:31:17 k8s-master2.com kubelet[87623]: "hairpinMode": true,
Sep 02 16:31:17 k8s-master2.com kubelet[87623]: "isDefaultGateway": true
Sep 02 16:31:17 k8s-master2.com kubelet[87623]: }
Sep 02 16:31:17 k8s-master2.com kubelet[87623]: },
Sep 02 16:31:17 k8s-master2.com kubelet[87623]: {
Sep 02 16:31:17 k8s-master2.com kubelet[87623]: "type": "portmap",
Sep 02 16:31:17 k8s-master2.com kubelet[87623]: "capabilities": {
Sep 02 16:31:17 k8s-master2.com kubelet[87623]: "portMappings": true
Sep 02 16:31:17 k8s-master2.com kubelet[87623]: }
Sep 02 16:31:17 k8s-master2.com kubelet[87623]: }
Sep 02 16:31:17 k8s-master2.com kubelet[87623]: ]
Sep 02 16:31:17 k8s-master2.com kubelet[87623]: }
Sep 02 16:31:17 k8s-master2.com kubelet[87623]: : [failed to find plugin "flannel" in path [/opt/cni/bin] failed to find plugin "portmap" in path [/opt/cni/bin]]
Sep 02 16:31:17 k8s-master2.com kubelet[87623]: W0902 16:31:17.260197 87623 cni.go:239] Unable to update cni config: no valid networks found in /etc/cni/net.d
Sep 02 16:31:17 k8s-master2.com kubelet[87623]: E0902 16:31:17.328048 87623 kubelet.go:2243] node "k8s-master2.com" not found
2.1 原因及解决办法
此台机器使用过 kubeadm 安装过 k8s，kubeadm 默认采用 flannel 的网络模式，再次使用二进制安装 k8s 需要重新安装操作系统
|