[系统运维] k8s解决 kubectl get pod 查看后No resources found问题

开发: C++知识库 Java知识库 JavaScript Python PHP知识库人工智能区块链大数据移动开发嵌入式开发工具数据结构与算法开发测试游戏开发网络协议系统运维
教程: HTML教程 CSS教程 JavaScript教程 Go语言教程 JQuery教程 VUE教程 VUE3教程 Bootstrap教程 SQL数据库教程 C语言教程 C++教程 Java教程 Python教程 Python3教程 C#教程
数码: 电脑笔记本显卡显示器固态硬盘硬盘耳机手机 iphone vivo oppo 小米华为单反装机图拉丁

-> 系统运维 -> k8s解决 kubectl get pod 查看后No resources found问题 -> 正文阅读

[系统运维]k8s解决 kubectl get pod 查看后No resources found问题

1. 创建一个资源

# kubectl create -f nginx-ds.yaml

2. 查看pod显示no resources found

# kubectl get pod
No resources found

3. 查看日志

3.1 查看kube-apiserver日志，无异常

3.2 查看kube-kubelet日志，无异常

3.3 查看kube-scheduler日志如下

查看链接127.0.0.1:8000被拒绝，后来在本地查看并没有8000端口，我的apiserver的端口设置的是8080，可能是kube-scheduler启动脚本端口写错

# tail -f /data/logs/kubernetes/kube-scheduler/scheduler.stdout.log
E0429 09:44:11.299768   48887 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1beta1.PodDisruptionBudget: Get http://127.0.0.1:8000/apis/policy/v1beta1/poddisruptionbudgets?limit=500&resourceVersion=0: dial tcp 127.0.0.1:8000: connect: connection refused
E0429 09:44:11.304570   48887 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.PersistentVolume: Get http://127.0.0.1:8000/api/v1/persistentvolumes?limit=500&resourceVersion=0: dial tcp 127.0.0.1:8000: connect: connection refused
E0429 09:44:11.308370   48887 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.Node: Get http://127.0.0.1:8000/api/v1/nodes?limit=500&resourceVersion=0: dial tcp 127.0.0.1:8000: connect: connection refused
E0429 09:44:11.308745   48887 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.StorageClass: Get http://127.0.0.1:8000/apis/storage.k8s.io/v1/storageclasses?limit=500&resourceVersion=0: dial tcp 127.0.0.1:8000: connect: connection refused
E0429 09:44:11.310308   48887 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.ReplicationController: Get http://127.0.0.1:8000/api/v1/replicationcontrollers?limit=500&resourceVersion=0: dial tcp 127.0.0.1:8000: connect: connection refused
E0429 09:44:11.311481   48887 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.StatefulSet: Get http://127.0.0.1:8000/apis/apps/v1/statefulsets?limit=500&resourceVersion=0: dial tcp 127.0.0.1:8000: connect: connection refused
E0429 09:44:12.293511   48887 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.Service: Get http://127.0.0.1:8000/api/v1/services?limit=500&resourceVersion=0: dial tcp 127.0.0.1:8000: connect: connection refused
E0429 09:44:12.297952   48887 reflector.go:125] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:226: Failed to list *v1.Pod: Get http://127.0.0.1:8000/api/v1/pods?fieldSelector=status.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&limit=500&resourceVersion=0: dial tcp 127.0.0.1:8000: connect: connection refused
E0429 09:44:12.299568   48887 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.PersistentVolumeClaim: Get http://127.0.0.1:8000/api/v1/persistentvolumeclaims?limit=500&resourceVersion=0: dial tcp 127.0.0.1:8000: connect: connection refused
E0429 09:44:12.300620   48887 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.ReplicaSet: Get http://127.0.0.1:8000/apis/apps/v1/replicasets?limit=500&resourceVersion=0: dial tcp 127.0.0.1:8000: connect: connection refused
E0429 09:44:12.301799   48887 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1beta1.PodDisruptionBudget: Get http://127.0.0.1:8000/apis/policy/v1beta1/poddisruptionbudgets?limit=500&resourceVersion=0: dial tcp 127.0.0.1:8000: connect: connection refused
E0429 09:44:12.305372   48887 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.PersistentVolume: Get http://127.0.0.1:8000/api/v1/persistentvolumes?limit=500&resourceVersion=0: dial tcp 127.0.0.1:8000: connect: connection refused
E0429 09:44:12.308955   48887 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.Node: Get http://127.0.0.1:8000/api/v1/nodes?limit=500&resourceVersion=0: dial tcp 127.0.0.1:8000: connect: connection refused
E0429 09:44:12.310113   48887 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.StorageClass: Get http://127.0.0.1:8000/apis/storage.k8s.io/v1/storageclasses?limit=500&resourceVersion=0: dial tcp 127.0.0.1:8000: connect: connection refused
E0429 09:44:12.311407   48887 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.ReplicationController: Get http://127.0.0.1:8000/api/v1/replicationcontrollers?limit=500&resourceVersion=0: dial tcp 127.0.0.1:8000: connect: connection refused
E0429 09:44:12.312201   48887 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.StatefulSet: Get http://127.0.0.1:8000/apis/apps/v1/statefulsets?limit=500&resourceVersion=0: dial tcp 127.0.0.1:8000: connect: connection refused

3.4 查看controller-manager日志如下

链接8000端口也是被拒绝了，这里我的apiserver的端口是8080，可能是controller-manage端口写错

# tail -f /data/logs/kubernetes/kube-controller-manager/controller.stdout.log 
E0429 09:45:25.841916   48885 leaderelection.go:324] error retrieving resource lock kube-system/kube-controller-manager: Get http://127.0.0.1:8000/api/v1/namespaces/kube-system/endpoints/kube-controller-manager?timeout=10s: dial tcp 127.0.0.1:8000: connect: connection refused
E0429 09:45:28.981869   48885 leaderelection.go:324] error retrieving resource lock kube-system/kube-controller-manager: Get http://127.0.0.1:8000/api/v1/namespaces/kube-system/endpoints/kube-controller-manager?timeout=10s: dial tcp 127.0.0.1:8000: connect: connection refused
E0429 09:45:32.367088   48885 leaderelection.go:324] error retrieving resource lock kube-system/kube-controller-manager: Get http://127.0.0.1:8000/api/v1/namespaces/kube-system/endpoints/kube-controller-manager?timeout=10s: dial tcp 127.0.0.1:8000: connect: connection refused
E0429 09:45:35.969067   48885 leaderelection.go:324] error retrieving resource lock kube-system/kube-controller-manager: Get http://127.0.0.1:8000/api/v1/namespaces/kube-system/endpoints/kube-controller-manager?timeout=10s: dial tcp 127.0.0.1:8000: connect: connection refused
E0429 09:45:38.060622   48885 leaderelection.go:324] error retrieving resource lock kube-system/kube-controller-manager: Get http://127.0.0.1:8000/api/v1/namespaces/kube-system/endpoints/kube-controller-manager?timeout=10s: dial tcp 127.0.0.1:8000: connect: connection refused
E0429 09:45:40.605084   48885 leaderelection.go:324] error retrieving resource lock kube-system/kube-controller-manager: Get http://127.0.0.1:8000/api/v1/namespaces/kube-system/endpoints/kube-controller-manager?timeout=10s: dial tcp 127.0.0.1:8000: connect: connection refused
E0429 09:45:42.638763   48885 leaderelection.go:324] error retrieving resource lock kube-system/kube-controller-manager: Get http://127.0.0.1:8000/api/v1/namespaces/kube-system/endpoints/kube-controller-manager?timeout=10s: dial tcp 127.0.0.1:8000: connect: connection refused
E0429 09:45:45.856238   48885 leaderelection.go:324] error retrieving resource lock kube-system/kube-controller-manager: Get http://127.0.0.1:8000/api/v1/namespaces/kube-system/endpoints/kube-controller-manager?timeout=10s: dial tcp 127.0.0.1:8000: connect: connection refused
E0429 09:45:48.169061   48885 leaderelection.go:324] error retrieving resource lock kube-system/kube-controller-manager: Get http://127.0.0.1:8000/api/v1/namespaces/kube-system/endpoints/kube-controller-manager?timeout=10s: dial tcp 127.0.0.1:8000: connect: connection refused
E0429 09:45:50.974880   48885 leaderelection.go:324] error retrieving resource lock kube-system/kube-controller-manager: Get http://127.0.0.1:8000/api/v1/namespaces/kube-system/endpoints/kube-controller-manager?timeout=10s: dial tcp 127.0.0.1:8000: connect: connection refused
E0429 09:45:54.336228   48885 leaderelection.go:324] error retrieving resource lock kube-system/kube-controller-manager: Get http://127.0.0.1:8000/api/v1/namespaces/kube-system/endpoints/kube-controller-manager?timeout=10s: dial tcp 127.0.0.1:8000: connect: connection refused
E0429 09:45:57.195092   48885 leaderelection.go:324] error retrieving resource lock kube-system/kube-controller-manager: Get http://127.0.0.1:8000/api/v1/namespaces/kube-system/endpoints/kube-controller-manager?timeout=10s: dial tcp 127.0.0.1:8000: connect: connection refused

3.5 查看kube-proxy日志

此处判断可能是没有链接到apiserver导致

# tail -f /data/logs/kubernetes/kube-proxy/proxy.stdout.log 
E0429 09:51:56.124383   48886 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.Service: the server rejected our request for an unknown reason (get services)
E0429 09:51:56.125239   48886 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.Endpoints: the server rejected our request for an unknown reason (get endpoints)
E0429 09:51:57.126052   48886 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.Service: the server rejected our request for an unknown reason (get services)
E0429 09:51:57.126586   48886 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.Endpoints: the server rejected our request for an unknown reason (get endpoints)
E0429 09:51:58.128520   48886 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.Service: the server rejected our request for an unknown reason (get services)
E0429 09:51:58.129030   48886 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.Endpoints: the server rejected our request for an unknown reason (get endpoints)
E0429 09:51:59.130244   48886 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.Service: the server rejected our request for an unknown reason (get services)
E0429 09:51:59.130979   48886 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.Endpoints: the server rejected our request for an unknown reason (get endpoints)
E0429 09:52:00.132002   48886 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.Service: the server rejected our request for an unknown reason (get services)
E0429 09:52:00.134413   48886 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.Endpoints: the server rejected our request for an unknown reason (get endpoints)
E0429 09:52:01.134364   48886 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.Service: the server rejected our request for an unknown reason (get services)
E0429 09:52:01.135871   48886 reflector.go:125] k8s.io/client-go/informers/factory.go:133: Failed to list *v1.Endpoints: the server rejected our request for an unknown reason (get endpoints)

4. 更改

4.1 更改kube-scheduler启动脚本

# 将8000端口改成8080，我的apiserver端口设置的是8080
# cat kube-scheduler.sh
#!/bin/bash
./kube-scheduler \
  --leader-elect \
  --log-dir /data/logs/kubernetes/kube-scheduler \
  --master http://127.0.0.1:8080 \
  --v 2

4.2 更改controller-manager启动脚本

# 此处将8000端口改成8080
# cat kube-controller-manager.sh
#!/bin/bash
./kube-controller-manager \
  --cluster-cidr 172.7.0.0/16 \
  --leader-elect true \
  --log-dir /data/logs/kubernetes/kube-controller-manager \
  --master http://127.0.0.1:8080 \
  --service-account-private-key-file ./cert/ca-key.pem \
  --service-cluster-ip-range 192.168.0.0/16 \
  --root-ca-file ./cert/ca.pem \
  --v 2

4.3 停止supervisord后在start

 # systemctl stop supervisord.service
 # systemctl start supervisord.service
 # supervisorctl status
etcd-server-7-21                 RUNNING   pid 66496, uptime 0:03:22
kube-apiserver-7-21              RUNNING   pid 66497, uptime 0:03:22
kube-controller-manager-7-21     RUNNING   pid 66499, uptime 0:03:22
kube-kubelet-7-21                RUNNING   pid 66706, uptime 0:03:17
kube-proxy-7-21                  RUNNING   pid 66500, uptime 0:03:22
kube-scheduler-7-21              RUNNING   pid 66501, uptime 0:03:22

5. 查看pod

# kubectl get pod
NAME             READY   STATUS              RESTARTS   AGE
nginx-ds-5q6mh   0/1     ContainerCreating   0          4m13s
nginx-ds-8bpc6   0/1     ContainerCreating   0          4m13s

可以能看到pod资源了，但是查看kub-proxy日志还是3.5描述的错误，继续排查。。。。。。