> 技术文档 > K8s报错:The connection to the server 192.168.109.100:6443 was refused - did you specify the right host_jul 24 16:49:26 master kubelet[103757]: i0724 16:4

K8s报错:The connection to the server 192.168.109.100:6443 was refused - did you specify the right host_jul 24 16:49:26 master kubelet[103757]: i0724 16:4


背景:

之前已经安装好的cluster重启之后忽然不能执行和查看,提示The connection to the server 192.168.109.100:6443 was refused - did you specify the right host or port?

初步推测:kubectl无法连接到API server,并且问题是master节点问题

分析:

查看日志:journalctl -xeu kubelet
日志中提到kubelet cgroup driver是 systemd不同于docker的cgroupfs

kubelet​​ 默认使用 systemd 管理容器资源限制(通过 systemd 的 cgroup 层级)。
​​Docker​​ 当前配置为 cgroupfs 模式(直接操作 cgroup 文件系统)。
​​冲突导致​​:kubelet 无法与 Docker 的容器运行时通信

解决

统一使用systemd驱动
或者统一选择cgroupfs驱动

如果要修改docker配置

vim /etc/docker/daemon.json
{
“exec-opts”: [“native.cgroupdriver=systemd”],
“log-driver”: “json-file”,
“log-opts”: {
“max-size”: “100m”
},
“storage-driver”: “overlay2”
}

然后重启服务
systemctl daemon-reload
systemctl restart docker
systemctl restart kubelet

如果要修改kubelet

vim /var/lib/kubelet/config.yaml
cgroupDriver: cgroupfs
然后重启systemctl restart kubelet

最终验证集群状态kubectl get nodes

附上当时日志:

7月 02 21:27:06 master systemd[1]: kubelet.service holdoff time over,
7月 02 21:27:06 master systemd[1]: Stopped kubelet: The Kubernetes Nod
– Subject: Unit kubelet.service has finished shutting down
– Defined-By: systemd
– Support: http://lists.freedesktop.org/mailman/listinfo/systemd-deve

– Unit kubelet.service has finished shutting down.
7月 02 21:27:06 master systemd[1]: Started kubelet: The Kubernetes Nod
– Subject: Unit kubelet.service has finished start-up
– Defined-By: systemd
– Support: http://lists.freedesktop.org/mailman/listinfo/systemd-deve

– Unit kubelet.service has finished starting up.

– The start-up result is done.
7月 02 21:27:06 master kubelet[13261]: Flag --cgroup-driver has been d
7月 02 21:27:06 master kubelet[13261]: Flag --cgroup-driver has been d
7月 02 21:27:06 master kubelet[13261]: I0702 21:27:06.257885 13261 s
7月 02 21:27:06 master kubelet[13261]: I0702 21:27:06.258207 13261 p
7月 02 21:27:06 master kubelet[13261]: I0702 21:27:06.258232 13261 s
7月 02 21:27:06 master kubelet[13261]: I0702 21:27:06.260428 13261 c
7月 02 21:27:06 master kubelet[13261]: I0702 21:27:06.325773 13261 s
7月 02 21:27:06 master kubelet[13261]: I0702 21:27:06.326322 13261 c
7月 02 21:27:06 master kubelet[13261]: I0702 21:27:06.326347 13261 c
7月 02 21:27:06 master kubelet[13261]: I0702 21:27:06.326429 13261 f
7月 02 21:27:06 master kubelet[13261]: I0702 21:27:06.326433 13261 c
7月 02 21:27:06 master kubelet[13261]: I0702 21:27:06.326451 13261 f
7月 02 21:27:06 master kubelet[13261]: I0702 21:27:06.326487 13261 s
7月 02 21:27:06 master kubelet[13261]: I0702 21:27:06.326551 13261 s
7月 02 21:27:06 master kubelet[13261]: I0702 21:27:06.326554 13261 s
7月 02 21:27:06 master kubelet[13261]: I0702 21:27:06.326561 13261 f
7月 02 21:27:06 master kubelet[13261]: I0702 21:27:06.326630 13261 k
7月 02 21:27:06 master kubelet[13261]: I0702 21:27:06.326681 13261 k
7月 02 21:27:06 master kubelet[13261]: E0702 21:27:06.333360 13261 r
7月 02 21:27:06 master kubelet[13261]: E0702 21:27:06.333562 13261 r
7月 02 21:27:06 master kubelet[13261]: E0702 21:27:06.333770 13261 r
7月 02 21:27:06 master kubelet[13261]: I0702 21:27:06.334412 13261 c
7月 02 21:27:06 master kubelet[13261]: I0702 21:27:06.334453 13261 c
7月 02 21:27:06 master kubelet[13261]: W0702 21:27:06.338708 13261 d
7月 02 21:27:06 master kubelet[13261]: I0702 21:27:06.338810 13261 d
7月 02 21:27:06 master kubelet[13261]: I0702 21:27:06.362705 13261 d
7月 02 21:27:06 master kubelet[13261]: I0702 21:27:06.383322 13261 d
7月 02 21:27:06 master kubelet[13261]: F0702 21:27:06.383470 13261 s
7月 02 21:27:06 master systemd[1]: kubelet.service: main process exite
7月 02 21:27:06 master systemd[1]: Unit kubelet.service entered failed
7月 02 21:27:06 master systemd[1]: kubelet.service failed.
lines 1304-1349/1349 (END)
7月 02 21:27:06 master systemd[1]: kubelet.service holdoff time over, scheduling restart.
7月 02 21:27:06 master systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
– Subject: Unit kubelet.service has finished shutting down
– Defined-By: systemd
– Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

– Unit kubelet.service has finished shutting down.
7月 02 21:27:06 master systemd[1]: Started kubelet: The Kubernetes Node Agent.
– Subject: Unit kubelet.service has finished start-up
– Defined-By: systemd
– Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

– Unit kubelet.service has finished starting up.

– The start-up result is done.
7月 02 21:27:06 master kubelet[13261]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kube
7月 02 21:27:06 master kubelet[13261]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kube
7月 02 21:27:06 master kubelet[13261]: I0702 21:27:06.257885 13261 server.go:416] Version: v1.17.4
7月 02 21:27:06 master kubelet[13261]: I0702 21:27:06.258207 13261 plugins.go:100] No cloud provider specified.
7月 02 21:27:06 master kubelet[13261]: I0702 21:27:06.258232 13261 server.go:821] Client rotation is on, will bootstrap in background
7月 02 21:27:06 master kubelet[13261]: I0702 21:27:06.260428 13261 certificate_store.go:129] Loading cert/key pair from \"/var/lib/kubelet/pki/kubelet
7月 02 21:27:06 master kubelet[13261]: I0702 21:27:06.325773 13261 server.go:641] --cgroups-per-qos enabled, but --cgroup-root was not specified. de
7月 02 21:27:06 master kubelet[13261]: I0702 21:27:06.326322 13261 container_manager_linux.go:265] container manager verified user specified cgroup-r
7月 02 21:27:06 master kubelet[13261]: I0702 21:27:06.326347 13261 container_manager_linux.go:270] Creating Container Manager object based on Node Co
7月 02 21:27:06 master kubelet[13261]: I0702 21:27:06.326429 13261 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
7月 02 21:27:06 master kubelet[13261]: I0702 21:27:06.326433 13261 container_manager_linux.go:305] Creating device plugin manager: true
7月 02 21:27:06 master kubelet[13261]: I0702 21:27:06.326451 13261 fake_topology_manager.go:39] [fake topologymanager] AddHintProvider HintProvider:
7月 02 21:27:06 master kubelet[13261]: I0702 21:27:06.326487 13261 state_mem.go:36] [cpumanager] initializing new in-memory state store
7月 02 21:27:06 master kubelet[13261]: I0702 21:27:06.326551 13261 state_mem.go:84] [cpumanager] updated default cpuset: “”
7月 02 21:27:06 master kubelet[13261]: I0702 21:27:06.326554 13261 state_mem.go:92] [cpumanager] updated cpuset assignments: “map[]”
7月 02 21:27:06 master kubelet[13261]: I0702 21:27:06.326561 13261 fake_topology_manager.go:39] [fake topologymanager] AddHintProvider HintProvider:
7月 02 21:27:06 master kubelet[13261]: I0702 21:27:06.326630 13261 kubelet.go:286] Adding pod path: /etc/kubernetes/manifests
7月 02 21:27:06 master kubelet[13261]: I0702 21:27:06.326681 13261 kubelet.go:311] Watching apiserver
7月 02 21:27:06 master kubelet[13261]: E0702 21:27:06.333360 13261 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/kubelet.go:458: Failed to list *v1
7月 02 21:27:06 master kubelet[13261]: E0702 21:27:06.333562 13261 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/kubelet.go:449: Failed to list *v1
7月 02 21:27:06 master kubelet[13261]: E0702 21:27:06.333770 13261 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to
7月 02 21:27:06 master kubelet[13261]: I0702 21:27:06.334412 13261 client.go:75] Connecting to docker on unix:///var/run/docker.sock
7月 02 21:27:06 master kubelet[13261]: I0702 21:27:06.334453 13261 client.go:104] Start docker client with request timeout=2m0s
7月 02 21:27:06 master kubelet[13261]: W0702 21:27:06.338708 13261 docker_service.go:563] Hairpin mode set to “promiscuous-bridge” but kubenet is not
7月 02 21:27:06 master kubelet[13261]: I0702 21:27:06.338810 13261 docker_service.go:240] Hairpin mode set to “hairpin-veth”
7月 02 21:27:06 master kubelet[13261]: I0702 21:27:06.362705 13261 docker_service.go:255] Docker cri networking managed by cni
7月 02 21:27:06 master kubelet[13261]: I0702 21:27:06.383322 13261 docker_service.go:260] Docker Info: &{ID:MYMU:BHBA:M3BR:M2BQ:JQJT:M7SE:UMOL:AAWZ:S
7月 02 21:27:06 master kubelet[13261]: F0702 21:27:06.383470 13261 server.go:273] failed to run Kubelet: failed to create kubelet: misconfiguration:
7月 02 21:27:06 master systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a
7月 02 21:27:06 master systemd[1]: Unit kubelet.service entered failed state.
7月 02 21:27:06 master systemd[1]: kubelet.service failed.