k8s-通过kubeadm安装最新版

前言

本次基于Kubeadm和Docker部署kubernetes高可用集群

架构图

基于Haproxy+keepalived非Etcd集群架构

基于Haproxy+keepalived Etcd独立集群架构

kubeadm 介绍和环境说明

kubeadm 介绍

官方相关文档地址

1
2
3
4
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/installkubeadm/
https://kubernetes.io/zh-cn/docs/setup/productionenvironment/tools/kubeadm/create-cluster-kubeadm/
https://github.com/kubernetes/kubeadm/blob/master/docs/design/design_v1.10.md

kubeadm是Kubernetes社区提供的集群构建工具

  • 负责执行构建一个最小化可用集群并将其启动等必要的基本步骤
  • Kubernetes集群全生命周期管理工具，可用于实现集群的部署、升级/降级及卸载等
  • kubeadm仅关心如何初始化并拉起一个集群，其职责仅限于下图中背景蓝色的部分
  • 蓝色的部分以外的其它组件还需要自行部署

    注意: 在kubeadm方式安装时,Kubernetes的所有组件中除kubelet是以传统服务进程的方式运行,其它都以容器运行

部署环镜说明

当前部署版本为当前次新版本1.36.1
部署环境:

IP 主机名称 角色
10.200.17.61 master1.luo.org K8s 集群主节点 1，Master和etcd
10.200.17.62 master2.luo.org K8s 集群主节点 2，Master和etcd
10.200.17.63 master3.luo.org K8s 集群主节点 3，Master和etcd
10.200.17.64 node1.luo.org K8s 集群工作节点 1
10.200.17.65 node2.luo.org K8s 集群工作节点 2
10.200.17.66 node3.luo.org K8s 集群工作节点 3
10.200.17.67 ha1.luo.org K8s 主节点访问入口 1,提供高可用及负载均衡
10.200.17.68 ha2.luo.org K8s 主节点访问入口 2,提供高可用及负载均衡
10.200.17.100 kubeapi.luo.org VIP，在ha1和ha2主机实现
10.200.17.42 harbor.luo.org 容器镜像仓库（可选）

注意： Master节点内存至少2G以上，否则在初始化时会出错
[ERROR Mem]: the system RAM (952 MB) is less than the minimum 1700 MB
网络地址规划：

1
2
3
物理主机网络 10.200.17/24
集群pod网络 --pod-network-cidr=10.244.0.0/16
应用service网络 --service-cidr=10.96.0.0/12

基于kubeadm和Docker实现Kubernetes集群流程说明:

  • 每个节点主机的初始环境准备
  • 准备代理服务,以便访问k8s.gcr.io，或根据部署过程提示的方法获取相应的I国内镜像的image（可选）
  • Kubernetes集群API访问入口的高可用和harbor（可选）
  • 在所有Master和Node节点都安装容器运行时 Docker
  • 在所有节点安装和配置cri-dockerd(kubernetes-v1.24版本以后需要)
  • 在所有Master和Node节点都安装kubeadm、kubelet、kubectl(集群管理工具,在node节点可不安装)
  • 在第一个master节点运行kubeadm init初始化命令,并验证master节点状态
  • 在第一个master节点安装配置CNI规范的网络插件flannel,calico等
  • 在其它master节点运行kubeadm join命令加入到控制平面集群中实现高可用(测试环境可选)
  • 在所有node节点使用 kubeadm join命令加入集群, 并验证node节点状态
  • 创建pod并启动容器测试访问,并测试网络通信

初始环境准备

  • 硬件准备环境: 每个主机至少2G以上内存,CPU2核以上
  • 操作系统: 最小化安装支持Kubernetes的Linux系统
  • 唯一的主机名，MAC地址以及product_uuid和主机名解析
  • 保证各个节点网络配置正确,并且保证通信正常
  • 禁用 swap
  • 禁用 SELinux
  • 放行Kubernetes使用到的相关端口或禁用firewalld/iptables
  • 配置正确的时区和时间同步
  • 内核参数优化
  • 所有节点实现基于 ssh key 验证(可选)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# 查看product_uuid
root@master1:~# cat /sys/class/dmi/id/product_uuid
fcd84d56-5e88-cb0a-120f-f0fc0a218549

# 按照前面规划配置唯一的主机名
root@master1:~# hostnamectl set-hostname master1.luo.org

# 查看本地解析
root@master1:~# cat /etc/hosts
10.200.17.61  master1.luo.org master1
10.200.17.62  master2.luo.org master2
10.200.17.63  master3.luo.org master3
10.200.17.64  node1.luo.org node1
10.200.17.65  node2.luo.org node2
10.200.17.66  node3.luo.org node3
10.200.17.67  ha1.luo.org   ha1
10.200.17.68  ha2.luo.org   ha2
10.200.17.100 kubeapi.luo.org kubeapi
10.200.17.42 harbor.luo.org harbor

操作系统及Kubernetes组件版本

以下案例中部署集群使用的操作系统、容器引擎、Kubernetes 版本信息:

  • OS: Ubuntu 26.04 LTS
  • CRI: Docker 28.4.0 Cgroup Driver: systemd
  • Kubernetes: 1.36.1

主机时间同步

在集群的Master和各node同步时间

1
2
3
root@master1:~# timedatectl set-timezone Asia/Shanghai
root@master1:~# apt -y install ntpsec-ntpdate
root@master1:~# ntpdate ntp7.aliyun.com

关闭防火墙

Ubuntu系统：

1
2
3
4
5
root@master1:~# ufw disable
root@master1:~# ufw status 
Status: inactive
# 如果安装firewalld
root@master1:~# systemctl disable --now firewalld

禁用Swap设备

在集群的Master和各node执行

1
2
3
4
5
6
7
8
9
10
11
12
# 第一种方法
root@master1:~# swapoff -a
root@master1:~# sed -i '/swap/s/^/#/' /etc/fstab
root@master1:~# for i in {62..66};do ssh 10.200.17.$i "sed -i '/swap/s/^/#/' /etc/fstab";ssh 10.200.17.$i swapoff -a ;done

# 第二种方法
root@master1:~# systemctl stop swap.img.swap
root@master1:~# systemctl mask swap.img.swap 或者 systemctl mask swap.target

# 确认是否禁用swap
root@master1:~# systemctl -t swap
root@master1:~# swapon -s

内核优化

  • 根据硬件和业务需求,对内核参数做相应的优化
  • 注意:安装docker时会自动修改内核参数
  • 如果是安装Docker没有配置，就需要手动配置以下的内核参数
  • 允许iptables检查桥接流量,若要显式加载此模块，需运行modprobe br_netfilter
  • 为了让Linux节点的iptables能够正确查看桥接流量，还需要确认net.bridge.bridge-nf-call-iptables设置为1。
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    root@master1:~# modprobe overlay    
    root@master1:~# modprobe br_netfilter
    root@master1:~# lsmod |grep -E 'overlay|br_netfilter'
    br_netfilter           32768  0
    bridge                425984  1 br_netfilter
    overlay               233472  16

    # 开机自动加载配置
    root@master1:~# vim /etc/modules-load.d/k8s.conf
    overlay
    br_netfilter

    # 设置所需的 sysctl 参数，参数在重新启动后保持不变
    root@master1:~# cat /etc/sysctl.d/network.conf 
    net.ipv4.ip_forward = 1
    net.bridge.bridge-nf-call-iptables = 1
    net.bridge.bridge-nf-call-ip6tables = 1
    net.bridge.bridge-nf-call-arptables = 1
    net.ipv4.tcp_tw_reuse = 0
    net.core.somaxconn = 32768
    net.netfilter.nf_conntrack_max = 1000000
    vm.swappiness = 0
    vm.max_map_count = 655360
    fs.file-max = 6553600
    # 有haproxy时加意思就是在启动haproxy时，允许忽视VIP的存在
    # net.ipv4.ip_nonlocal_bind = 1
    #应用sysctl参数生效而不重新启动
    root@master1:~# sysctl --system

实现高可用的反向代理

可以利用Haproxy（或nginx）和keepalived实现Kube-API server的高可用和负载均衡

实现keepalived

在两台主机ha1和ha2按下面步骤部署和配置keepalived

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
root@ha1:~# apt update && apt -y install keepalived
root@ha1:~# cat /etc/keepalived/keepalived.conf
global_defs {
	notification_email {
		acassen
}
	notification_email_from Alexandre.Cassen@firewall.loc
	smtp_server 192.168.200.1
	smtp_connect_timeout 30
	router_id ha1.luo.org #指定router_id,#在ha2上为ha2.luo.org
}
vrrp_script check_haproxy {
	#script "/etc/keepalived/check_haproxy.sh"
	script "killall -0 haproxy"
	interval 1
	weight -30
	fall 3
	rise 2
	timeout 2
}
vrrp_instance VI_1 {
	state MASTER #在ha2上为BACKUP
	interface eth0
	garp_master_delay 10
	smtp_alert
	virtual_router_id 66 #指定虚拟路由器ID,ha1和ha2此值必须相同
	priority 100 #在ha2上为80
	advert_int 1
	authentication {
	auth_type PASS
	auth_pass 123456 #指定验证密码,ha1和ha2此值必须相同
}
virtual_ipaddress {
	10.200.17.100/24 dev eth0 label eth0:1 #指定VIP,ha1和ha2此值必须相同
}
track_script {
	check_haproxy
}
}

实现Haproxy

通过Harproxy实现kubernetes Api-server的四层反向代理和负载均衡功能

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
root@ha1:~# apt -y install haproxy
root@ha1:~# cat /etc/haproxy/haproxy.cfg 
global
	log /dev/log	local0
	log /dev/log	local1 notice
	chroot /var/lib/haproxy
	stats socket /run/haproxy/admin.sock mode 660 level admin
	stats timeout 30s
	user haproxy
	group haproxy
	daemon

	# Default SSL material locations
	ca-base /etc/ssl/certs
	crt-base /etc/ssl/private

	# See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
        ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
        ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
        ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
	log	global
	mode	http
	option	httplog
	option	dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
	errorfile 400 /etc/haproxy/errors/400.http
	errorfile 403 /etc/haproxy/errors/403.http
	errorfile 408 /etc/haproxy/errors/408.http
	errorfile 500 /etc/haproxy/errors/500.http
	errorfile 502 /etc/haproxy/errors/502.http
	errorfile 503 /etc/haproxy/errors/503.http
	errorfile 504 /etc/haproxy/errors/504.http


listen stats
    mode http
    bind 0.0.0.0:8888
    stats enable
    log global
    stats uri /status
    stats auth admin:123456

listen kubernetes-api-6443
    bind 10.200.17.100:6443
    mode tcp
    server master1 10.200.17.61:6443 check inter 3s fall 3 rise 3
    #先暂时禁用master2和master3，等kubernetes安装完成后，再启用
    server master2 10.200.17.62:6443 check inter 3s fall 3 rise 3
    server master3 10.200.17.63:6443 check inter 3s fall 3 rise 3

所有节点安装Docker

这里使用脚本方式安装二进制Docker

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# 这里下载所需Docker的安装包和脚本都在这个压缩包中
root@master1:~# cd /usr/local/src &&  wget http://imxx.oaxxx.com/download/deploy_auto_0920.tar.gz
root@master1:/usr/local/src# tar xf deploy_auto_0920.tar.gz
root@master1:/usr/local/src# cd docker/
root@master1:/usr/local/src/docker# bash deploy_docker.sh
root@master1:/usr/local/src/docker# cat daemon.json 
{
  "registry-mirrors": [
    "https://docker.1ms.run",
    "https://docker.xuanyuan.me",
    "https://docker.1panel.live",
    "https://docker.m.daocloud.io"
   ],
  "insecure-registries":["10.200.17.42"],
  "exec-opts": ["native.cgroupdriver=systemd"],
  "data-root": "/app/dk-work",
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  }
}

所有主机安装cri-dockerd(v1.24以后版本)

Kubernetes自v1.24移除了对docker-shim的支持，而Docker Engine默认又不支持CRI规范，因而二者将无法直接完成整合。为此，Mirantis和Docker联合创建了cri-dockerd项目，用于为Docker Engine提供一个能够支持到CRI规范的垫片，从而能够让Kubernetes基于CRI控制Docker。
项目地址：https://github.com/Mirantis/cri-dockerd
cri-dockerd项目提供了预制的二制格式的程序包，用户按需下载相应的系统和对应平台的版本即可完成安装，这里以Ubuntu 26.04 64bits系统环境，以及cri-dockerd目前最新的程序版本v0.4.2为例。
由于Ubuntu系统是最新的cri-dockerd还没有制作好对应的deb包，本次用到二进制文件以及手动创建service和socket文件

1
2
3
4
5
6
7
8
root@master1:/usr/local/src# wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.4.2/cri-dockerd-0.4.2.amd64.tgz
root@master1:/usr/local/src# tar xf cri-dockerd-0.4.2.amd64.tgz
root@master1:/usr/local/src# mv cri-dockerd/cri-dockerd /usr/bin/
# 准备service和socket文件
# https://github.com/Mirantis/cri-dockerd/tree/master/packaging/systemd
root@master1:/usr/local/src# wget -O /lib/systemd/system/cri-docker.service https://raw.githubusercontent.com/Mirantis/cridockerd/refs/heads/master/packaging/systemd/cri-docker.service
root@master1:/usr/local/src# wget -O /lib/systemd/system/cri-docker.socket https://raw.githubusercontent.com/Mirantis/cridockerd/refs/heads/master/packaging/systemd/cri-docker.socket
root@master1:/usr/local/src# systemctl daemon-reload && systemctl enable --now cri-docker

Cri-dockerd配置

众所周知的原因,从国内 cri-dockerd 服务无法下载 k8s.gcr.io上面相关镜像,导致无法启动,所以需要修改cri-dockerd使用国内镜像源

1
2
3
4
root@master1:/usr/local/src# vim /lib/systemd/system/cri-docker.service
# 最初是没有--pod-infra-container-image
ExecStart=/usr/bin/cri-dockerd --container-runtime-endpoint fd://
ExecStart=/usr/bin/cri-dockerd --container-runtime-endpoint fd:// --pod-infra-container-image registry.aliyuncs.com/google_containers/pause:3.10.1

所有master和node节点安装kubeadm等相关包

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
root@master1:~# curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.36/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
root@master1:~# echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.36/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
root@master1:~# apt update
root@master1:~# apt-cache madison kubeadm
   kubeadm | 1.36.1-1.1 | https://pkgs.k8s.io/core:/stable:/v1.36/deb  Packages
   kubeadm | 1.36.0-1.1 | https://pkgs.k8s.io/core:/stable:/v1.36/deb  Packages
root@master1:~# apt list kubeadm kubectl kubelet
kubeadm/unknown,now 1.36.1-1.1 amd64 [installed]
kubeadm/unknown 1.36.1-1.1 arm64
kubeadm/unknown 1.36.1-1.1 ppc64el
kubeadm/unknown 1.36.1-1.1 s390x
kubectl/unknown,now 1.36.1-1.1 amd64 [installed]
kubectl/unknown 1.36.1-1.1 arm64
kubectl/unknown 1.36.1-1.1 ppc64el
kubectl/unknown 1.36.1-1.1 s390x
kubelet/unknown,now 1.36.1-1.1 amd64 [installed]
kubelet/unknown 1.36.1-1.1 arm64
kubelet/unknown 1.36.1-1.1 ppc64el
kubelet/unknown 1.36.1-1.1 s390x
root@master1:~# apt-get install -y kubelet kubeadm kubectl
# 实现kubeadm命令补全
root@master1:~# kubeadm completion bash > /etc/profile.d/kubeadm_completion.sh

在第一台master初始化集群

在第一台master上执行初始化操作，有以下两种方式

  • 基于命令行初始化，此方法虽然灵活，但是后期配置不可追溯
  • 调用初始化对应的配置文件进行初始化，此方法繁琐，但是所有配置都可以控制和回溯
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    root@master1:~# K8S_RELEASE_VERSION=1.36.1
    root@master1:~# kubeadm init --kubernetes-version=v${K8S_RELEASE_VERSION} \
    --control-plane-endpoint kubeapi.luo.org --pod-network-cidr 10.244.0.0/16 \
    --service-cidr 10.96.0.0/12 --token-ttl=0  --upload-certs \
    --cri-socket=unix:///run/cri-dockerd.sock  #注意： v1.24版本+需要添加此项才能支持docker
    ....
    Your Kubernetes control-plane has initialized successfully!

    To start using your cluster, you need to run the following as a regular user:

      mkdir -p $HOME/.kube
      sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
      sudo chown $(id -u):$(id -g) $HOME/.kube/config

    Alternatively, if you are the root user, you can run:

      export KUBECONFIG=/etc/kubernetes/admin.conf

    You should now deploy a pod network to the cluster.
    Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
      https://kubernetes.io/docs/concepts/cluster-administration/addons/

    You can now join any number of control-plane nodes running the following command on each as root:

      kubeadm join kubeapi.luo.org:6443 --token h62s77.px47n0pau2446fjh \
       --discovery-token-ca-cert-hash sha256:61ed5fe51148022bf4a60296a49addace05aa5dc1206310c786407c1f36dd2ad \
       --control-plane --certificate-key 408bceb1f1d0157a69960291f10a06cfa5d1cadff4e2550cbed742d9e85f15b0 --cri-socket=unix:///run/cri-dockerd.sock

    Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
    As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
    "kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

    Then you can join any number of worker nodes by running the following on each as root:

    kubeadm join kubeapi.luo.org:6443 --token h62s77.px47n0pau2446fjh \
       --discovery-token-ca-cert-hash sha256:61ed5fe51148022bf4a60296a49addace05aa5dc1206310c786407c1f36dd2ad --cri-socket=unix:///run/cri-dockerd.sock

    root@master1:~# mkdir -p $HOME/.kube
    root@master1:~# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    root@master1:~# chown $(id -u):$(id -g) $HOME/.kube/config
    root@master1:~# kubectl get node
    NAME              STATUS     ROLES           AGE   VERSION
    master1.luo.org   NotReady   control-plane   64s   v1.36.1
    root@master1:~# kubectl get pod -A 
    NAMESPACE     NAME                                      READY   STATUS    RESTARTS   AGE
    kube-system   coredns-589f44dc88-62n7g                  0/1     Pending   0          74s
    kube-system   coredns-589f44dc88-hx7hg                  0/1     Pending   0          74s
    kube-system   etcd-master1.luo.org                      1/1     Running   0          79s
    kube-system   kube-apiserver-master1.luo.org            1/1     Running   0          82s
    kube-system   kube-controller-manager-master1.luo.org   1/1     Running   0          79s
    kube-system   kube-proxy-42blr                          1/1     Running   0          74s
    kube-system   kube-scheduler-master1.luo.org            1/1     Running   0          79s

部署网络插件

这里选用flannel

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
root@master1:~# kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
namespace/kube-flannel created
serviceaccount/flannel created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created
root@master1:~# kubectl get pod -A 
NAMESPACE      NAME                                      READY   STATUS     RESTARTS   AGE
kube-flannel   kube-flannel-ds-thh2z                     0/1     Init:0/2   0          7s
kube-system    coredns-589f44dc88-62n7g                  0/1     Pending    0          3m26s
kube-system    coredns-589f44dc88-hx7hg                  0/1     Pending    0          3m26s
kube-system    etcd-master1.luo.org                      1/1     Running    0          3m31s
kube-system    kube-apiserver-master1.luo.org            1/1     Running    0          3m34s
kube-system    kube-controller-manager-master1.luo.org   1/1     Running    0          3m31s
kube-system    kube-proxy-42blr                          1/1     Running    0          3m26s
kube-system    kube-scheduler-master1.luo.org            1/1     Running    0          3m31s
root@master1:~# kubectl get pod -A 
NAMESPACE      NAME                                      READY   STATUS    RESTARTS   AGE
kube-flannel   kube-flannel-ds-thh2z                     1/1     Running   0          89s
kube-system    coredns-589f44dc88-62n7g                  1/1     Running   0          4m48s
kube-system    coredns-589f44dc88-hx7hg                  1/1     Running   0          4m48s
kube-system    etcd-master1.luo.org                      1/1     Running   0          4m53s
kube-system    kube-apiserver-master1.luo.org            1/1     Running   0          4m56s
kube-system    kube-controller-manager-master1.luo.org   1/1     Running   0          4m53s
kube-system    kube-proxy-42blr                          1/1     Running   0          4m48s
kube-system    kube-scheduler-master1.luo.org            1/1     Running   0          4m53s

第二台master加入集群

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
root@master2:/etc/apt/sources.list.d# kubeadm join kubeapi.luo.org:6443 --token h62s77.px47n0pau2446fjh \
    --discovery-token-ca-cert-hash sha256:61ed5fe51148022bf4a60296a49addace05aa5dc1206310c786407c1f36dd2ad \
    --control-plane --certificate-key 408bceb1f1d0157a69960291f10a06cfa5d1cadff4e2550cbed742d9e85f15b0 --cri-socket=unix:///run/cri-dockerd.sock

....
* Certificate signing request was sent to apiserver and approval was received.
* The Kubelet was informed of the new secure connection details.
* Control plane label and taint were applied to the new node.
* The Kubernetes control plane instances scaled up.
* A new etcd member was added to the local/stacked etcd cluster.

To start administering your cluster from this node, you need to run the following as a regular user:

	mkdir -p $HOME/.kube
	sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
	sudo chown $(id -u):$(id -g) $HOME/.kube/config

Run 'kubectl get nodes' to see this node join the cluster.

root@master2:/etc/apt/sources.list.d# mkdir -p $HOME/.kube
root@master2:/etc/apt/sources.list.d# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
	sudo chown $(id -u):$(id -g) $HOME/.kube/configroot@master2:/etc/apt/sources.list.d# 
root@master2:/etc/apt/sources.list.d# kubectl get nodes 
NAME              STATUS     ROLES           AGE   VERSION
master1.luo.org   Ready      control-plane   11m   v1.36.1
master2.luo.org   NotReady   control-plane   10s   v1.36.1

第三台master加入集群

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
root@master3:/usr/local/src/docker#   kubeadm join kubeapi.luo.org:6443 --token h62s77.px47n0pau2446fjh \
    --discovery-token-ca-cert-hash sha256:61ed5fe51148022bf4a60296a49addace05aa5dc1206310c786407c1f36dd2ad \
    --control-plane --certificate-key 408bceb1f1d0157a69960291f10a06cfa5d1cadff4e2550cbed742d9e85f15b0 --cri-socket=unix:///run/cri-dockerd.sock
...
To start administering your cluster from this node, you need to run the following as a regular user:

	mkdir -p $HOME/.kube
	sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
	sudo chown $(id -u):$(id -g) $HOME/.kube/config

Run 'kubectl get nodes' to see this node join the cluster.

root@master3:/usr/local/src/docker# mkdir -p $HOME/.kube
root@master3:/usr/local/src/docker# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
	sudo chown $(id -u):$(id -g) $HOME/.kube/configroot@master3:/usr/local/src/docker# 
root@master3:/usr/local/src/docker# 
root@master3:/usr/local/src/docker# kubectl get nodes 
NAME              STATUS     ROLES           AGE     VERSION
master1.luo.org   Ready      control-plane   16m     v1.36.1
master2.luo.org   Ready      control-plane   4m49s   v1.36.1
master3.luo.org   NotReady   control-plane   14s     v1.36.1

Node节点加入集群

1
2
3
4
5
6
root@node1:/usr/local/src/docker# kubeadm join kubeapi.luo.org:6443 --token h62s77.px47n0pau2446fjh \
   --discovery-token-ca-cert-hash sha256:61ed5fe51148022bf4a60296a49addace05aa5dc1206310c786407c1f36dd2ad --cri-socket=unix:///run/cri-dockerd.sock
root@node2:/usr/local/src/docker# kubeadm join kubeapi.luo.org:6443 --token h62s77.px47n0pau2446fjh \
   --discovery-token-ca-cert-hash sha256:61ed5fe51148022bf4a60296a49addace05aa5dc1206310c786407c1f36dd2ad --cri-socket=unix:///run/cri-dockerd.sock
root@node3:/usr/local/src/docker# kubeadm join kubeapi.luo.org:6443 --token h62s77.px47n0pau2446fjh \
   --discovery-token-ca-cert-hash sha256:61ed5fe51148022bf4a60296a49addace05aa5dc1206310c786407c1f36dd2ad --cri-socket=unix:///run/cri-dockerd.sock

查看集群状态

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
root@master1:~# kubectl get node 
NAME              STATUS     ROLES           AGE     VERSION
master1.luo.org   Ready      control-plane   18m     v1.36.1
master2.luo.org   Ready      control-plane   6m33s   v1.36.1
master3.luo.org   Ready      control-plane   118s    v1.36.1
node1.luo.org     NotReady   <none>          24s     v1.36.1
node2.luo.org     NotReady   <none>          20s     v1.36.1
node3.luo.org     NotReady   <none>          17s     v1.36.1
# 三台node节点notready 是因为网络插件还没部署好
root@master1:~# kubectl get pod -A 
NAMESPACE      NAME                                      READY   STATUS              RESTARTS   AGE
kube-flannel   kube-flannel-ds-4tb9b                     1/1     Running             0          6m39s
kube-flannel   kube-flannel-ds-9fzj7                     0/1     Init:0/2            0          26s
kube-flannel   kube-flannel-ds-9wv7b                     1/1     Running             0          2m4s
kube-flannel   kube-flannel-ds-hwq8f                     0/1     Init:0/2            0          23s
kube-flannel   kube-flannel-ds-thh2z                     1/1     Running             0          14m
kube-flannel   kube-flannel-ds-vm22x                     0/1     Init:1/2            0          30s
kube-system    coredns-589f44dc88-62n7g                  1/1     Running             0          18m
kube-system    coredns-589f44dc88-hx7hg                  1/1     Running             0          18m
kube-system    etcd-master1.luo.org                      1/1     Running             0          18m
kube-system    etcd-master2.luo.org                      1/1     Running             0          6m32s
kube-system    etcd-master3.luo.org                      1/1     Running             0          101s
kube-system    kube-apiserver-master1.luo.org            1/1     Running             0          18m
kube-system    kube-apiserver-master2.luo.org            1/1     Running             0          6m32s
kube-system    kube-apiserver-master3.luo.org            1/1     Running             0          101s
kube-system    kube-controller-manager-master1.luo.org   1/1     Running             0          18m
kube-system    kube-controller-manager-master2.luo.org   1/1     Running             0          6m32s
kube-system    kube-controller-manager-master3.luo.org   1/1     Running             0          101s
kube-system    kube-proxy-42blr                          1/1     Running             0          18m
kube-system    kube-proxy-44shr                          1/1     Running             0          2m4s
kube-system    kube-proxy-b4t4r                          0/1     ContainerCreating   0          30s
kube-system    kube-proxy-c7ml9                          1/1     Running             0          6m39s
kube-system    kube-proxy-csrm9                          0/1     ContainerCreating   0          23s
kube-system    kube-proxy-wqq4q                          0/1     ContainerCreating   0          26s
kube-system    kube-scheduler-master1.luo.org            1/1     Running             0          18m
kube-system    kube-scheduler-master2.luo.org            1/1     Running             0          6m32s
kube-system    kube-scheduler-master3.luo.org            1/1     Running             0          100s

最终完整结果

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
root@master1:~# kubectl get pod -A 
NAMESPACE      NAME                                      READY   STATUS    RESTARTS   AGE
kube-flannel   kube-flannel-ds-4tb9b                     1/1     Running   0          8m1s
kube-flannel   kube-flannel-ds-9fzj7                     1/1     Running   0          108s
kube-flannel   kube-flannel-ds-9wv7b                     1/1     Running   0          3m26s
kube-flannel   kube-flannel-ds-hwq8f                     1/1     Running   0          105s
kube-flannel   kube-flannel-ds-thh2z                     1/1     Running   0          16m
kube-flannel   kube-flannel-ds-vm22x                     1/1     Running   0          112s
kube-system    coredns-589f44dc88-62n7g                  1/1     Running   0          19m
kube-system    coredns-589f44dc88-hx7hg                  1/1     Running   0          19m
kube-system    etcd-master1.luo.org                      1/1     Running   0          19m
kube-system    etcd-master2.luo.org                      1/1     Running   0          7m54s
kube-system    etcd-master3.luo.org                      1/1     Running   0          3m3s
kube-system    kube-apiserver-master1.luo.org            1/1     Running   0          19m
kube-system    kube-apiserver-master2.luo.org            1/1     Running   0          7m54s
kube-system    kube-apiserver-master3.luo.org            1/1     Running   0          3m3s
kube-system    kube-controller-manager-master1.luo.org   1/1     Running   0          19m
kube-system    kube-controller-manager-master2.luo.org   1/1     Running   0          7m54s
kube-system    kube-controller-manager-master3.luo.org   1/1     Running   0          3m3s
kube-system    kube-proxy-42blr                          1/1     Running   0          19m
kube-system    kube-proxy-44shr                          1/1     Running   0          3m26s
kube-system    kube-proxy-b4t4r                          1/1     Running   0          112s
kube-system    kube-proxy-c7ml9                          1/1     Running   0          8m1s
kube-system    kube-proxy-csrm9                          1/1     Running   0          105s
kube-system    kube-proxy-wqq4q                          1/1     Running   0          108s
kube-system    kube-scheduler-master1.luo.org            1/1     Running   0          19m
kube-system    kube-scheduler-master2.luo.org            1/1     Running   0          7m54s
kube-system    kube-scheduler-master3.luo.org            1/1     Running   0          3m2s
root@master1:~# kubectl get nodes 
NAME              STATUS   ROLES           AGE     VERSION
master1.luo.org   Ready    control-plane   19m     v1.36.1
master2.luo.org   Ready    control-plane   8m5s    v1.36.1
master3.luo.org   Ready    control-plane   3m30s   v1.36.1
node1.luo.org     Ready    <none>          116s    v1.36.1
node2.luo.org     Ready    <none>          112s    v1.36.1
node3.luo.org     Ready    <none>          109s    v1.36.1
云原生
#云原生
k8s-通过kubeadm安装最新版
https://www.dklwj.com/2026/06/k8s-kubeadm-1.36-install.html
作者
阿伟
发布于
2026年6月7日
许可协议