二进制部署Kubernetes高可用集群

一、环境说明

1.1基本环境配置

该环境使用的是vmware虚拟的六台服务器，使用二进制的方式安装部署可用于生产环境的k8s集群！具体信息规划如下：

主机名 ip地址 说明
k8s-master01 192.168.0.81 主节点
k8s-master02 192.168.0.82 主节点
k8s-master03 192.168.0.83 主节点
k8s-node01 192.168.0.84 从节点
k8s-node02 192.168.0.85 从节点
k8s-node03 192.168.0.86 从节点
keepalived虚拟ip 192.168.0.236 keepalived虚拟ip(不占用机器)
配置信息 备注
系统版本 Rocky Linux 9.5
Containerd latest
Pod网段 172.16.0.0/16
Service网段 10.96.0.0/16

如果需要修改这些网段，统一替换。Pod网段、Service网段以及宿主机网段不能重复，不能重复！！！

VIP（虚拟IP）不要和公司内网IP重复，首先去ping一下，不通才可用。VIP需要和你的主机在同一个局域网内（不是直接用我的VIP，根据自身情况定义）！

公有云上搭建VIP是公有云的负载均衡的IP，比如阿里云的内网SLB/NLB的地址，腾讯云内网ELB的地址。不需要再搭建keepalived和haproxy。

1.2基础配置

所有节点修改主机名：(其他节点按规划修改)

1
hostnamectl set-hostname k8s-master01

所有节点配置hosts，修改后的文件内容如下：

1
2
3
4
5
6
7
8
9
[root@k8s-node03 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.0.81 k8s-master01
192.168.0.82 k8s-master02
192.168.0.83 k8s-master03
192.168.0.84 k8s-node01
192.168.0.85 k8s-node02
192.168.0.86 k8s-node03

所有节点yum源配置：

1
2
3
4
5
6
sed -e 's|^mirrorlist=|#mirrorlist=|g' \
    -e 's|^#baseurl=http://dl.rockylinux.org/$contentdir|baseurl=https://mirrors.aliyun.com/rockylinux|g' \
    -i.bak \
    /etc/yum.repos.d/rocky*.repo

yum makecache

所有节点必备工具安装：

1
yum install wget jq psmisc vim net-tools telnet yum-utils device-mapper-persistent-data lvm2 git rsyslog -y

所有节点关闭防火墙、selinux、dnsmasq、swap、开启rsyslog。服务器配置如下：

1
2
3
4
5
6
systemctl disable --now firewalld
systemctl disable --now dnsmasq  #如果报错Failed to disable unit: Unit file dnsmasq.service does not exist.忽略即可
setenforce 0
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/sysconfig/selinux
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config
systemctl enable --now rsyslog

所有节点关闭swap分区：

1
2
swapoff -a && sysctl -w vm.swappiness=0
sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab

所有节点安装ntpdate：

1
2
3
dnf install epel-release -y
dnf config-manager --set-enabled epel
dnf install ntpsec -y

所有节点同步时间并配置上海时区：

1
2
3
4
5
6
7
8
9
ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
echo 'Asia/Shanghai' >/etc/timezone
ntpdate time2.aliyun.com

# 加入到crontab中
crontab -e
#将下面内容添加到计划任务
*/5 * * * * /usr/sbin/ntpdate time2.aliyun.com

生产环境时间同步方案：推荐安装并使用 chrony，

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
# 生产环境不建议长期依赖定时任务每 5 分钟一次的 ntpdate。更推荐使用 chrony。
# RHEL/CentOS/Anolis/Rocky/AlmaLinux

# 安装chrony服务
yum install -y chrony  或者  dnf install -y chrony


#配置 NTP 服务器：
vi /etc/chrony.conf
# 在文件开头加入下面123行内容，并注释第6行内容
  1 server time2.aliyun.com iburst
  2 server ntp1.aliyun.com iburst
  3 server ntp2.aliyun.com iburst
  4 # Use public servers from the pool.ntp.org project.
  5 # Please consider joining the pool (https://www.pool.ntp.org/join.html).
  6 #pool 2.rocky.pool.ntp.org iburst
  7
  8 # Use NTP servers from DHCP.
  9 sourcedir /run/chrony-dhcp
 10
 11 # Record the rate at which the system clock gains/losses time.
 12 driftfile /var/lib/chrony/drift
 
 
 # 保存退出，启动服务并设置开机自启
 systemctl enable --now chronyd
 
 # 强制立即进行一次校正：
 chronyc makestep
 
 # 此时可以删除定时任务中的*/5 * * * * /usr/sbin/ntpdate time2.aliyun.com

所有节点配置limit：

1
2
3
4
5
6
7
8
9
10
11
ulimit -SHn 65535

vim /etc/security/limits.conf
# 向文件末尾添加如下内容
* soft nofile 65536
* hard nofile 131072
* soft nproc 65535
* hard nproc 655350
* soft memlock unlimited
* hard memlock unlimited

所有节点升级系统并重启（如果用于学习环境可以不用操作）：

1
yum update -y

只操作Master01节点免密钥登录其他节点，安装过程中生成配置文件和证书均在Master01上操作，集群管理也在Master01上操作：

注意：公有云环境，可能需要把kubectl放在一个非Master节点上，根据实际情况定

1
2
3
4
5
for i in k8s-master01 k8s-master02 k8s-master03 k8s-node01 k8s-node02 k8s-node03;do ssh-copy-id -i .ssh/id_rsa.pub $i;done

#验证是否能免密登录，输出类似下面信息表示配置正确
[root@k8s-master01 ~]# ssh k8s-node03
Last login: Fri Dec 13 21:50:33 2024 from 192.168.0.102

Master01节点下载安装所有的源码文件：

1
2
3
4
5
6
cd /root/ ; git clone https://gitee.com/dukuan/k8s-ha-install.git

[root@k8s-master01 ~]# ll
total 4
-rw-------. 1 root root 1239 Dec  8 11:17 anaconda-ks.cfg
drwxr-xr-x. 5 root root  141 Dec 13 22:39 k8s-ha-install  #这个目录要存在

1.3内核配置

所有节点安装ipvsadm：

1
yum install ipvsadm ipset sysstat conntrack libseccomp -y

所有节点配置ipvs模块：

1
2
3
4
5
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack

所有节点创建ipvs.conf，并配置开机自动加载：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
vim /etc/modules-load.d/ipvs.conf 
# 加入以下内容
ip_vs
ip_vs_lc
ip_vs_wlc
ip_vs_rr
ip_vs_wrr
ip_vs_lblc
ip_vs_lblcr
ip_vs_dh
ip_vs_sh
ip_vs_fo
ip_vs_nq
ip_vs_sed
ip_vs_ftp
ip_vs_sh
nf_conntrack
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip

所有节点然后执行systemctl enable –now systemd-modules-load.service即可（报错不用管）

1
systemctl enable --now systemd-modules-load.service

所有节点内核优化配置：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
cat <<EOF > /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
fs.may_detach_mounts = 1
net.ipv4.conf.all.route_localnet = 1
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
fs.file-max=52706963
fs.nr_open=52706963
net.netfilter.nf_conntrack_max=2310720

net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl =15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.ip_conntrack_max = 65536
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 16384
EOF

所有节点应用配置：

1
sysctl --system

所有节点配置完内核后，重启机器，之后查看内核模块是否已自动加载：

1
2
reboot
lsmod | grep --color=auto -e ip_vs -e nf_conntrack

1.4高可用组件HAProxy和KeepAlived安装

注意：如果安装的不是高可用集群，haproxy和keepalived无需安装

注意：公有云要用公有云自带的负载均衡，比如阿里云的SLB、NLB，腾讯云的ELB，用来替代haproxy和keepalived，因为公有云大部分都是不支持keepalived的。

所有Master节点通过yum安装HAProxy和KeepAlived：

1
yum install keepalived haproxy -y

所有Master节点配置HAProxy，需要注意IP部分，修改为自己的ip地址：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
[root@k8s-master03 ~]# mkdir /etc/haproxy
mkdir: cannot create directory ‘/etc/haproxy’: File exists  #已经存在，报错忽略

[root@k8s-master03 ~]# vim /etc/haproxy/haproxy.cfg
#删除文件内所有原内容，加入下面配置
global
  maxconn  2000
  ulimit-n  16384
  log  127.0.0.1 local0 err
  stats timeout 30s

defaults
  log global
  mode  http
  option  httplog
  timeout connect 5000
  timeout client  50000
  timeout server  50000
  timeout http-request 15s
  timeout http-keep-alive 15s

frontend monitor-in
  bind *:33305
  mode http
  option httplog
  monitor-uri /monitor

frontend k8s-master
  bind 0.0.0.0:8443
  bind 127.0.0.1:8443
  mode tcp
  option tcplog
  tcp-request inspect-delay 5s
  default_backend k8s-master

backend k8s-master
  mode tcp
  option tcplog
  option tcp-check
  balance roundrobin
  default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
  server k8s-master01	192.168.0.81:6443  check
  server k8s-master02	192.168.0.82:6443  check
  server k8s-master03	192.168.0.83:6443  check

所有Master节点配置KeepAlived，需要注意黄色部分的配置。

1.Master01节点的配置：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
[root@k8s-master01 ~]# mkdir /etc/keepalived
mkdir: cannot create directory ‘/etc/keepalived’: File exists #报错忽略，因为目录已经存在


[root@k8s-master01 ~]# vim /etc/keepalived/keepalived.conf
#删除文件内原有配置，加入以下内容
! Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
script_user root
    enable_script_security
}
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
    interval 5
    weight -5
    fall 2
rise 1
}
vrrp_instance VI_1 {
    state MASTER
    interface eth0  #eth0表示网卡的名称，有些机器网卡名是ens160，根据自己的情况修改
    mcast_src_ip 192.168.0.81  #k8s-master01节点的ip
    virtual_router_id 51
    priority 101
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
        192.168.0.236  #这里的ip是实现规划的虚拟VIP地址
    }
    track_script {
       chk_apiserver
    }
}

2.Master02节点的配置：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
[root@k8s-master02 ~]# vim /etc/keepalived/keepalived.conf
#删除文件内原有配置，加入以下内容
! Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
script_user root
    enable_script_security
}
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
   interval 5
    weight -5
    fall 2  
rise 1
}
vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    mcast_src_ip 192.168.0.82
    virtual_router_id 51
    priority 100
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
        192.168.0.236
    }
    track_script {
       chk_apiserver
    }
}

3.Master03节点的配置：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
[root@k8s-master03 ~]# vim /etc/keepalived/keepalived.conf
#删除文件内原有配置，加入以下内容
! Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
script_user root
    enable_script_security
}
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
 interval 5
    weight -5
    fall 2  
rise 1
}
vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    mcast_src_ip 192.168.0.83
    virtual_router_id 51
    priority 100
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
        192.168.0.236
    }
    track_script {
       chk_apiserver
    }
}

所有master节点配置KeepAlived健康检查文件：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
vim /etc/keepalived/check_apiserver.sh 
#向文件添加下面配置，三台master节点的配置一样
#!/bin/bash

err=0
for k in $(seq 1 3)
do
    check_code=$(pgrep haproxy)
    if [[ $check_code == "" ]]; then
        err=$(expr $err + 1)
        sleep 1
        continue
    else
        err=0
        break
    fi
done

if [[ $err != "0" ]]; then
    echo "systemctl stop keepalived"
    /usr/bin/systemctl stop keepalived
    exit 1
else
    exit 0
fi

给文件添加执行权限：

1
chmod +x /etc/keepalived/check_apiserver.sh

所有master节点启动haproxy和keepalived：

1
2
3
4
5
6
7
systemctl daemon-reload
systemctl enable --now haproxy
systemctl enable --now keepalived

#检查haproxy和keepalived是否正常启动，看到active (running)表示正确
systemctl status haproxy
systemctl status keepalived

重要：如果安装了keepalived和haproxy，需要测试keepalived是否是正常的

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
#所有节点测试是否能和虚拟VIP(192.168.0.236)通信
[root@k8s-node03 ~]# ping 192.168.0.236 -c 4
PING 192.168.0.236 (192.168.0.236) 56(84) bytes of data.
64 bytes from 192.168.0.236: icmp_seq=1 ttl=64 time=0.828 ms
64 bytes from 192.168.0.236: icmp_seq=2 ttl=64 time=0.097 ms
64 bytes from 192.168.0.236: icmp_seq=3 ttl=64 time=0.254 ms
64 bytes from 192.168.0.236: icmp_seq=4 ttl=64 time=1.37 ms

--- 192.168.0.236 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3055ms
rtt min/avg/max/mdev = 0.097/0.637/1.371/0.503 ms

#探测8443端口是否正常
[root@k8s-master01 ~]# telnet 192.168.0.236 8443
Trying 192.168.0.236...
Connected to 192.168.0.236.
Escape character is '^]'.  #要看到这个标识符才行
Connection closed by foreign host.

如果ping不通且telnet没有出现 ] ，则认为VIP不可以，不可在继续往下执行，需要排查keepalived的问题，比如防火墙和selinux，haproxy和keepalived的状态，监听端口等

所有节点查看防火墙状态必须为disable和inactive：systemctl status firewalld

所有节点查看selinux状态，必须为disable：getenforce

master节点查看haproxy和keepalived状态：systemctl status keepalived haproxy

master节点查看监听端口：netstat -lntp

如果以上都没有问题，需要确认：

  1. 是否是公有云机器
  2. 是否是私有云机器（类似OpenStack）

上述公有云一般都是不支持keepalived，私有云可能也有限制，需要和自己的私有云管理员咨询

二、Runtime安装

如果安装的版本低于1.24，选择Docker和Containerd均可，高于1.24建议选择Containerd作为Runtime，不再推荐使用Docker作为Runtime。

2.1 安装Containerd

所有节点配置安装源：

1
2
3
yum install wget jq psmisc vim net-tools telnet yum-utils device-mapper-persistent-data lvm2 git -y

yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

所有节点安装docker-ce（如果在以前已经安装过，需要重新安装更新一下）：

1
yum install docker-ce containerd -y

可以无需启动Docker，只需要配置和启动Containerd即可(所有节点操作)。

1
2
3
4
5
cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF

所有节点加载模块：

1
2
modprobe -- overlay
modprobe -- br_netfilter

所有节点，配置Containerd所需的内核：

1
2
3
4
5
6
cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

所有节点加载内核：

1
sysctl --system

所有节点生成Containerd的配置文件：

1
2
3
mkdir -p /etc/containerd

containerd config default | tee /etc/containerd/config.toml

所有节点更改Containerd的Cgroup和Pause镜像配置：

1
2
3
4
5
sed -i 's#SystemdCgroup = false#SystemdCgroup = true#g' /etc/containerd/config.toml
sed -i 's#k8s.gcr.io/pause#registry.cn-hangzhou.aliyuncs.com/google_containers/pause#g'  /etc/containerd/config.toml
sed -i 's#registry.gcr.io/pause#registry.cn-hangzhou.aliyuncs.com/google_containers/pause#g'  /etc/containerd/config.toml
sed -i 's#registry.k8s.io/pause#registry.cn-hangzhou.aliyuncs.com/google_containers/pause#g'  /etc/containerd/config.toml

所有节点启动Containerd，并配置开机自启动：

1
2
3
systemctl daemon-reload
systemctl enable --now containerd

所有节点配置crictl客户端连接的运行时位置：

1
2
3
4
5
6
7
cat > /etc/crictl.yaml <<EOF
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
EOF

三、k8s及etcd安装

Master01下载kubernetes安装包（1.31.0需要更改为你看到的最新版本）：

1
[root@k8s-master01 ~]# wget https://dl.k8s.io/v1.31.0/kubernetes-server-linux-amd64.tar.gz

最新版获取地址：https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/

Master01下载etcd安装包：

下载地址：https://github.com/etcd-io/etcd/releases/

1
[root@k8s-master01 ~]# wget https://github.com/etcd-io/etcd/releases/download/v3.5.17/etcd-v3.5.17-linux-amd64.tar.gz

也可以直接浏览器下载软件包，然后上传到k8s-master01上：

这里使用的etcd版本是3.5.17，k8s版本是1.31：

以下操作都在master01执行：

解压kubernetes安装文件：

1
2
3
4
5
6
7
8
9
10
11
[root@k8s-master01 ~]# tar -xf kubernetes-server-linux-amd64.tar.gz --strip-components=3 -C /usr/local/bin kubernetes/server/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy}

#解压后/usr/local/bin目录会有以下文件
[root@k8s-master01 ~]# ll /usr/local/bin
total 426500
-rwxr-xr-x 1 root root 90542232 Dec 10 19:52 kube-apiserver
-rwxr-xr-x 1 root root 84742296 Dec 10 19:52 kube-controller-manager
-rwxr-xr-x 1 root root 56381592 Dec 10 19:52 kubectl
-rwxr-xr-x 1 root root 76902744 Dec 10 19:52 kubelet
-rwxr-xr-x 1 root root 64417944 Dec 10 19:52 kube-proxy
-rwxr-xr-x 1 root root 63725720 Dec 10 19:52 kube-scheduler

解压etcd安装文件：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[root@k8s-master01 ~]# tar -zxvf etcd-v3.5.17-linux-amd64.tar.gz --strip-components=1 -C /usr/local/bin etcd-v3.5.17-linux-amd64/etcd{,ctl}


#解压后/usr/local/bin目录会多出etcd和etcdctl文件
[root@k8s-master01 ~]# ll /usr/local/bin
total 467060
-rwxr-xr-x 1 1000 1000 23625880 Nov 13 00:32 etcd
-rwxr-xr-x 1 1000 1000 17899672 Nov 13 00:32 etcdctl
-rwxr-xr-x 1 root root 90542232 Dec 10 19:52 kube-apiserver
-rwxr-xr-x 1 root root 84742296 Dec 10 19:52 kube-controller-manager
-rwxr-xr-x 1 root root 56381592 Dec 10 19:52 kubectl
-rwxr-xr-x 1 root root 76902744 Dec 10 19:52 kubelet
-rwxr-xr-x 1 root root 64417944 Dec 10 19:52 kube-proxy
-rwxr-xr-x 1 root root 63725720 Dec 10 19:52 kube-scheduler

查看版本：

1
2
3
4
5
[root@k8s-master01 ~]# kubelet --version
Kubernetes v1.31.4
[root@k8s-master01 ~]# etcdctl version
etcdctl version: 3.5.17
API version: 3.5

将组件发送到其他节点：

1
2
3
4
5
6
MasterNodes='k8s-master02 k8s-master03'
WorkNodes='k8s-node01 k8s-node02 k8s-node03'
#将所有组件发送到master节点
for NODE in $MasterNodes; do echo $NODE; scp /usr/local/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy} $NODE:/usr/local/bin/; scp /usr/local/bin/etcd* $NODE:/usr/local/bin/; done
#将kubelet和kube-proxy组件发送到node节点
for NODE in $WorkNodes; do     scp /usr/local/bin/kube{let,-proxy} $NODE:/usr/local/bin/ ; done

切换分支，切换分支，切换分支！！！

Master01节点切换到1.31.x分支（其他版本可以切换到其他分支，.x即可，不需要更改为具体的小版本）：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
[root@k8s-master01 ~]# cd k8s-ha-install/

#查看所有分支
[root@k8s-master01 k8s-ha-install]# git branch -a
* master
  remotes/origin/HEAD -> origin/master
  remotes/origin/manual-installation
  remotes/origin/manual-installation-v1.16.x
  remotes/origin/manual-installation-v1.17.x
  remotes/origin/manual-installation-v1.18.x
  remotes/origin/manual-installation-v1.19.x
  remotes/origin/manual-installation-v1.20.x
  remotes/origin/manual-installation-v1.20.x-csi-hostpath
  remotes/origin/manual-installation-v1.21.x
  remotes/origin/manual-installation-v1.22.x
  remotes/origin/manual-installation-v1.23.x
  remotes/origin/manual-installation-v1.24.x
  remotes/origin/manual-installation-v1.25.x
  remotes/origin/manual-installation-v1.26.x
  remotes/origin/manual-installation-v1.27.x
  remotes/origin/manual-installation-v1.28.x
  remotes/origin/manual-installation-v1.29.x
  remotes/origin/manual-installation-v1.30.x
  remotes/origin/manual-installation-v1.31.x
  remotes/origin/manual-installation-v1.32.x
  remotes/origin/master
  
#切换到对应版本的分支
[root@k8s-master01 k8s-ha-install]# git checkout manual-installation-v1.31.x
branch 'manual-installation-v1.31.x' set up to track 'origin/manual-installation-v1.31.x'.
Switched to a new branch 'manual-installation-v1.31.x'

四、生成证书

二进制安装最关键步骤，一步错误全盘皆输，一定要注意每个步骤都要是正确的

Master01下载生成证书工具：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
#在线下载
wget "https://pkg.cfssl.org/R1.2/cfssl_linux-amd64" -O /usr/local/bin/cfssl
wget "https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64" -O /usr/local/bin/cfssljson
chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson


#如果wget在线下载不成功可以通过浏览器下载，然后上传到服务器/usr/local/bin/目录中（如果从本地上传记得修改下文件名cfssl  cfssljson）
[root@k8s-master01 ~]# cd /usr/local/bin/
[root@k8s-master01 bin]# mv cfssl_linux-amd64 cfssl
[root@k8s-master01 bin]# mv cfssljson_linux-amd64 cfssljson

#添加可执行权限
[root@k8s-master01 bin]# chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson

[root@k8s-master01 k8s-ha-install]# ll /usr/local/bin/
total 479424
-rwxr-xr-x 1 root root 10376657 Dec 14 15:30 cfssl
-rwxr-xr-x 1 root root  2277873 Dec 14 15:31 cfssljson
-rwxr-xr-x 1 1000 1000 23625880 Nov 13 00:32 etcd
-rwxr-xr-x 1 1000 1000 17899672 Nov 13 00:32 etcdctl
-rwxr-xr-x 1 root root 90542232 Dec 10 19:52 kube-apiserver
-rwxr-xr-x 1 root root 84742296 Dec 10 19:52 kube-controller-manager
-rwxr-xr-x 1 root root 56381592 Dec 10 19:52 kubectl
-rwxr-xr-x 1 root root 76902744 Dec 10 19:52 kubelet
-rwxr-xr-x 1 root root 64417944 Dec 10 19:52 kube-proxy
-rwxr-xr-x 1 root root 63725720 Dec 10 19:52 kube-scheduler

4.1生成etcd证书

所有Master节点创建etcd证书目录：

1
2
#存放etcd证书
mkdir /etc/etcd/ssl -p

所有节点创建kubernetes证书目录：

1
2
#存放kubernetes相关证书
mkdir -p /etc/kubernetes/pki

Master01节点生成etcd证书:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
#生成证书的CSR（证书签名请求文件，配置了一些域名、公司、单位）文件：
cd /root/k8s-ha-install/pki

# 生成etcd CA证书和CA证书的key
cfssl gencert -initca etcd-ca-csr.json | cfssljson -bare /etc/etcd/ssl/etcd-ca

#用ca证书颁发etcd客户端证书，执行如下命令：
#如果etcd没有部署在三台主节点，而是单独的节点，那么-hostname=127.0.0.1,k8s-master01,k8s-master02,k8s-master03,192.168.0.81,192.168.0.82,192.168.0.83 就改成etcd所在节点的主机名和ip
cfssl gencert \
   -ca=/etc/etcd/ssl/etcd-ca.pem \
   -ca-key=/etc/etcd/ssl/etcd-ca-key.pem \
   -config=ca-config.json \
   -hostname=127.0.0.1,k8s-master01,k8s-master02,k8s-master03,192.168.0.81,192.168.0.82,192.168.0.83 \
   -profile=kubernetes \
   etcd-csr.json | cfssljson -bare /etc/etcd/ssl/etcd
   
#执行结果
2024/12/14 15:45:32 [INFO] generate received request
2024/12/14 15:45:32 [INFO] received CSR
2024/12/14 15:45:32 [INFO] generating key: rsa-2048
2024/12/14 15:45:32 [INFO] encoded CSR
2024/12/14 15:45:32 [INFO] signed certificate with serial number 537854604052826019092478615702757454761015290503

# 文件命中不带ca的是客户端证书
[root@k8s-master01 pki]# ll /etc/etcd/ssl/
total 24
-rw-r--r-- 1 root root 1005 Dec 14 15:43 etcd-ca.csr
-rw------- 1 root root 1679 Dec 14 15:43 etcd-ca-key.pem
-rw-r--r-- 1 root root 1367 Dec 14 15:43 etcd-ca.pem
-rw-r--r-- 1 root root 1005 Dec 14 15:45 etcd.csr
-rw------- 1 root root 1679 Dec 14 15:45 etcd-key.pem
-rw-r--r-- 1 root root 1509 Dec 14 15:45 etcd.pem

#复制证书到其他节点：
[root@k8s-master01 pki]# MasterNodes='k8s-master02 k8s-master03'

#执行下面命令
for NODE in $MasterNodes; do
     ssh $NODE "mkdir -p /etc/etcd/ssl"
     for FILE in etcd-ca-key.pem  etcd-ca.pem  etcd-key.pem  etcd.pem; do
       scp /etc/etcd/ssl/${FILE} $NODE:/etc/etcd/ssl/${FILE}
     done
 done

4.2生成k8s组件证书

Master01节点生成kubernetes CA证书：

1
2
3
4
5
6
7
8
9
10
11
12
#进入pki目录
[root@k8s-master01 pki]# cd /root/k8s-ha-install/pki
#执行下面命令
cfssl gencert -initca ca-csr.json | cfssljson -bare /etc/kubernetes/pki/ca

#执行结果：
2024/12/14 16:05:07 [INFO] generating a new CA key and certificate from CSR
2024/12/14 16:05:07 [INFO] generate received request
2024/12/14 16:05:07 [INFO] received CSR
2024/12/14 16:05:07 [INFO] generating key: rsa-2048
2024/12/14 16:05:07 [INFO] encoded CSR
2024/12/14 16:05:07 [INFO] signed certificate with serial number 729687791941312910444490997689935941350118427433

4.2.1生成apiserver证书

注意：10.96.0.1是k8s service的网段的第一个ip地址，如果service网段跟我的不一致，那就把10.96.0.1更改为符合你自己的service网段的第一个IP地址即可。

1
2
3
4
5
6
7
8
9
#在k8s-master01节点上执行
cfssl gencert   -ca=/etc/kubernetes/pki/ca.pem   -ca-key=/etc/kubernetes/pki/ca-key.pem   -config=ca-config.json   -hostname=10.96.0.1,192.168.0.236,127.0.0.1,kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.default.svc.cluster.local,192.168.0.81,192.168.0.82,192.168.0.83   -profile=kubernetes   apiserver-csr.json | cfssljson -bare /etc/kubernetes/pki/apiserver

#执行结果
2024/12/14 16:13:07 [INFO] generate received request
2024/12/14 16:13:07 [INFO] received CSR
2024/12/14 16:13:07 [INFO] generating key: rsa-2048
2024/12/14 16:13:07 [INFO] encoded CSR
2024/12/14 16:13:07 [INFO] signed certificate with serial number 197233707103554284528084495082138925973658376347

生成apiserver的聚合证书：

1
2
3
4
5
#在k8s-master01节点上执行
cfssl gencert   -initca front-proxy-ca-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-ca 

cfssl gencert   -ca=/etc/kubernetes/pki/front-proxy-ca.pem   -ca-key=/etc/kubernetes/pki/front-proxy-ca-key.pem   -config=ca-config.json   -profile=kubernetes   front-proxy-client-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-client

4.2.2生成controller-manage证书

k8s-master01节点操作即可：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
cfssl gencert \
   -ca=/etc/kubernetes/pki/ca.pem \
   -ca-key=/etc/kubernetes/pki/ca-key.pem \
   -config=ca-config.json \
   -profile=kubernetes \
   manager-csr.json | cfssljson -bare /etc/kubernetes/pki/controller-manager
   
# 注意，如果不是高可用集群，192.168.0.236:8443改为master01的地址，8443改为apiserver的端口，默认是6443
# set-cluster：设置一个集群项，
kubectl config set-cluster kubernetes \
     --certificate-authority=/etc/kubernetes/pki/ca.pem \
     --embed-certs=true \
     --server=https://192.168.0.236:8443 \
     --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

# 设置一个环境项，一个上下文
kubectl config set-context system:kube-controller-manager@kubernetes \
    --cluster=kubernetes \
    --user=system:kube-controller-manager \
    --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

# set-credentials 设置一个用户项
kubectl config set-credentials system:kube-controller-manager \
     --client-certificate=/etc/kubernetes/pki/controller-manager.pem \
     --client-key=/etc/kubernetes/pki/controller-manager-key.pem \
     --embed-certs=true \
     --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig


# 使用某个环境当做默认环境
kubectl config use-context system:kube-controller-manager@kubernetes \
     --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig

4.2.3生成Scheduler证书

k8s-master01节点操作即可：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
cfssl gencert \
   -ca=/etc/kubernetes/pki/ca.pem \
   -ca-key=/etc/kubernetes/pki/ca-key.pem \
   -config=ca-config.json \
   -profile=kubernetes \
   scheduler-csr.json | cfssljson -bare /etc/kubernetes/pki/scheduler

# 注意，如果不是高可用集群，192.168.0.236:8443改为master01的地址，8443改为apiserver的端口，默认是6443

kubectl config set-cluster kubernetes \
     --certificate-authority=/etc/kubernetes/pki/ca.pem \
     --embed-certs=true \
     --server=https://192.168.0.236:8443 \
     --kubeconfig=/etc/kubernetes/scheduler.kubeconfig


kubectl config set-credentials system:kube-scheduler \
     --client-certificate=/etc/kubernetes/pki/scheduler.pem \
     --client-key=/etc/kubernetes/pki/scheduler-key.pem \
     --embed-certs=true \
     --kubeconfig=/etc/kubernetes/scheduler.kubeconfig

kubectl config set-context system:kube-scheduler@kubernetes \
     --cluster=kubernetes \
     --user=system:kube-scheduler \
     --kubeconfig=/etc/kubernetes/scheduler.kubeconfig


kubectl config use-context system:kube-scheduler@kubernetes \
     --kubeconfig=/etc/kubernetes/scheduler.kubeconfig

4.2.4生成管理员证书

k8s-master01节点操作即可：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
cfssl gencert \
   -ca=/etc/kubernetes/pki/ca.pem \
   -ca-key=/etc/kubernetes/pki/ca-key.pem \
   -config=ca-config.json \
   -profile=kubernetes \
   admin-csr.json | cfssljson -bare /etc/kubernetes/pki/admin

# 注意，如果不是高可用集群，192.168.0.236:8443改为master01的地址，8443改为apiserver的端口，默认是6443

kubectl config set-cluster kubernetes     --certificate-authority=/etc/kubernetes/pki/ca.pem     --embed-certs=true     --server=https://192.168.0.236:8443     --kubeconfig=/etc/kubernetes/admin.kubeconfig
kubectl config set-credentials kubernetes-admin     --client-certificate=/etc/kubernetes/pki/admin.pem     --client-key=/etc/kubernetes/pki/admin-key.pem     --embed-certs=true     --kubeconfig=/etc/kubernetes/admin.kubeconfig


kubectl config set-context kubernetes-admin@kubernetes     --cluster=kubernetes     --user=kubernetes-admin     --kubeconfig=/etc/kubernetes/admin.kubeconfig


kubectl config use-context kubernetes-admin@kubernetes     --kubeconfig=/etc/kubernetes/admin.kubeconfig

4.2.5创建ServiceAccount证书

k8s-master01节点操作即可：

创建一对公钥，用来签发ServiceAccount的Token：

1
2
3
4
5
6
7
8
openssl genrsa -out /etc/kubernetes/pki/sa.key 2048

#如果没有返回结果检查文件是否生成了
[root@k8s-master01 pki]# ll /etc/kubernetes/pki/sa.key
-rw------- 1 root root 1704 Dec 14 19:37 /etc/kubernetes/pki/sa.key

#生成公钥
openssl rsa -in /etc/kubernetes/pki/sa.key -pubout -out /etc/kubernetes/pki/sa.pub

发送证书至其他节点：

1
2
3
4
5
6
7
8
9
for NODE in k8s-master02 k8s-master03; do 
  for FILE in $(ls /etc/kubernetes/pki | grep -v etcd); do 
    scp /etc/kubernetes/pki/${FILE} $NODE:/etc/kubernetes/pki/${FILE};
  done; 
  for FILE in admin.kubeconfig controller-manager.kubeconfig scheduler.kubeconfig; do 
    scp /etc/kubernetes/${FILE} $NODE:/etc/kubernetes/${FILE};
  done;
done

查看三台master节点的证书文件：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
#三台master节点/etc/kubernetes/pki/目录要有以下证书，一共23个
[root@k8s-master01 pki]# ll /etc/kubernetes/pki/
total 92
-rw-r--r-- 1 root root 1025 Dec 14 19:35 admin.csr
-rw------- 1 root root 1679 Dec 14 19:35 admin-key.pem
-rw-r--r-- 1 root root 1444 Dec 14 19:35 admin.pem
-rw-r--r-- 1 root root 1029 Dec 14 16:13 apiserver.csr
-rw------- 1 root root 1679 Dec 14 16:13 apiserver-key.pem
-rw-r--r-- 1 root root 1692 Dec 14 16:13 apiserver.pem
-rw-r--r-- 1 root root 1025 Dec 14 16:05 ca.csr
-rw------- 1 root root 1679 Dec 14 16:05 ca-key.pem
-rw-r--r-- 1 root root 1411 Dec 14 16:05 ca.pem
-rw-r--r-- 1 root root 1082 Dec 14 19:23 controller-manager.csr
-rw------- 1 root root 1675 Dec 14 19:23 controller-manager-key.pem
-rw-r--r-- 1 root root 1501 Dec 14 19:23 controller-manager.pem
-rw-r--r-- 1 root root  891 Dec 14 16:15 front-proxy-ca.csr
-rw------- 1 root root 1675 Dec 14 16:15 front-proxy-ca-key.pem
-rw-r--r-- 1 root root 1143 Dec 14 16:15 front-proxy-ca.pem
-rw-r--r-- 1 root root  903 Dec 14 16:15 front-proxy-client.csr
-rw------- 1 root root 1675 Dec 14 16:15 front-proxy-client-key.pem
-rw-r--r-- 1 root root 1188 Dec 14 16:15 front-proxy-client.pem
-rw------- 1 root root 1704 Dec 14 19:37 sa.key
-rw-r--r-- 1 root root  451 Dec 14 19:41 sa.pub
-rw-r--r-- 1 root root 1058 Dec 14 19:30 scheduler.csr
-rw------- 1 root root 1679 Dec 14 19:30 scheduler-key.pem
-rw-r--r-- 1 root root 1476 Dec 14 19:30 scheduler.pem
[root@k8s-master01 pki]# ls /etc/kubernetes/pki/ |wc -l
23

五、Kubernetes组件配置

5.1Etcd配置

etcd配置大致相同，注意修改每个Master节点的etcd配置的主机名和IP地址。

三台master节点的配置文件不一样，所以不要用发送键到所有会话

5.1.1k8s-master01配置

k8s-master01节点操作：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
vim /etc/etcd/etcd.config.yml
#加入以下内容
name: 'k8s-master01'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://192.168.0.81:2380'
listen-client-urls: 'https://192.168.0.81:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://192.168.0.81:2380'
advertise-client-urls: 'https://192.168.0.81:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-master01=https://192.168.0.81:2380,k8s-master02=https://192.168.0.82:2380,k8s-master03=https://192.168.0.83:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  peer-client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false

5.1.2k8s-master02配置

k8s-master02节点操作：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
vim /etc/etcd/etcd.config.yml
#加入以下内容
name: 'k8s-master02'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://192.168.0.82:2380'
listen-client-urls: 'https://192.168.0.82:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://192.168.0.82:2380'
advertise-client-urls: 'https://192.168.0.82:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-master01=https://192.168.0.81:2380,k8s-master02=https://192.168.0.82:2380,k8s-master03=https://192.168.0.83:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  peer-client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false

5.1.3k8s-master03配置

k8s-master03节点操作：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
vim /etc/etcd/etcd.config.yml
#加入以下内容
name: 'k8s-master03'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://192.168.0.83:2380'
listen-client-urls: 'https://192.168.0.83:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://192.168.0.83:2380'
advertise-client-urls: 'https://192.168.0.83:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-master01=https://192.168.0.81:2380,k8s-master02=https://192.168.0.82:2380,k8s-master03=https://192.168.0.83:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  peer-client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false

5.1.4创建Etcd的service

所有Master节点创建etcd service并启动：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
 vim /usr/lib/systemd/system/etcd.service
 #加入以下内容
[Unit]
Description=Etcd Service
Documentation=https://coreos.com/etcd/docs/latest/
After=network.target

[Service]
Type=notify
ExecStart=/usr/local/bin/etcd --config-file=/etc/etcd/etcd.config.yml
Restart=on-failure
RestartSec=10
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
Alias=etcd3.service

所有Master节点创建etcd的证书目录：

1
2
3
4
5
mkdir /etc/kubernetes/pki/etcd
ln -s /etc/etcd/ssl/* /etc/kubernetes/pki/etcd/

systemctl daemon-reload
systemctl enable --now etcd

查看etcd集群状态：

1
2
3
#任意一台master节点都能执行
export ETCDCTL_API=3
etcdctl --endpoints="192.168.0.83:2379,192.168.0.82:2379,192.168.0.81:2379" --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem  endpoint status --write-out=table

显示如下图表示etcd集群状态正常：

5.2ApiServer配置

5.2.1k8s-master01配置

注意：本文档使用的k8s service网段为10.96.0.0/16，该网段不能和宿主机的网段、Pod网段的重复，请按自己的情况修改：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
vim /usr/lib/systemd/system/kube-apiserver.service
#加入以下内容
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-apiserver \
      --v=2  \
      --allow-privileged=true  \
      --bind-address=0.0.0.0  \
      --secure-port=6443  \
      --advertise-address=192.168.0.81 \
      --service-cluster-ip-range=10.96.0.0/16  \
      --service-node-port-range=30000-32767  \
      --etcd-servers=https://192.168.0.81:2379,https://192.168.0.82:2379,https://192.168.0.83:2379 \
      --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem  \
      --etcd-certfile=/etc/etcd/ssl/etcd.pem  \
      --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem  \
      --client-ca-file=/etc/kubernetes/pki/ca.pem  \
      --tls-cert-file=/etc/kubernetes/pki/apiserver.pem  \
      --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem  \
      --kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem  \
      --kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem  \
      --service-account-key-file=/etc/kubernetes/pki/sa.pub  \
      --service-account-signing-key-file=/etc/kubernetes/pki/sa.key  \
      --service-account-issuer=https://kubernetes.default.svc.cluster.local \
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname  \
      --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota  \
      --authorization-mode=Node,RBAC  \
      --enable-bootstrap-token-auth=true  \
      --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem  \
      --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem  \
      --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem  \
      --requestheader-allowed-names=front-proxy-client  \
      --requestheader-group-headers=X-Remote-Group  \
      --requestheader-extra-headers-prefix=X-Remote-Extra-  \
      --requestheader-username-headers=X-Remote-User
      # --token-auth-file=/etc/kubernetes/token.csv

Restart=on-failure
RestartSec=10s
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

--requestheader-allowed-names=aggregator \参数有不要配置为aggregator ，如果配置为aggregator 在后续使用keda扩缩容时会遇到证书不匹配的问题，出现401报错，正确配置是--requestheader-allowed-names=front-proxy-client

5.2.2k8s-master02配置

注意：本文档使用的k8s service网段为10.96.0.0/16，该网段不能和宿主机的网段、Pod网段的重复，请按自己的情况修改：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
vim /usr/lib/systemd/system/kube-apiserver.service
#加入以下内容
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-apiserver \
      --v=2  \
      --allow-privileged=true  \
      --bind-address=0.0.0.0  \
      --secure-port=6443  \
      --advertise-address=192.168.0.82 \
      --service-cluster-ip-range=10.96.0.0/16  \
      --service-node-port-range=30000-32767  \
      --etcd-servers=https://192.168.0.81:2379,https://192.168.0.82:2379,https://192.168.0.83:2379 \
      --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem  \
      --etcd-certfile=/etc/etcd/ssl/etcd.pem  \
      --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem  \
      --client-ca-file=/etc/kubernetes/pki/ca.pem  \
      --tls-cert-file=/etc/kubernetes/pki/apiserver.pem  \
      --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem  \
      --kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem  \
      --kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem  \
      --service-account-key-file=/etc/kubernetes/pki/sa.pub  \
      --service-account-signing-key-file=/etc/kubernetes/pki/sa.key  \
      --service-account-issuer=https://kubernetes.default.svc.cluster.local \
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname  \
      --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota  \
      --authorization-mode=Node,RBAC  \
      --enable-bootstrap-token-auth=true  \
      --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem  \
      --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem  \
      --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem  \
      --requestheader-allowed-names=front-proxy-client  \
      --requestheader-group-headers=X-Remote-Group  \
      --requestheader-extra-headers-prefix=X-Remote-Extra-  \
      --requestheader-username-headers=X-Remote-User

Restart=on-failure
RestartSec=10s
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

--requestheader-allowed-names=aggregator \参数有不要配置为aggregator ，如果配置为aggregator 在后续使用keda扩缩容时会遇到证书不匹配的问题，出现401报错，正确配置是--requestheader-allowed-names=front-proxy-client

5.2.3k8s-master03配置

注意：本文档使用的k8s service网段为10.96.0.0/16，该网段不能和宿主机的网段、Pod网段的重复，请按自己的情况修改：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
vim /usr/lib/systemd/system/kube-apiserver.service
#加入以下内容
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-apiserver \
      --v=2  \
      --allow-privileged=true  \
      --bind-address=0.0.0.0  \
      --secure-port=6443  \
      --advertise-address=192.168.0.83 \
      --service-cluster-ip-range=10.96.0.0/16  \
      --service-node-port-range=30000-32767  \
      --etcd-servers=https://192.168.0.81:2379,https://192.168.0.82:2379,https://192.168.0.83:2379 \
      --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem  \
      --etcd-certfile=/etc/etcd/ssl/etcd.pem  \
      --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem  \
      --client-ca-file=/etc/kubernetes/pki/ca.pem  \
      --tls-cert-file=/etc/kubernetes/pki/apiserver.pem  \
      --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem  \
      --kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem  \
      --kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem  \
      --service-account-key-file=/etc/kubernetes/pki/sa.pub  \
      --service-account-signing-key-file=/etc/kubernetes/pki/sa.key  \
      --service-account-issuer=https://kubernetes.default.svc.cluster.local \
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname  \
      --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota  \
      --authorization-mode=Node,RBAC  \
      --enable-bootstrap-token-auth=true  \
      --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem  \
      --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem  \
      --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem  \
      --requestheader-allowed-names=front-proxy-client  \
      --requestheader-group-headers=X-Remote-Group  \
      --requestheader-extra-headers-prefix=X-Remote-Extra-  \
      --requestheader-username-headers=X-Remote-User
      # --token-auth-file=/etc/kubernetes/token.csv

Restart=on-failure
RestartSec=10s
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

--requestheader-allowed-names=aggregator \参数有不要配置为aggregator ，如果配置为aggregator 在后续使用keda扩缩容时会遇到证书不匹配的问题，出现401报错，正确配置是--requestheader-allowed-names=front-proxy-client

5.22.4启动ApiServer

所有Master节点开启kube-apiserver：

1
systemctl daemon-reload && systemctl enable --now kube-apiserver

检查kube-apiserver状态是否正常：

5.3Controller Manager配置

所有Master节点配置kube-controller-manager service（所有master节点配置一样）

注意：本文档使用的k8s Pod网段为172.16.0.0/16，该网段不能和宿主机的网段、k8s Service网段的重复，请按需修改：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
vim /usr/lib/systemd/system/kube-controller-manager.service
#加入以下内容
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-controller-manager \
      --v=2 \
      --root-ca-file=/etc/kubernetes/pki/ca.pem \
      --cluster-signing-cert-file=/etc/kubernetes/pki/ca.pem \
      --cluster-signing-key-file=/etc/kubernetes/pki/ca-key.pem \
      --service-account-private-key-file=/etc/kubernetes/pki/sa.key \
      --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig \
      --authentication-kubeconfig=/etc/kubernetes/controller-manager.kubeconfig \
      --authorization-kubeconfig=/etc/kubernetes/controller-manager.kubeconfig \
      --leader-elect=true \
      --use-service-account-credentials=true \
      --node-monitor-grace-period=40s \
      --node-monitor-period=5s \
      --controllers=*,bootstrapsigner,tokencleaner \
      --allocate-node-cidrs=true \
      --cluster-cidr=172.16.0.0/16 \
      --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \
      --node-cidr-mask-size=24

Restart=always
RestartSec=10s

[Install]
WantedBy=multi-user.target

所有Master节点启动kube-controller-manager

1
2
systemctl daemon-reload
systemctl enable --now kube-controller-manager

查看服务启动状态：

1
systemctl status kube-controller-manager

5.4Scheduler配置

所有Master节点配置kube-scheduler service（所有master节点配置一样）

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
vim /usr/lib/systemd/system/kube-scheduler.service 
#加入以下内容
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-scheduler \
      --v=2 \
      --leader-elect=true \
      --authentication-kubeconfig=/etc/kubernetes/scheduler.kubeconfig \
      --authorization-kubeconfig=/etc/kubernetes/scheduler.kubeconfig \
      --kubeconfig=/etc/kubernetes/scheduler.kubeconfig

Restart=always
RestartSec=10s

[Install]
WantedBy=multi-user.target

所有Master节点启动kube-scheduler.service 

1
2
systemctl daemon-reload
systemctl enable --now kube-scheduler.service

查看服务启动状态：

1
systemctl status kube-scheduler.service

六、TLS Bootstrapping配置

Bootstrapping自动为node节点颁发证书

6.1bootstrap配置

只需要在Master01创建Bootstrapping

1
2
3
4
5
6
7
8
9
10
cd /root/k8s-ha-install/bootstrap/

kubectl config set-cluster kubernetes     --certificate-authority=/etc/kubernetes/pki/ca.pem     --embed-certs=true     --server=https://192.168.0.236:8443     --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig

kubectl config set-credentials tls-bootstrap-token-user     --token=c8ad9c.2e4d610cf3e7426e --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig

kubectl config set-context tls-bootstrap-token-user@kubernetes     --cluster=kubernetes     --user=tls-bootstrap-token-user     --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig

kubectl config use-context tls-bootstrap-token-user@kubernetes     --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig

拷贝证书：

1
2
mkdir -p /root/.kube
cp /etc/kubernetes/admin.kubeconfig /root/.kube/config

可以正常查询集群状态，才可以继续往下，否则不行，需要排查k8s组件是否有故障（只要有结果即可，如果返回不一样不影响）

1
2
3
4
5
6
7
[root@k8s-master01 bootstrap]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE   ERROR
scheduler            Healthy   ok
controller-manager   Healthy   ok
etcd-0               Healthy   ok
#MESSAGE状态都是ok表示证书配置正确，可以往下操作，如果有错误需要检查上述证书配置

创建bootstrap：

1
2
cd /root/k8s-ha-install/bootstrap
kubectl create -f bootstrap.secret.yaml

七、Node节点配置

7.1复制证书

k8s-master01节点操作复制证书至Node节点：

1
2
3
4
5
6
7
8
9
cd /etc/kubernetes/

#进入到/etc/kubernetes/执行下面命令
for NODE in k8s-master02 k8s-master03 k8s-node01 k8s-node02 k8s-node03; do
     ssh $NODE mkdir -p /etc/kubernetes/pki
     for FILE in pki/ca.pem pki/ca-key.pem pki/front-proxy-ca.pem bootstrap-kubelet.kubeconfig; do
       scp /etc/kubernetes/$FILE $NODE:/etc/kubernetes/${FILE}
 done
 done

7.2Kubelet配置

所有节点创建Kubelet配置目录:

1
mkdir -p /var/lib/kubelet /var/log/kubernetes /etc/systemd/system/kubelet.service.d /etc/kubernetes/manifests/

所有节点配置kubelet service:（所有节点的文件配置相同）

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
vim  /usr/lib/systemd/system/kubelet.service
#添加以下内容
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes

[Service]
ExecStart=/usr/local/bin/kubelet

Restart=always
StartLimitInterval=0
RestartSec=10

[Install]
WantedBy=multi-user.target

所有节点配置kubelet service的配置文件（也可以写到kubelet.service）：

1
2
3
4
5
6
7
8
9
10
# Runtime为Containerd
vim /etc/systemd/system/kubelet.service.d/10-kubelet.conf
#添加以下内容
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig"
Environment="KUBELET_SYSTEM_ARGS=--container-runtime-endpoint=unix:///run/containerd/containerd.sock"
Environment="KUBELET_CONFIG_ARGS=--config=/etc/kubernetes/kubelet-conf.yml"
Environment="KUBELET_EXTRA_ARGS=--node-labels=node.kubernetes.io/node='' "
ExecStart=
ExecStart=/usr/local/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_SYSTEM_ARGS $KUBELET_EXTRA_ARGS

所有节点创建kubelet的配置文件

注意：如果更改了k8s的service网段，需要更改kubelet-conf.yml 的clusterDNS:配置，改成k8s Service网段的第十个地址，比如10.96.0.10

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
vim /etc/kubernetes/kubelet-conf.yml
#添加以下内容
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.pem
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
cgroupDriver: systemd
cgroupsPerQOS: true
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
containerLogMaxFiles: 5
containerLogMaxSize: 10Mi
contentType: application/vnd.kubernetes.protobuf
cpuCFSQuota: true
cpuManagerPolicy: none
cpuManagerReconcilePeriod: 10s
enableControllerAttachDetach: true
enableDebuggingHandlers: true
enforceNodeAllocatable:
- pods
eventBurst: 10
eventRecordQPS: 5
evictionHard:
  imagefs.available: 15%
  memory.available: 100Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
evictionPressureTransitionPeriod: 5m0s
failSwapOn: true
fileCheckFrequency: 20s
hairpinMode: promiscuous-bridge
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 20s
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
imageMinimumGCAge: 2m0s
iptablesDropBit: 15
iptablesMasqueradeBit: 14
kubeAPIBurst: 10
kubeAPIQPS: 5
makeIPTablesUtilChains: true
maxOpenFiles: 1000000
maxPods: 110
nodeStatusUpdateFrequency: 10s
oomScoreAdj: -999
podPidsLimit: -1
registryBurst: 10
registryPullQPS: 5
resolvConf: /etc/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 2m0s
serializeImagePulls: true
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 4h0m0s
syncFrequency: 1m0s
volumeStatsAggPeriod: 1m0s

所有节点启动kubelet：

1
2
systemctl daemon-reload
systemctl enable --now kubelet

此时系统日志/var/log/messages显示如下信息为正常，安装calico后即可恢复：

1
Dec 14 21:19:33 k8s-master01 kubelet[13986]: E1214 21:19:33.945524   13986 kubelet.go:2902] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"

k8s-master01节点执行命苦，查看集群状态(Ready或NotReady都正常)

1
2
3
4
5
6
7
8
[root@k8s-master01 kubernetes]# kubectl get nodes
NAME           STATUS     ROLES    AGE     VERSION
k8s-master01   NotReady   <none>   4m56s   v1.31.4
k8s-master02   NotReady   <none>   4m57s   v1.31.4
k8s-master03   NotReady   <none>   4m57s   v1.31.4
k8s-node01     NotReady   <none>   4m57s   v1.31.4
k8s-node02     NotReady   <none>   4m57s   v1.31.4
k8s-node03     NotReady   <none>   4m57s   v1.31.4

7.3kube-proxy配置

生成kube-proxy的证书，以下操作只在Master01执行

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
#进入pki目录
cd /root/k8s-ha-install/pki

#执行下列命令
cfssl gencert \
   -ca=/etc/kubernetes/pki/ca.pem \
   -ca-key=/etc/kubernetes/pki/ca-key.pem \
   -config=ca-config.json \
   -profile=kubernetes \
   kube-proxy-csr.json | cfssljson -bare /etc/kubernetes/pki/kube-proxy
   


kubectl config set-cluster kubernetes \
     --certificate-authority=/etc/kubernetes/pki/ca.pem \
     --embed-certs=true \
     --server=https://192.168.0.236:8443 \
     --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig
     
     
kubectl config set-credentials system:kube-proxy \
     --client-certificate=/etc/kubernetes/pki/kube-proxy.pem \
     --client-key=/etc/kubernetes/pki/kube-proxy-key.pem \
     --embed-certs=true \
     --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig
     
     
kubectl config set-context system:kube-proxy@kubernetes \
     --cluster=kubernetes \
     --user=system:kube-proxy \
     --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig
     
kubectl config use-context system:kube-proxy@kubernetes \
     --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig

将kubeconfig发送至其他节点：

1
2
3
4
5
6
7
for NODE in k8s-master02 k8s-master03; do
     scp /etc/kubernetes/kube-proxy.kubeconfig  $NODE:/etc/kubernetes/kube-proxy.kubeconfig
 done

for NODE in k8s-node01 k8s-node02 k8s-node03; do
     scp /etc/kubernetes/kube-proxy.kubeconfig $NODE:/etc/kubernetes/kube-proxy.kubeconfig
 done

所有节点添加kube-proxy的配置和service文件：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
vim /usr/lib/systemd/system/kube-proxy.service
#所有节点的配置文件相同
[Unit]
Description=Kubernetes Kube Proxy
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-proxy \
  --config=/etc/kubernetes/kube-proxy.yaml \
  --v=2

Restart=always
RestartSec=10s

[Install]
WantedBy=multi-user.target

所有节点创建kube-proxy.yaml文件，如果更改了集群Pod的网段，需要更改kube-proxy.yaml的clusterCIDR为自己的Pod网段：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
vim /etc/kubernetes/kube-proxy.yaml
#添加以下内容
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
clientConnection:
  acceptContentTypes: ""
  burst: 10
  contentType: application/vnd.kubernetes.protobuf
  kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig
  qps: 5
clusterCIDR: 172.16.0.0/16 
configSyncPeriod: 15m0s
conntrack:
  max: null
  maxPerCore: 32768
  min: 131072
  tcpCloseWaitTimeout: 1h0m0s
  tcpEstablishedTimeout: 24h0m0s
enableProfiling: false
healthzBindAddress: 0.0.0.0:10256
hostnameOverride: ""
iptables:
  masqueradeAll: false
  masqueradeBit: 14
  minSyncPeriod: 0s
  syncPeriod: 30s
ipvs:
  masqueradeAll: true
  minSyncPeriod: 5s
  scheduler: "rr"
  syncPeriod: 30s
kind: KubeProxyConfiguration
metricsBindAddress: 127.0.0.1:10249
mode: "ipvs"
nodePortAddresses: null
oomScoreAdj: -999
portRange: ""
udpIdleTimeout: 250ms

所有节点启动kube-proxy：

1
2
systemctl daemon-reload
systemctl enable --now kube-proxy

此时系统日志/var/log/messagesb报错error: cni plugin notinitialized”属于正常，安装calico后即可恢复。

1
2
#报错信息
Dec 14 21:33:08 k8s-node03 kubelet[2234]: E1214 21:33:08.894562    2234 kubelet.go:2902] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"

八、安装Calico

8.1 安装calico

以下步骤只在master01执行

1
2
3
#进入安装目录
cd /root/k8s-ha-install/calico/

更改calico的网段，主要需要将172.16.0.0/16网段，改为自己的Pod网段：

1
sed -i "s#POD_CIDR#172.16.0.0/16#g" calico.yaml

检查网段是自己的Pod网段：

1
grep "IPV4POOL_CIDR" calico.yaml -A 1

更改后如下所示：

创建calico，master01执行：

1
kubectl apply -f calico.yaml

查看pod是否正常启动：

1
2
3
4
5
6
7
8
9
10
[root@k8s-master01 calico]# kubectl get po -n kube-system
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-7d67d4b9c7-78t8t   1/1     Running   0          2m8s
calico-node-7wmww                          1/1     Running   0          2m8s
calico-node-88nrk                          1/1     Running   0          2m8s
calico-node-cmkbl                          1/1     Running   0          2m8s
calico-node-ft6c4                          1/1     Running   0          2m8s
calico-node-lxjwd                          1/1     Running   0          2m8s
calico-node-zn86s                          1/1     Running   0          2m8s
[root@k8s-master01 calico]#

如果容器状态存在异常，可以使用kubectl describe或者kubectl logs -f查看容器日志：

1
2
3
4
5
6
7
8
9
#如果pod有错误使用logs -f查看
[root@k8s-master01 ~]# kubectl logs -f calico-node-wtb5w -n kube-system

#describe
#po表示资源类型
#资源名称（这里是pod的名字）
#-n 指定命名空间 
# kube-system 命名空间的名字
kubectl describe po calico-node-22gvx -n kube-system

九、安装CoreDNS

9.1安装CoreDNS

只在k8s-master01操作：

1
2
#进入安装目录
cd /root/k8s-ha-install/

如果更改了k8s service的网段需要将coredns的serviceIP改成k8s service网段的第10个IP：

1
2
3
4
5
6
#将service的第10个ip赋值给变量COREDNS_SERVICE_IP
COREDNS_SERVICE_IP=`kubectl get svc | grep kubernetes | awk '{print $3}'`0

#查看COREDNS_SERVICE_IP变量是否是service的第10个ip
[root@k8s-master01 calico]# echo $COREDNS_SERVICE_IP
10.96.0.10

使用sed命令将coredns.yaml文件的clusterIP改为10.96.0.10：

1
sed -i "s#KUBEDNS_SERVICE_IP#${COREDNS_SERVICE_IP}#g" CoreDNS/coredns.yaml

修改前的文件配置：

修改后的文件配置：

安装CoreDNS：

1
kubectl  create -f CoreDNS/coredns.yaml

查看coredns的pod启动情况：

1
2
3
4
5
6
7
8
9
10
[root@k8s-master01 k8s-ha-install]# kubectl get pod -n kube-system
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-7d67d4b9c7-78t8t   1/1     Running   0          14m
calico-node-7wmww                          1/1     Running   0          14m
calico-node-88nrk                          1/1     Running   0          14m
calico-node-cmkbl                          1/1     Running   0          14m
calico-node-ft6c4                          1/1     Running   0          14m
calico-node-lxjwd                          1/1     Running   0          14m
calico-node-zn86s                          1/1     Running   0          14m
coredns-6fbc9bdf46-8xqvh                   1/1     Running   0          45s

如果集群过大，需要对coredns扩容可以按照下列方法操作：

1
2
3
4
5
6
7
[root@k8s-master01 k8s-ha-install]# kubectl get deploy -n kube-system
NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
calico-kube-controllers   1/1     1            1           39h
coredns                   1/1     1            1           24m

#修改coredns副本的数量，将replicas改为3（一般5-10就够了），这里不做修改
[root@k8s-master01 k8s-ha-install]# kubectl edit deploy coredns -n kube-system

十、安装metrics-server

10.1安装metrics-server

在新版的Kubernetes中系统资源的采集均使用Metrics-server，可以通过Metrics采集节点和Pod的内存、磁盘、CPU和网络的使用率。

k8s-master01操作：

1
2
cd /root/k8s-ha-install/metrics-server
kubectl  create -f .

等待metrics server启动然后查看状态：

1
2
3
4
5
6
7
8
9
10
11
12
[root@k8s-master01 metrics-server]# kubectl get po -n kube-system
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-7d67d4b9c7-78t8t   1/1     Running   0          26m
calico-node-7wmww                          1/1     Running   0          26m
calico-node-88nrk                          1/1     Running   0          26m
calico-node-cmkbl                          1/1     Running   0          26m
calico-node-ft6c4                          1/1     Running   0          26m
calico-node-lxjwd                          1/1     Running   0          26m
calico-node-zn86s                          1/1     Running   0          26m
coredns-6fbc9bdf46-8xqvh                   1/1     Running   0          12m
coredns-6fbc9bdf46-p6vb6                   1/1     Running   0          4m38s
metrics-server-6c79fcf899-7clhl            1/1     Running   0          33s

验证是否能获取到节点的指标：

1
2
3
4
5
6
7
8
9
#看到下面信息表示正常
[root@k8s-master01 metrics-server]# kubectl top node
NAME           CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
k8s-master01   84m          4%     1268Mi          16%
k8s-master02   72m          3%     1241Mi          16%
k8s-master03   74m          3%     1193Mi          15%
k8s-node01     36m          1%     588Mi           7%
k8s-node02     33m          1%     567Mi           7%
k8s-node03     35m          1%     590Mi           7%

十一、安装更好的图形化管理-krm

11.1安装krm

k8s版本要求1.13+（1.13之前版本未验证）

1
安装参考连接：https://github.com/dotbalo/krm/blob/main/deploy.md

服务部署：在安装KRM的集群中创建Namespace，并授权

注意: 下述步骤将KRM安装到了krm命名空间，如果需要更改Namespace，需要把下面步骤所有关于Namespace的地方更改为自己的Namespace，推荐不更改Namespace

1
2
3
4
5
kubectl create ns krm
kubectl create sa krm-backend -n krm
kubectl create rolebinding krm-backend --clusterrole=edit --serviceaccount=krm:krm-backend --namespace=krm
kubectl create clusterrole namespace-creater --verb=create --resource=namespaces
kubectl create clusterrolebinding krm-backend-ns-creater --clusterrole=namespace-creater --serviceaccount=krm:krm-backend --namespace=krm

部署后端服务：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
cat<<EOF | kubectl -n krm apply -f -
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: krm-backend
  name: krm-backend
spec:
  ports:
  - name: http
    port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app: krm-backend
  sessionAffinity: None
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: krm-backend
  name: krm-backend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: krm-backend
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: krm-backend
    spec:
      serviceAccountName: krm-backend
      containers:
      - env:
        - name: TZ
          value: Asia/Shanghai
        - name: LANG
          value: C.UTF-8
        - name: GIN_MODE
          value: release
        - name: LOG_LEVEL
          value: info
        - name: USERNAME
          value: 21232F297A57A5A743894A0E4A801FC3 #这里定义登录的用户名，md5加密后的值
        - name: PASSWORD
          value: 21232F297A57A5A743894A0E4A801FC3 #这里定义登录的用户名密码，md5加密后的值
        - name: "IN_CLUSTER"
          value: "true"
        image: registry.cn-beijing.aliyuncs.com/dotbalo/krm-backend:latest
        lifecycle: {}
        livenessProbe:
          failureThreshold: 2
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            port: 8080
          timeoutSeconds: 2
        name: krm-backend
        ports:
        - containerPort: 8080
          name: web
          protocol: TCP
        readinessProbe:
          failureThreshold: 2
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            port: 8080
          timeoutSeconds: 2
        resources:
          limits:
            cpu: 1
            memory: 1024Mi
          requests:
            cpu: 200m
            memory: 256Mi
      restartPolicy: Always
EOF

部署前端服务：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
cat<<EOF | kubectl -n krm apply -f -
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: krm-frontend
  name: krm-frontend
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: krm-frontend
  sessionAffinity: None
  type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: krm-frontend
  name: krm-frontend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: krm-frontend
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: krm-frontend
    spec:
      containers:
      - env:
        - name: TZ
          value: Asia/Shanghai
        - name: LANG
          value: C.UTF-8
        image: registry.cn-beijing.aliyuncs.com/dotbalo/krm-frontend:latest
        lifecycle: {}
        livenessProbe:
          failureThreshold: 2
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            port: 80
          timeoutSeconds: 2
        name: krm-backend
        ports:
        - containerPort: 80
          name: web
          protocol: TCP
        readinessProbe:
          failureThreshold: 2
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            port: 80
          timeoutSeconds: 2
        resources:
          limits:
            cpu: 1
            memory: 512Mi
          requests:
            cpu: 100m
            memory: 256Mi
      restartPolicy: Always
EOF

部署成功后，通过kubectl get svc -n krm查看krm-frontend的Service的NodePort，之后通过任意一台Kubernetes工作节点的IP:NodePort即可访问KRM

默认用户名密码：admin / admin 用户名密码请在后端的USERNAME/PASSWORD变量中更改，注意更改的值为用户名密码的大写的32位MD5值

1
2
3
4
5
6
7
8
9
[root@k8s-master01 metrics-server]# kubectl get po -n krm
NAME                            READY   STATUS    RESTARTS   AGE
krm-backend-f4f5c95f9-zdxmw     1/1     Running   0          2m19s
krm-frontend-856b4fb57b-kls8b   1/1     Running   0          48s

[root@k8s-master01 metrics-server]#  kubectl get svc  -n krm
NAME           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
krm-backend    ClusterIP   10.96.252.180   <none>        8080/TCP       2m41s
krm-frontend   NodePort    10.96.17.251    <none>        80:31740/TCP   70s

11.2krm基本使用

访问krm：

1
2
http://192.168.0.81:31740/#/login
#默认账号admin，密码也是admin

11.3krm-添加集群：

[集群资源]-[集群管理]-[添加]

将/etc/kubernetes/admin.kubeconfig文件内容按照下图添加：

点击[添加]后显示如下：

十二、集群验证

12.1验证集群可用性

①查看节点：

1
2
3
4
5
6
7
8
9
#节点状态需要都是Ready
[root@k8s-master01 ~]# kubectl get node
NAME           STATUS   ROLES    AGE   VERSION
k8s-master01   Ready    <none>   94m   v1.31.4
k8s-master02   Ready    <none>   95m   v1.31.4
k8s-master03   Ready    <none>   95m   v1.31.4
k8s-node01     Ready    <none>   95m   v1.31.4
k8s-node02     Ready    <none>   95m   v1.31.4
k8s-node03     Ready    <none>   95m   v1.31.4

②查看集群所有pod是否正常：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
#pod状态要全是Running，READY字段前后数字要一致
[root@k8s-master01 ~]# kubectl get po -A
NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE
krm           krm-backend-f4f5c95f9-zdxmw                1/1     Running   0          28m
krm           krm-frontend-856b4fb57b-kls8b              1/1     Running   0          26m
kube-system   calico-kube-controllers-7d67d4b9c7-78t8t   1/1     Running   0          68m
kube-system   calico-node-7wmww                          1/1     Running   0          68m
kube-system   calico-node-88nrk                          1/1     Running   0          68m
kube-system   calico-node-cmkbl                          1/1     Running   0          68m
kube-system   calico-node-ft6c4                          1/1     Running   0          68m
kube-system   calico-node-lxjwd                          1/1     Running   0          68m
kube-system   calico-node-zn86s                          1/1     Running   0          68m
kube-system   coredns-6fbc9bdf46-8xqvh                   1/1     Running   0          54m
kube-system   coredns-6fbc9bdf46-p6vb6                   1/1     Running   0          47m
kube-system   metrics-server-6c79fcf899-7clhl            1/1     Running   0          43m

③查看集群网段ip是否冲突

1
2
3
4
5
6
7
8
#svc网段
[root@k8s-master01 ~]# kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   144m


#pod网段
[root@k8s-master01 ~]#  kubectl get po -A -owide

④要能够正常创建资源：

1
2
3
4
5
6
7
8
#在default命名空间下创建一个deploy测试
[root@k8s-master01 dashboard]# kubectl create deploy cluster-test --image=registry.cn-beijing.aliyuncs.com/dotbalo/debug-tools -- sleep 3600
deployment.apps/cluster-test created

#查看刚才创建的pod
[root@k8s-master01 dashboard]# kubectl get po
NAME                            READY   STATUS    RESTARTS   AGE
cluster-test-66bb44bd88-7kct4   1/1     Running   0          48s

⑤Pod 必须能够解析 Service（同 namespace 和跨 namespace）

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
#进入步骤④中创建的pod
[root@k8s-master01 ~]# kubectl exec -ti cluster-test-5dbf5c5d-vt8kc -- bash
(14:58 cluster-test-5dbf5c5d-vt8kc:/)

#在pod中执行nslookup
(14:58 cluster-test-5dbf5c5d-vt8kc:/) nslookup kubernetes
Server:		10.96.0.10
Address:	10.96.0.10#53

Name:	kubernetes.default.svc.cluster.local
Address: 10.96.0.1


(14:59 cluster-test-5dbf5c5d-vt8kc:/) nslookup kube-dns.kube-system
Server:		10.96.0.10
Address:	10.96.0.10#53

Name:	kube-dns.kube-system.svc.cluster.local
Address: 10.96.0.10

⑥每个节点都必须要能访问 Kubernetes 的 kubernetes svc 443 和 kube-dns 的 service 53端口

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
#集群所有节点都要执行，这里可以使用一键所有会话
[root@k8s-node03 ~]# curl https://10.96.0.1:443 -k
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {},
  "code": 403
 
#curl 53端口返回52代码表示正常
[root@k8s-node03 ~]# curl 10.96.0.10:53
curl: (52) Empty reply from server

⑦Pod 和 Pod 之间要能够正常通讯（同 namespace 和跨 namespace）

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
#只在k8s-master01节点上执行
#获取任意不同节点的pod的ip地址，进入步骤④中创建pod，ping 172.16.135.129和 172.16.58.193 看是否可以通信
[root@k8s-master01 ~]# kubectl get po -A -o wide |grep coredns
kube-system   coredns-6fbc9bdf46-8xqvh                   1/1     Running   0          72m   172.16.135.129   k8s-node03     <none>           <none>
kube-system   coredns-6fbc9bdf46-p6vb6                   1/1     Running   0          64m   172.16.58.193    k8s-node02     <none>           <none>

#进入pod中 cluster-test-5dbf5c5d-vt8kc
[root@k8s-master01 ~]# kubectl exec -ti cluster-test-5dbf5c5d-vt8kc -- bash
(15:11 cluster-test-5dbf5c5d-vt8kc:/) ping
ping   ping6
(15:11 cluster-test-5dbf5c5d-vt8kc:/) ping 172.16.135.129 #正常ping通
PING 172.16.135.129 (172.16.135.129) 56(84) bytes of data.
64 bytes from 172.16.135.129: icmp_seq=1 ttl=62 time=0.520 ms
64 bytes from 172.16.135.129: icmp_seq=2 ttl=62 time=0.310 ms
^C
--- 172.16.135.129 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1040ms
rtt min/avg/max/mdev = 0.310/0.415/0.520/0.105 ms
(15:11 cluster-test-5dbf5c5d-vt8kc:/) ping 172.16.58.193  #正常ping通
PING 172.16.58.193 (172.16.58.193) 56(84) bytes of data.
64 bytes from 172.16.58.193: icmp_seq=1 ttl=62 time=1.70 ms
64 bytes from 172.16.58.193: icmp_seq=2 ttl=62 time=0.249 ms
^C
--- 172.16.58.193 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.249/0.977/1.706/0.729 ms
(15:12 cluster-test-5dbf5c5d-vt8kc:/)

⑧Pod 和 节点 之间要能够正常通讯：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
#所有节点测试
[root@k8s-master02 ~]# ping 172.16.135.129
PING 172.16.135.129 (172.16.135.129) 56(84) bytes of data.
64 bytes from 172.16.135.129: icmp_seq=1 ttl=63 time=0.481 ms
64 bytes from 172.16.135.129: icmp_seq=2 ttl=63 time=0.396 ms
64 bytes from 172.16.135.129: icmp_seq=3 ttl=63 time=1.24 ms
64 bytes from 172.16.135.129: icmp_seq=4 ttl=63 time=3.05 ms
64 bytes from 172.16.135.129: icmp_seq=5 ttl=63 time=0.731 ms
64 bytes from 172.16.135.129: icmp_seq=6 ttl=63 time=0.500 ms
64 bytes from 172.16.135.129: icmp_seq=7 ttl=63 time=1.51 ms
^C
--- 172.16.135.129 ping statistics ---
7 packets transmitted, 7 received, 0% packet loss, time 6118ms
rtt min/avg/max/mdev = 0.396/1.130/3.051/0.875 ms

至此k8s集群安装完成！！！