etcd部署及常见问题

  • A+
所属分类:日志

etcd是什么不解释,分享一篇笔记文章,内容不是原创。但是,肯定是自己部署过,学习过,填过坑。

这篇文章会解决你再部署etcd过程中遇见的常见问题,个人博客倾情推荐。

下载:


官网地址:https://github.com/coreos/etcd/releases

下载二进制文件(3master机器都需要)

wget https://github.com/etcd-io/etcd/releases/download/v3.3.10/etcd-v3.3.10-linux-amd64.tar.gz

etcd部署及常见问题

解压缩


tar xzvf etcd-v3.3.10-linux-amd64.tar.gz

 

拷贝etcd etcdctl


cd etcd-v3.3.10-linux-amd64 && mv etcd etcdctl /usr/bin/

备注:几台etcd服务器都需要执行该操作

 

创建etcd证书


etcd 证书这里,默认配置三个,后续如果需要增加,更多的 etcd 节点 这里的认证IP 请多预留几个,以备后续添加能通过认证,不需要重新签发。

 

生成json文件


mkdir /opt/ssl/

vi etcd-csr.json

{

"CN": "etcd",

"hosts": [

"127.0.0.1",

"10.5.11.30",

"10.5.11.31",

"10.5.11.32"

],

"key": {

"algo": "rsa",

"size": 2048

},

"names": [

{

"C": "CN",

"ST": "ShenZhen",

"L": "ShenZhen",

"O": "k8s",

"OU": "System"

}

]

}

 

生成 etcd 密钥


/opt/local/cfssl/cfssl gencert -ca=/opt/ssl/ca.pem \
-ca-key=/opt/ssl/ca-key.pem \
-config=/opt/ssl/config.json \
-profile=kubernetes etcd-csr.json | /opt/local/cfssl/cfssljson -bare etcd

# 查看生成

[root@master1 ssl]# ls etcd*
etcd.csr etcd-csr.json etcd-key.pem   etcd.pem

# 检查证书

# /opt/local/cfssl/cfssl-certinfo -cert etcd.pem

# 拷贝到etcd服务器

# docker-node1
cp etcd*.pem /etc/kubernetes/ssl/

# docker-node2
scp etcd*.pem docker-node2:/etc/kubernetes/ssl/

# docker-node3
scp etcd*.pem docker-node3:/etc/kubernetes/ssl/

# 如果 etcd 非 root 用户,读取证书会提示没权限

chmod 644 /etc/kubernetes/ssl/etcd-key.pem

 

修改etcd配置


由于 etcd 是最重要的组件,所以 –data-dir 请配置到其他路径中

创建 etcd data 目录, 并授权

useradd etcd

mkdir -p /opt/etcd

chown -R etcd:etcd /opt/etcd

[root@docker-node1 ~]# cat /etc/systemd/system/etcd.service

[Unit]

Description=Etcd Server

After=network.target

After=network-online.target

Wants=network-online.target

[Service]

Type=notify

WorkingDirectory=/opt/etcd/

User=root

ExecStart=/usr/bin/etcd \

--name=docker-node1 \

--cert-file=/etc/kubernetes/ssl/etcd.pem \

--key-file=/etc/kubernetes/ssl/etcd-key.pem \

--peer-cert-file=/etc/kubernetes/ssl/etcd.pem \

--peer-key-file=/etc/kubernetes/ssl/etcd-key.pem \

--trusted-ca-file=/etc/kubernetes/ssl/ca.pem \

--peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \

--initial-advertise-peer-urls=https://10.5.11.30:2380 \

--listen-peer-urls=https://10.5.11.30:2380 \

--listen-client-urls=https://10.5.11.30:2379,http://127.0.0.1:2379 \

--advertise-client-urls=https://10.5.11.30:2379 \

--initial-cluster-token=k8s-etcd-cluster \

--initial-cluster=docker-node1=https://10.5.11.30:2380,docker-node2=https://10.5.11.31:2380,docker-node3=https://10.5.11.32:2380 \

--initial-cluster-state=new \

--data-dir=/opt/etcd/

Restart=on-failure

RestartSec=5

LimitNOFILE=65536

[Install]

WantedBy=multi-user.target

[root@docker-node2 ~]# cat /etc/systemd/system/etcd.service

[Unit]

Description=Etcd Server

After=network.target

After=network-online.target

Wants=network-online.target

[Service]

Type=notify

WorkingDirectory=/opt/etcd/

User=root

ExecStart=/usr/bin/etcd \

--name=docker-node2 \

--cert-file=/etc/kubernetes/ssl/etcd.pem \

--key-file=/etc/kubernetes/ssl/etcd-key.pem \

--peer-cert-file=/etc/kubernetes/ssl/etcd.pem \

--peer-key-file=/etc/kubernetes/ssl/etcd-key.pem \

--trusted-ca-file=/etc/kubernetes/ssl/ca.pem \

--peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \

--initial-advertise-peer-urls=https://10.5.11.31:2380 \

--listen-peer-urls=https://10.5.11.31:2380 \

--listen-client-urls=https://10.5.11.31:2379,http://127.0.0.1:2379 \

--advertise-client-urls=https://10.5.11.31:2379 \

--initial-cluster-token=k8s-etcd-cluster \

--initial-cluster=docker-node1=https://10.5.11.30:2380,docker-node2=https://10.5.11.31:2380,docker-node3=https://10.5.11.32:2380 \

--initial-cluster-state=new \

--data-dir=/opt/etcd/

Restart=on-failure

RestartSec=5

LimitNOFILE=65536

[Install]

WantedBy=multi-user.target

[root@docker-node3 ~]# cat /etc/systemd/system/etcd.service

[Unit]

Description=Etcd Server

After=network.target

After=network-online.target

Wants=network-online.target

[Service]

Type=notify

WorkingDirectory=/opt/etcd/

User=root

ExecStart=/usr/bin/etcd \

--name=docker-node3 \

--cert-file=/etc/kubernetes/ssl/etcd.pem \

--key-file=/etc/kubernetes/ssl/etcd-key.pem \

--peer-cert-file=/etc/kubernetes/ssl/etcd.pem \

--peer-key-file=/etc/kubernetes/ssl/etcd-key.pem \

--trusted-ca-file=/etc/kubernetes/ssl/ca.pem \

--peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \

--initial-advertise-peer-urls=https://10.5.11.32:2380 \

--listen-peer-urls=https://10.5.11.32:2380 \

--listen-client-urls=https://10.5.11.32:2379,http://127.0.0.1:2379 \

--advertise-client-urls=https://10.5.11.32:2379 \

--initial-cluster-token=k8s-etcd-cluster \

--initial-cluster=docker-node1=https://10.5.11.30:2380,docker-node2=https://10.5.11.31:2380,docker-node3=https://10.5.11.32:2380 \

--initial-cluster-state=new \

--data-dir=/opt/etcd/

Restart=on-failure

RestartSec=5

LimitNOFILE=65536

[Install]

WantedBy=multi-user.target

      • User:指定以 k8s 账户运行;
      • WorkingDirectory、--data-dir:指定工作目录和数据目录为 /var/lib/etcd,需在启动服务前创建这个目录;
      • --name:指定节点名称,当 --initial-cluster-state 值为 new 时,--name 的参数值必须位于 --initial-cluster 列表中;
      • --cert-file、--key-file:etcd server 与 client 通信时使用的证书和私钥;
      • --trusted-ca-file:签名 client 证书的 CA 证书,用于验证 client 证书;
      • --peer-cert-file、--peer-key-file:etcd 与 peer 通信使用的证书和私钥;
      • --peer-trusted-ca-file:签名 peer 证书的 CA 证书,用于验证 peer 证书;

启动 etcd

分别启动 所有节点的 etcd 服务

systemctl daemon-reload
systemctl enable etcd
systemctl start etcd
systemctl status etcd

journalctl -u etcd -f       ##用此命令来动态查看具体日志

验证 etcd 集群状态

etcdctl --endpoints=https://10.5.11.30:2379,https://10.5.11.31:2379,https://10.5.11.32:2379\
--cert-file=/etc/kubernetes/ssl/etcd.pem \
--ca-file=/etc/kubernetes/ssl/ca.pem \
--key-file=/etc/kubernetes/ssl/etcd-key.pem \
cluster-health

状态:

member 27ee84d353820205 is healthy: got healthy result from https://10.5.11.30:2379

member 6d0ce3bab16da6f9 is healthy: got healthy result from https://10.5.11.32:2379

member f58d3add3476888c is healthy: got healthy result from https://10.5.11.31:2379

 

查看 etcd 集群成员:

etcdctl --endpoints=https://10.5.11.30:2379,https://10.5.11.31:2379,https://10.5.11.32:2379\
--cert-file=/etc/kubernetes/ssl/etcd.pem \
--ca-file=/etc/kubernetes/ssl/ca.pem \
--key-file=/etc/kubernetes/ssl/etcd-key.pem \
member list

状态:

27ee84d353820205: name=docker-node1 peerURLs=https://10.5.11.30:2380 clientURLs=https://10.5.11.30:2379 isLeader=false

6d0ce3bab16da6f9: name=docker-node3 peerURLs=https://10.5.11.32:2380 clientURLs=https://10.5.11.32:2379 isLeader=true

f58d3add3476888c: name=docker-node2 peerURLs=https://10.5.11.31:2380 clientURLs=https://10.5.11.31:2379 isLeader=false

 

备注:


1,publish error: etcdserver: request timed out

出现该问题,请同时启动所有etcd节点的服务。

2,couldn't find local name "docker-node1" in the initial cluster configuratio

出现该问题,请将/etc/systemd/system/etcd.serivce中的空格清除。

3, open /etc/kubernetes/ssl/etcd-key.pem: permiss

chmod 0644   /etc/kubernetes/ssl/etcd-key.pem

4,x509: certificate is not valid for any names, but wanted to match docker-node2

/etc/systemd/system/etcd.serivce文件中, --initial-advertise-peer-urls=https://docker-node3:2380 --advertise-client-urls=https://docker-node3:2379 ,需要设置为ip地址。否则查看健康状态会报错。

5,etcd: create snapshot directory error: mkdir /opt/etcd/member/snap: permission denied

出现该问题解决方法是rm -rf /opt/etcd/*

6,remote error: tls: bad certificate", ServerName ""

证书不匹配,检查vi etcd-csr.json 中Ip地址设置。

 

 

 

 

weinxin
付生保个人博客
一个运维工程师的自媒体个人博客网站。也是关注IT技术学习和经验分享的原创独立自媒体个人博客。
avatar

发表评论

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen: