[大数据] K8S1.19.16部署MongoDB复制集集群并设置安全验证

开发: C++知识库 Java知识库 JavaScript Python PHP知识库人工智能区块链大数据移动开发嵌入式开发工具数据结构与算法开发测试游戏开发网络协议系统运维
教程: HTML教程 CSS教程 JavaScript教程 Go语言教程 JQuery教程 VUE教程 VUE3教程 Bootstrap教程 SQL数据库教程 C语言教程 C++教程 Java教程 Python教程 Python3教程 C#教程
数码: 电脑笔记本显卡显示器固态硬盘硬盘耳机手机 iphone vivo oppo 小米华为单反装机图拉丁

-> 大数据 -> K8S1.19.16部署MongoDB复制集集群并设置安全验证 -> 正文阅读

[大数据]K8S1.19.16部署MongoDB复制集集群并设置安全验证

前提
三台k8s集群环境机器
安装了glusterfs文件系统（主要用于做持久化，省略的话需要调整一下持久化方式）
安装了docker

1.生成认证文件

openssl rand -base64 768 > mongodb.key

2.重新构建镜像
这里主要是加入认证文件和设置目录权限，为了省事直接设置了777，可以设置600

#镜像打包文件，置入认证文件、设置文件夹权限
cat >> Dockerfile << EOF
FROM mongo:4.4.4
RUN mkdir -p /data/logs
RUN mkdir -p /data/key
COPY ./mongodb.key /data/key/mongodb.key
RUN chmod 400 /data/key/mongodb.key
RUN chmod 777 /data/logs/
RUN chown -R mongodb:mongodb /data/
EOF

docker build -f Dockerfile -t 127.0.0.1:8089/kubernetes/mongo:4.4.4 .

3.部署服务
①创建持久化端点（可以省略或者使用其他任意持久化方式）

apiVersion: v1
kind: Endpoints
metadata:
  name: glusterfs-cluster
  namespace: default
subsets:
- addresses:
  - ip: 172.27.12.250
  - ip: 172.27.12.249
  - ip: 172.27.12.248
  ports:
  - port: 49152
    protocol: TCP

②创建MongoDB配置文件
注意配置目录与上面构建镜像设置的目录一致

apiVersion: v1
data:
  mongod.conf: |-
    dbpath=/data/db
    logpath=/data/logs/mongodb.log
    journal=true
    #pidfilepath=/data/run/mongodb.pid
    directoryperdb=true
    logappend=true
    bind_ip=0.0.0.0
    port=27017
    auth=true
    keyFile=/data/key/mongodb.key
kind: ConfigMap
metadata:
  name: mongodb-config
  namespace: default

③创建pv和pvc（可以根据自己的环境调整pv使用的端点）

apiVersion: v1
kind: PersistentVolume
metadata:
  name: system-mongodb-replica-pv-1
  labels:
    pv: system-mongodb-replica-pv-1
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteMany
  glusterfs:
    endpoints: glusterfs-cluster
    path: system-mongodb-replica-1
    readOnly: false

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mongodb-replica-pvc-1
  namespace: default
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
  selector:
    matchLabels:
      pv: mongodb-replica-pv-1

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: mongodb-replica-pv-2
  labels:
    pv: mongodb-replica-pv-2
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteMany
  glusterfs:
    endpoints: glusterfs-cluster
    path: mongodb-replica-2
    readOnly: false

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mongodb-replica-pvc-2
  namespace: default
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
  selector:
    matchLabels:
      pv: mongodb-replica-pv-2

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mongodb-replica-pvc-3
  namespace: default
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
  selector:
    matchLabels:
      pv: mongodb-replica-pv-3

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mongodb-replica-pvc-3
  namespace: default
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
  selector:
    matchLabels:
      pv: mongodb-replica-pv-3
apiVersion: v1
kind: PersistentVolume
metadata:
  name: mongodb-replica-pv-3
  labels:
    pv: mongodb-replica-pv-3
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteMany
  glusterfs:
    endpoints: glusterfs-cluster
    path: mongodb-replica-3
    readOnly: false

④部署有状态应用和无头服务
这里注意使用的是前面构建好的镜像
注意踩坑点：挂载目录位置、开启ipv6（似乎不开启也没事）、无头服务（后面会提到）

apiVersion: apps/v1
kind: StatefulSet
metadata: 
  namespace: default
  name: mongodb-1
spec: 
  selector: 
    matchLabels: 
      name: mongodb-1
  serviceName: "mongodb-1"
  replicas: 1
  podManagementPolicy: Parallel
  template: 
    metadata: 
      labels: 
        name: mongodb-1
        app: mongodb-cluster-1
    spec: 
      terminationGracePeriodSeconds: 10
      containers: 
      - name: mongodb
        image: 127.0.0.1:8090/kubernetes/mongo:4.4.4
        imagePullPolicy: Always
        command:  
        - mongod 
        - "-f"
        - "/etc/mongod/mongod.conf"
        - "--ipv6"
        - "--replSet"
        - MongoDBCluster
        ports: 
        - containerPort: 27017
        volumeMounts: 
        - name: mongodb-config
          mountPath: /etc/mongod/
        - name: date-config
          mountPath: /etc/localtime
        - name: volume01
          mountPath: /data/db/
      volumes:
        - name: mongodb-config
          configMap:
            name: mongodb-config
        - name: date-config
          hostPath:
            path: /etc/localtime
        - name: volume01
          persistentVolumeClaim:
            claimName: mongodb-replica-pvc-1
---
apiVersion: v1
kind: Service
metadata:
  namespace: default
  name: mongodb-1
  labels:
    name: mongodb
spec:
  ports:
  - name: mongodb-port
    port: 27017
  clusterIP: None
  selector:
    name: mongodb-1

---
apiVersion: apps/v1
kind: StatefulSet
metadata: 
  namespace: default
  name: mongodb-2
spec: 
  selector: 
    matchLabels: 
      name: mongodb-2
  serviceName: "mongodb-2"
  replicas: 1
  podManagementPolicy: Parallel
  template: 
    metadata: 
      labels: 
        name: mongodb-2
        app: mongodb-cluster-2
    spec: 
      terminationGracePeriodSeconds: 10
      containers: 
      - name: mongodb
        image: 127.0.0.1:8090/kubernetes/mongo:4.4.4
        imagePullPolicy: Always
        command:  
        - mongod 
        - "-f"
        - "/etc/mongod/mongod.conf"
        - "--ipv6"
        - "--replSet"
        - MongoDBCluster
        ports: 
        - containerPort: 27017
        volumeMounts: 
        - name: mongodb-config
          mountPath: /etc/mongod/
        - name: date-config
          mountPath: /etc/localtime
        - name: volume01
          mountPath: /data/db/
      volumes:
        - name: mongodb-config
          configMap:
            name: mongodb-config
        - name: date-config
          hostPath:
            path: /etc/localtime
        - name: volume01
          persistentVolumeClaim:
            claimName: mongodb-replica-pvc-2
---
apiVersion: v1
kind: Service
metadata:
  namespace: default
  name: mongodb-2
  labels:
    name: mongodb
spec:
  ports:
  - name: mongodb-port
    port: 27017
  clusterIP: None
  selector:
    name: mongodb-2

---
apiVersion: apps/v1
kind: StatefulSet
metadata: 
  namespace: default
  name: mongodb-3
spec: 
  selector: 
    matchLabels: 
      name: mongodb-3
  serviceName: "mongodb-3"
  replicas: 1
  podManagementPolicy: Parallel
  template: 
    metadata: 
      labels: 
        name: mongodb-3
        app: mongodb-cluster-3
    spec: 
      terminationGracePeriodSeconds: 10
      containers: 
      - name: mongodb
        image: 127.0.0.1:8090/kubernetes/mongo:4.4.4
        imagePullPolicy: Always
        command:  
        - mongod 
        - "-f"
        - "/etc/mongod/mongod.conf"
        - "--ipv6"
        - "--replSet"
        - MongoDBCluster
        ports: 
        - containerPort: 27017
        volumeMounts: 
        - name: mongodb-config
          mountPath: /etc/mongod/
        - name: date-config
          mountPath: /etc/localtime
        - name: volume01
          mountPath: /data/db/
      volumes:
        - name: mongodb-config
          configMap:
            name: mongodb-config
        - name: date-config
          hostPath:
            path: /etc/localtime
        - name: volume01
          persistentVolumeClaim:
            claimName: mongodb-replica-pvc-3
---
apiVersion: v1
kind: Service
metadata:
  namespace: default
  name: mongodb-3
  labels:
    name: mongodb
spec:
  ports:
  - name: mongodb-port
    port: 27017
  clusterIP: None
  selector:
    name: mongodb-3

⑤测试运行
部署busybox测试无头服务网络，这里使用1.28.3的版本，注意有些版本有bug

apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  containers:
  - name: busybox
    image: busybox:1.28.3
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
  restartPolicy: Always

进入到busybox容器中

kubectl exec -it busybox sh

#输入命令测试无头服务解析地址情况
nslookup mongodb-1

输出结果：
结果格式类似于下面，其中Name表示解析到的服务域名，Address表示绑定到的POD和其DNS主机地址

/ # nslookup mongodb-1
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      mongodb-1
Address 1: 10.244.1.224 mongodb-1-0.mongodb-1.provide-service.svc.cluster.local

这里主要是用于负载均衡，详情可以自行百度无头服务

⑥创建外部访问服务

apiVersion: v1
kind: Service
metadata:
  name: mongodb-cluster-1
  namespace: default
  labels:
    name: mongodb-cluster-1
spec:
  selector:
    app: mongodb-cluster-1
  type: NodePort
  externalTrafficPolicy: Cluster
  ports:
    - name: mongodb-cluster
      nodePort: 30881
      port: 27017
      protocol: TCP
      targetPort: 27017
---
apiVersion: v1
kind: Service
metadata:
  name: mongodb-cluster-2
  namespace: default
  labels:
    name: mongodb-cluster-2
spec:
  selector:
    app: mongodb-cluster-2
  type: NodePort
  externalTrafficPolicy: Cluster
  ports:
    - name: mongodb-cluster
      nodePort: 30882
      port: 27017
      protocol: TCP
      targetPort: 27017
---
apiVersion: v1
kind: Service
metadata:
  name: mongodb-cluster-3
  namespace: default
  labels:
    name: mongodb-cluster-3
spec:
  selector:
    app: mongodb-cluster-3
  type: NodePort
  externalTrafficPolicy: Cluster
  ports:
    - name: mongodb-cluster
      nodePort: 30883
      port: 27017
      protocol: TCP
      targetPort: 27017

⑦初始化集群

#进入mongodb-1容器
kubectl exec -it mongodb-1-0 bash

#连接数据库
mongo

连接到数据库后

#初始化集群，arbiterOnly表示仲裁节点
config = {_id: 'MongoDBCluster', members: [{_id: 0, host: '172.27.12.250:30881'},{_id: 1, host: '172.27.12.250:30882'},{_id: 2, host: '172.27.12.250:30883','arbiterOnly':true}] }
rs.initiate(config)
#查看节点状态，会展示节点类型和地址等信息
rs.status()

#过几秒后本节点出现 MongoDBCluster:PRIMARY> 则表示本节点为主节点，在主节点上创建初始账号
use admin
db.createUser({user: "root",pwd: "123456",roles: [ { role: "root",db: "admin" } ]})
#测试登录
db.auth('root','123456')

注意：这里有个大坑
我在一开始的时候初始化集群使用的是无头服务域名地址，即

config = {_id: 'MongoDBCluster', members: [{_id: 0, host: 'mongodb-1:27017'},{_id: 1, host: 'mongodb-2:27017'},{_id: 2, host: 'mongodb-3:27017','arbiterOnly':true}] }

然后使用java或者Navicat数据库连接工具进行连接，连接使用的地址是外部访问地址，即172.27.12.250:30881,172.27.12.250:30882,172.27.12.250:30883这些。
结果单独使用主节点或者副节点进行连接是没问题的，但是使用集群方式进行访问就一直提无法解析服务地址。
Java连接提示：
Timed out after 30000 ms while waiting for a server that matches com.mongodb.client.internal.MongoClientDelegate$1@73ffe7b0. Client view of cluster state is {type=REPLICA_SET, servers=[{address=mongodb-3:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketException: mongodb-3}, caused by {java.net.UnknownHostException: mongodb-3}}, {address=mongodb-1:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketException: mongodb-1}, caused by {java.net.UnknownHostException: mongodb-1}}, {address=mongodb-2:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketException: mongodb-2}, caused by {java.net.UnknownHostException: mongodb-2}}]

Navicat连接显示：
no suitable servers found: serverSelectionTimeoutMS expired: [Failed to resolve ‘mongodb-1’] [Failed to resolve ‘mongodb-2’] [Failed to resolve ‘mongodb-3’]

查找资料，很多说是主机开启了ipv6但是mongodb没有开启，然后我开启ipv6再次尝试结果还是一样。然后找到一个答案说是mongodb集群的连接是由客户端指定的，java连接集群时需要调整连接串，我尝试了一下也是没用，这里我没有进行太多的尝试，有兴趣的可以自己百度看看，也许是我设置的不对。最后看到一个答案说是DNS解析失败，需要对主机的host进行调整，对照提示的错误信息，这个应该是最符合我这个情况的，经过诸多尝试最后还是没有解决使用无头服务初始化集群这个问题，无奈，只有将集群初始化地址改为外部可以访问的到的一个地址。
最后，java和navicat连接成功。

参考资料：