k8s 部署 TCP node应用

之前我们已经成功的将node的http应用部署到k8s集群内，但某些业务不仅仅需要http应用还需要TCP长连接应用，本文将对部署tcp应用的步骤及问题进行总结
本文需要具备一定nodejs基础，涉及框架：koa
为什么采用koa而不是express或其他web框架：koa轻量级，中间件按需加载
相关技术可参考：
centos8部署k8s:《k8s部署》
k8s部署web应用：《k8s部署 nodejs web应用，使用ingress-nginx映射公网》

k8s环境
linux： centos 8.x
k8s：1.21

$ kubectl get node -owide

NAME         STATUS   ROLES                  AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE         KERNEL-VERSION                 CONTAINER-RUNTIME
k8s-master   Ready    control-plane,master   11d   v1.21.0   172.16.66.167   <none>        CentOS Linux 8   4.18.0-305.19.1.el8_4.x86_64   docker://20.10.10
k8s-node1    Ready    <none>                 11d   v1.21.0   172.16.66.168   <none>        CentOS Linux 8   4.18.0-305.19.1.el8_4.x86_64   docker://20.10.10
k8s-node2    Ready    <none>                 11d   v1.21.0   172.16.66.170   <none>        CentOS Linux 8   4.18.0-305.19.1.el8_4.x86_64   docker://20.10.10

tcp项目准备
微服务集群中，单个业务除了需要对外提供服务外，不同业务之间还存在通信
比如玩家通过tcp服务连接进入我们的游戏服，此时官方通过后台web服务给该用户发放了一封邮件，此时就需要web服务通知到tcp服，那既然tcp有对外开放的接口，为什么web服不会通过tcp通信告诉tcp业务呢？
第一，由于该端口对外开放，暴露给公网，如果有人通过该tcp端口模拟发送业务消息，会导致安全隐患，简单说，tcp不应该处理玩家操作以外的业务，比如发放奖励邮件。
第二，由于后台操作频率很低（远远达不到tcp操作的频率），此处如果改用tcp长连接会有一定的资源浪费，http这种通知后就断开的方式被广泛用于业务通知，比如现在的三方支付（微信或支付宝）的异步通知就是通过http/https协议实现
考虑到这种情况，我们需要在tcp业务中，集成http/https业务

$ vim app/app.js

const net = require("net");
const path = require("path");
const fs = require("fs");
const Koa = require('koa');
const router = require('koa-router')();
const tools = require('./tools/tools')

class App {
    constructor(net) {
        this.env = process.argv[2] || 'dev';
        this.tcpPort = 10086;                       //tcp开放端口
        this.httpPort = 30086;                      //http开放端口
        this.net = net;                             //tcp实例
        this.app = new Koa();                       //koa实例
        this.server = null;

        //初始化
        this.init();
    }

    //初始化
    async init() {
        //启动tcp server
        this.tcpRun();

        //启动koa http server
        this.koaRun();
    }

    //tcp run
    tcpRun() {
        this.server = this.net.createServer();
        this.server.on('listening', () => {
            console.log('tcp server is listening on port', this.tcpPort);
        });
        this.server.on('connection', socket => {
            console.log('tcp has a new connection', socket.remoteAddress);
            socket.on('data', data => {
                data = data.toString();
                console.log(`client:${socket.remoteAddress}`, data);
            })
            this.server.getConnections((err, count) => {
                console.log(err, count);
            })
            // socket.end();
            // this.server.close();
        });
        this.server.on('close', () => {
            console.log('tcp server is now closed');
        });
        this.server.on('error', err => {
            console.log('Error occurred:', err.message);
        });
        this.server.listen(this.tcpPort);
    }

    //koa run
    koaRun() {
        this.app.use(router.routes(), router.allowedMethods());

        //路由配置
        router.get('/web/login', async (ctx, next) => {
            ctx.body = 'ok';
        })

        this.app.listen(this.httpPort);
    }
}

new App(net);

编写dockerfile文件并构建镜像
在app同级文件夹创建Dockerfile文件

$ vim Dockerfile

#运行环境
FROM node:10.15.1
#作者
MAINTAINER [SCH]
#将同级app文件夹加入docker内
ADD app /opt/app
#指定工作目录
WORKDIR /opt/app
#执行语句
RUN npm install
#指定对外开放端口
EXPOSE 10086 30086
#启动命令
CMD ["nohup", "npm", "run", "dev", "&"]

在k8s-node1与k8s-node2节点构建镜像：

#构建镜像并命名为app v1版本
$ docker build -t app:v1 .

#查看镜像
$ docker images

REPOSITORY                                                TAG       IMAGE ID       CREATED         SIZE
app                                                       v1        7d462447d52e   4 days ago      899MB
calico/node                                               v3.20.1   355c1ee44040   8 weeks ago     156MB
calico/pod2daemon-flexvol                                 v3.20.1   55fa5eb71e09   8 weeks ago     21.7MB
calico/cni                                                v3.20.1   e69ccb66d1b6   8 weeks ago     146MB
k8s.gcr.io/ingress-nginx/controller                       v0.48.1   ac0e4fe3e6b0   4 months ago    279MB
registry.aliyuncs.com/google_containers/kube-proxy        v1.21.0   38ddd85fe90e   7 months ago    122MB
registry.aliyuncs.com/google_containers/pause             3.4.1     0f8457a4c2ec   10 months ago   683kB
coredns/coredns                                           1.8.0     296a6d5035e2   12 months ago   42.5MB
registry.aliyuncs.com/google_containers/coredns/coredns   v1.8.0    296a6d5035e2   12 months ago   42.5MB
node                                                      10.15.1   8fc2110c6978   2 years ago     897MB

可以看到app已经成功构建为docker 镜像

编写yaml文件部署app容器到k8s

$ vim app.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-deployment
spec:
  selector:
    matchLabels:
      app: app
  replicas: 6                          #启动6个app容器副本
  template:
    metadata:
      labels:
        app: app
    spec:
      containers:
      - name: app
        image: app
        ports:
        - containerPort: 10086         #对外开放端口 10086与30086
        - containerPort: 30086

---

apiVersion: v1         
kind: Service                          #使用service统一副本的入口
metadata:
  name: app-service
spec:
  selector:
    app: app
  ports:
  - protocol: TCP
    name: tcp
    port: 10086
    targetPort: 10086
  - protocol: TCP
    name: http
    port: 30086
    targetPort: 30086

$ kubectl apply -f app.yaml

$ kubectl get pod -owide

NAME                                     READY   STATUS    RESTARTS   AGE     IP               NODE        NOMINATED NODE   READINESS GATES
app-deployment-56d8495cd8-6b4jt          1/1     Running   0          3d23h   10.244.36.91     k8s-node1   <none>           <none>
app-deployment-56d8495cd8-cphqj          1/1     Running   0          3d23h   10.244.36.92     k8s-node1   <none>           <none>
app-deployment-56d8495cd8-dtpkx          1/1     Running   0          3d23h   10.244.169.144   k8s-node2   <none>           <none>
app-deployment-56d8495cd8-q59xm          1/1     Running   0          3d23h   10.244.36.90     k8s-node1   <none>           <none>
app-deployment-56d8495cd8-vphbn          1/1     Running   0          3d23h   10.244.169.145   k8s-node2   <none>           <none>
app-deployment-56d8495cd8-w4g2g          1/1     Running   1          3d23h   10.244.36.89     k8s-node1   <none>           <none>

此时已经成功部署app到两个节点内

部署ingress规则

$ vim app-ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/affinity-mode: "persistent"
    nginx.ingress.kubernetes.io/session-cookie-name: "route"
spec:
  rules:
    - host: k8s.test.com            #指定域名
      http:
        paths:
        - path: /tcp                  #一级路由        
          pathType: Prefix            #匹配规则 Prefix：前缀
          backend:
            service:
              name: app-service        #指向的service
              port:
                number: 30086          #对应的service暴露的端口

$ kubectl apply -f app-ingress.yaml

此时我们已经将http30086端口开放对外，后面如果该业务只用于接收内部请求，可以通过阿里云安全组或linux防火墙对该端口的请求IP进行管控

部署tcp-ingress tcp配置参数

$ vim tcp.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: tcp-services
  namespace: ingress-nginx
data:
  10086: "default/app-service:10086"  #端口名："ns名/service名：端口"

$ kubectl apply -f tcp.yaml

修改ingress-controller配置
在args添加- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services

$ ingress.yaml

# Source: ingress-nginx/templates/controller-deployment.yaml
apiVersion: apps/v1
#kind: Deployment
#apiVersion: extensions/v1beta1
# 修改为DaemonSet类型，随每个node节点创建和删除，配合污点容忍可以实现ingress-nginx高可用
kind: DaemonSet
metadata:
  labels:
    helm.sh/chart: ingress-nginx-3.34.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.48.1
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/instance: ingress-nginx
      app.kubernetes.io/component: controller
  revisionHistoryLimit: 10
  minReadySeconds: 0
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/component: controller
    spec:
      dnsPolicy: ClusterFirst
      #开启本机网络
      hostNetwork: true
      containers:
        - name: controller
          image: k8s.gcr.io/ingress-nginx/controller:v0.48.1
          imagePullPolicy: IfNotPresent
          lifecycle:
            preStop:
              exec:
                command:
                  - /wait-shutdown
          args:
            - /nginx-ingress-controller
            - --election-id=ingress-controller-leader
            - --ingress-class=nginx
            - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
            - --validating-webhook=:8443
            - --validating-webhook-certificate=/usr/local/certificates/cert
            - --validating-webhook-key=/usr/local/certificates/key
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services