使用k8s部署项目的流程

容器交付流程

image-20220205084826604

k8s部署项目的流程：

image-20220205085018910

案例：交付一个java应用程序

项目地址：https://github.com/lizhenliang/tomcat-java-demo.git

1. 制作镜像

1.1 编写dockerfile

使用dockerfile的多阶段构建

FROM alpine/git:v2.32.0 as cloner
MAINTAINER "[e-mail]"
WORKDIR /opt
# 拉取代码
RUN git clone https://github.com/lizhenliang/tomcat-java-demo.git

FROM maven:3.8-openjdk-8 as builder
MAINTAINER "[e-mail]"
WORKDIR /opt
# 将cloner阶段的opt/tomcat-java-demo这个文件拷贝到当前阶段的/opt/路径下
COPY --from=cloner /opt/tomcat-java-demo/ /opt/
# 添加阿里云maven镜像
RUN echo '<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 https://maven.apache.org/xsd/settings-1.0.0.xsd"><localRepository>/usr/share/maven/ref/repository</localRepository><mirrors><mirror><id>aliyunmaven</id><mirrorOf>*</mirrorOf><name>阿里云公共仓库</name><url>https://maven.aliyun.com/repository/public</url></mirror></mirrors></settings>' > /opt/settings.xml
# 执行构建
RUN mvn clean package --settings=/opt/settings.xml -Dmaven.test.skip=true

FROM tomcat:8-jdk8 as runner
MAINTAINER "[e-mail]"
WORKDIR /usr/local/tomcat/webapps/
COPY --from=builder /opt/target/*.war /usr/local/tomcat/webapps/ROOT.war

1.2 构建镜像到本地

docker build -t yangsx95/hello-girl .

测试镜像：

docker run --rm -it yangsx95/hello-girl -p 8080:8080

1.3 创建harbor项目

image-20220206160543078

·

image-20220206160828222

1.4 登录harbor

docker login 192.168.121.67

注意，如果使用HTTP访问harbor，需要将harbor加入到docker的http仓库白名单中

vim /etc/docker/daemon.json
{
    "insecure-registries": ["ip:端口"]
}

1.5 推送镜像到harbor

1. 在habor界面可以看到镜像推送的命令

   

   image-20220206162035724
2. 给镜像打tag，并推送到harbor

   docker tag yangsx95/hello-girl:latest 192.168.121.67/hello-girl/hello-girl:v1
   docker push 192.168.121.67/hello-girl/hello-girl:v1

3. 查看harbor项目

   

   image-20220206162249579

1.6 配置harbor凭据到k8s Secret中

将harbor的登录凭据保存到k8s Secret中，以供k8s服务拉取镜像使用：

kubectl create secret docker-registry harbor-admin \
  --docker-username=harbor \
  --docker-password=Harbor12345 \
  --docker-email=[e-mail] \
  --docker-server=192.168.121.67

可以通过如下方式在拉取镜像时使用：

imagePullSecrets:
- name: registry-auth

使用kubectl get secrets可以查看Secret是否被正常创建：

kubectl get secrets
NAME                                 TYPE                                  DATA   AGE
default-token-zl5r6                  kubernetes.io/service-account-token   3      5d1h
harbor-admin                         kubernetes.io/dockerconfigjson        1      6s
nfs-client-provisioner-token-sntk5   kubernetes.io/service-account-token   3      2d14h

2. 使用控制器部署镜像

2.1 部署项目依赖的Mysql数据库

注意，不建议在k8s中部署数据库：

1. Mysql 属于重量级应用（负载高、吃资源、部署复杂）

2. Mysql部署难度大，主要针对集群方式，每个集群的角色不同，需要使用StatefulSet

3. Mysql是比较稳定的服务，极少会更新，享受不到k8s的特性

4. 对服务性能会有一定的衰减，但是Mysql对性能要求很高

5. 可以在k8s中部署单节点，可以享受k8s的快速部署

---

创建如下mysql deployment资源清单文件 mysql.yml（mysql5.7版本，不支持arm64）：

# 创建Secret，为准备数据库密码配置
apiVersion: v1
kind: Secret
metadata:
  name: java-demo-db 
  namespace: default
type: Opaque
data:
  mysql-root-password: "MTIzNDU2"
  mysql-password: "MTIzNDU2"

---

# 创建Deployment部署Mysql
apiVersion: apps/v1
kind: Deployment
metadata:
  name: java-demo-db 
  namespace: default
spec:
  selector:
    matchLabels:
      project: www
      app: mysql
  template:
    metadata:
      labels:
        project: www
        app: mysql
    spec:
      containers:
      - name: db
        image: mysql:5.7.30
        resources:
          requests:
            cpu: 500m
            memory: 512Mi
          limits: 
            cpu: 500m
            memory: 512Mi
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: java-demo-db
              key: mysql-root-password
        - name: MYSQL_PASSWORD
          valueFrom:
            secretKeyRef:
              name: java-demo-db
              key: mysql-password
        - name: MYSQL_USER
          value: "yangsx"
        - name: MYSQL_DATABASE
          value: "k8s"
        ports:
        - name: mysql
          containerPort: 3306
        livenessProbe:
          exec:
            command:
            - sh
            - -c
            - "mysqladmin ping -u root -p${MYSQL_ROOT_PASSWORD}"
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          exec:
            command:
            - sh
            - -c
            - "mysqladmin ping -u root -p${MYSQL_ROOT_PASSWORD}"
          initialDelaySeconds: 5
          periodSeconds: 10
        volumeMounts:
        - name: data
          mountPath: /var/lib/mysql
        
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: java-demo-db

---

# 配置PVC，使用nfs动态分配PV
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: java-demo-db 
  namespace: default
spec:
  storageClassName: "managed-nfs-storage"
  accessModes:
    - "ReadWriteOnce"
  resources:
    requests:
      storage: "8Gi"

---

# 配置Service为集群其他pod提供服务
apiVersion: v1
kind: Service
metadata:
  name: java-demo-db
  namespace: default
spec:
  type: ClusterIP
  ports:
  - name: mysql
    port: 3306
    targetPort: mysql
  selector:
    project: www
    app: mysql

---

执行部署并查看部署结果：

$ kubectl apply -f mysql.yaml
$ kubectl get svc java-demo-db
NAME           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
java-demo-db   ClusterIP   10.103.36.119   <none>        3306/TCP   2m19s

测试mysql是否正常连接（使用mysql-client镜像）：

$ kubectl run mysql-client --rm -it --image=mysql:5.7.30 -- bash
If you don't see a command prompt, try pressing enter.
root@mysql-client:/# mysql -h 10.101.140.88 -uroot -p123456
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 99
Server version: 5.7.30 MySQL Community Server (GPL)

Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| k8s                |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
5 rows in set (0.07 sec)

2.2 导入项目sql文件

准备sql文件：https://github.com/lizhenliang/tomcat-java-demo/blob/master/db/tables_ly_tomcat.sql

将该sql文件下载并拷贝到mysql-client中：

kubectl cp tables_ly_tomcat.sql mysql-client:/

在mysql-client中使用mysql命令导入到数据库中：

mysql> create database test;
Query OK, 1 row affected (0.09 sec)

mysql> use test;
Database changed

mysql> source /tables_ly_tomcat.sql
Query OK, 1 row affected, 1 warning (0.02 sec)

Database changed
Query OK, 0 rows affected (0.12 sec)

Query OK, 0 rows affected (0.00 sec)

mysql> show tables;
+----------------+
| Tables_in_test |
+----------------+
| user           |
+----------------+
1 row in set (0.01 sec)

2.3 创建ConfigMap保存项目配置文件

创建一个ConfigMap，它拥有一个 application.yml键，用于对应spring-boot项目的根配置，其内容拷贝自 https://github.com/lizhenliang/tomcat-java-demo/blob/master/src/main/resources/application.yml ：

apiVersion: v1
kind: ConfigMap
metadata:
  name: java-demo-config
data:
    application.yml: |
        server:
          port: 8080
        spring:
          datasource:
            url: jdbc:mysql://localhost:3306/test?characterEncoding=utf-8
            username: root
            password: 123456789
            driver-class-name: com.mysql.jdbc.Driver
          freemarker:
            allow-request-override: false
            cache: true
            check-template-location: true
            charset: UTF-8
            content-type: text/html; charset=utf-8
            expose-request-attributes: false
            expose-session-attributes: false
            expose-spring-macro-helpers: false
            suffix: .ftl
            template-loader-path:
              - classpath:/templates/

2.4 创建deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: java-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      project: www
      app: java-demo
  template:
    metadata:
      labels:
        project: www
        app: java-demo
    spec:
      # 指定镜像拉取的所需要的secret
      imagePullSecrets:
      - name: registry-auth
      containers:
      - image: 192.168.121.67/microservice/java-demo:v1 
        name: java-demo
        volumeMounts:
        - name: config
          mountPath: "/usr/local/tomcat/webapps/ROOT/WEB-INF/classes/application.yml"
          subPath: application.yml
        resources:
          requests:
            cpu: 0.5
            memory: 500Mi
          limits: 
            cpu: 1
            memory: 1Gi
        livenessProbe:
          httpGet:
            path: /
            port: 8080
          initialDelaySeconds: 50
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /
            port: 8080
          initialDelaySeconds: 50
          periodSeconds: 10
      volumes:
      - name: config
        configMap:
          name: java-demo-config 
          items:
          - key: "application.yml"
            path: "application.yml"

一般必须配置的几个属性：

1. 环境变量

2. 资源配额 （重要）

3. 健康检查

4. 卷挂载点

3. 对外暴露应用

3.1 配置Service

apiVersion: v1
kind: Service
metadata:
  name: java-demo 
spec:
  selector:
    project: www
    app: java-demo
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080 

3.2 配置Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: java-demo
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: hello.girl.com  # 将域名映射到 java-web 服务
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service: 
            name: java-web  # 将所有请求发送到 java-web 服务的 8080 端口
            port: 
              number: 8080 

3.3 增加公网负载均衡

image-20220208104847193

upstream ingress-controller {
  server 192.168.121.11:80;
  server 192.168.121.12:80;
}

server {
  listen 80;
  server_name: _;
  location /{
    proxy_pass http://ingress-controller;
    proxy_set_header Host $Host; # $Host是浏览器访问网址时指定的域名
  }
}