本文主要简单记录学习在k8s集群中部署单节点 的Mysql服务（本地开发学习使用，正式环境请使用其它高可用的方案），以及Springboot项目打包构建Docker镜像（使用Google jib插件）并推送到本地Harbor私服仓库，然后在K8S集群中部署。
1.环境准备 准备搭建好的k8s集群，以及Harbor私服仓库。这里使用五台虚拟机做测试。
主机名
IP地址
角色说明
k8s-master01
192.168.1.201
K8S Master节点
k8s-node01
192.168.1.202
K8S node1节点
k8s-node02
192.168.1.203
K8S node2节点
k8s-node03
192.168.1.204
K8S node3节点
harbor
192.168.1.220
本地Harbor私服
2.部署Mysql服务 2.1 准备镜像 Mysql镜像文件比较大，可以在一台节点上下载好之后，再推送到本地Harbor仓库。先在Harbor私服上创建项目k8s
1 2 3 4 5 6 7 8 9 # 从docker hub拉取Mysql镜像 docker pull mysql:5.7.32 # 重新打tag docker tag mysql:5.7.32 192.168.1.220:5000/k8s/mysql:5.7.32 # 推送到Harbor docker push 192.168.1.220:5000/k8s/mysql:5.7.32 # 删除 docker rmi 192.168.1.220:5000/k8s/mysql:5.7.32 docker rmi mysql:5.7.32
接下来我们就直接使用
2.2 资源配置文件 2.2.1 用ConfigMap管理Mysql配置 首先，用docker run 临时启动一个Mysql服务，然后获取mysqld.cnf文件
1 2 3 4 5 6 7 8 docker run --name mysql -e MYSQL_ROOT_PASSWORD=123456 -d 192.168.1.220:5000/k8s/mysql:5.7.32 docker cp mysql:/etc/mysql/mysql.conf.d/mysqld.cnf /data # 停止容器 docker stop mysql # 删除容器 docker rm mysql # 删除镜像 docker rmi -f 192.168.1.220:5000/k8s/mysql:5.7.32
这样在然后我们可以编辑/data/mysqld.cnf文件，添加如下参数参数
1 2 3 4 5 6 7 8 vi /data/mysqld.cnf # 添加参数 # 忽略表名大小写 lower_case_table_names=1 # 指定时区 default-time-zone=+8:00 # 指定密码验证插件 default-authentication-plugin=mysql_native_password
修改完成后，生成configmap,并导出yml格式的资源文件
1 2 kubectl create configmap mysqlini --from-file=my.cnf kubectl get configmap mysqlini -o yaml > mysqlconfigmap.yml
最终的mysqlconfigmap.yml如下
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 apiVersion: v1 data: mysqld.cnf: "# Copyright (c) 2014, 2016, Oracle and/or its affiliates. All rights reserved.\n#\n# This program is free software; you can redistribute it and/or modify\n# it under the terms of the GNU General Public License, version 2.0,\n# as published by the Free Software Foundation.\n#\n# This program is also distributed with certain software (including\n# but not limited to OpenSSL) that is licensed under separate terms,\n# as designated in a particular file or component or in included license\n# documentation. The authors of MySQL hereby grant you an additional\n# permission to link the program and your derivative works with the\n# separately licensed software that they have included with MySQL.\n#\n# This program is distributed in the hope that it will be useful,\n# but WITHOUT ANY WARRANTY; without even the implied warranty of\n# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. \ See the\n# GNU General Public License, version 2.0, for more details.\n#\n# You should have received a copy of the GNU General Public License\n# along with this program; if not, write to the Free Software\n# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA\n\n#\n# The MySQL Server configuration file.\n#\n# For explanations see\n# http://dev.mysql.com/doc/mysql/en/server-system-variables.html\n\n[mysqld]\npid-file\t= /var/run/mysqld/mysqld.pid\nsocket\t\t= /var/run/mysqld/mysqld.sock\ndatadir\t\t= /var/lib/mysql\n#log-error\t= /var/log/mysql/error.log\n# By default we only accept connections from localhost\n#bind-address\t= 127.0.0.1\n# Disabling symbolic-links is recommended to prevent assorted security risks\nsymbolic-links=0\nlower_case_table_names=1\ndefault-time-zone=+8:00\ndefault-authentication-plugin=mysql_native_password\n" kind: ConfigMap metadata: name: mysqlconfigmap
2.2.2 使用Opaque Secret隐藏mysql初始密码 Opaque 类型的数据是一个 map 类型，要求 value 是 base64 编码格式。创建资源文件 mysqlsecret.yml
1 2 3 4 5 6 7 apiVersion: v1 kind: Secret metadata: name: mysqlsecret type: Opaque data: password: YWRtaW4=
2.2.3 配置PV和PVC 部署mysql之前我们需要先了解一个概念有状态服务。这是一种特殊的服务，简单的归纳下就是会产生需要持久化的数据，并且有很强的I/O需求，且重启需要依赖上次存储到磁盘的数据。如典型的mysql，kafka，zookeeper等等。在我们有比较优秀的商业存储的前提下，非常推荐使用有状态服务进行部署，计算和存储分离。在实际生产中如果没有这种存储，localPV也是不错的选择，当然local pv其实和hostPath是一样的。当然我们在开发测试环境也是可以自己搭建一套简单的如NFS服务，来进行存储和计算分离。
kubernetes中定义一种了资源类型Stateful Service即有状态服务，有状态服务需要的持久化数据动态绑定我们可以利用存储的API PersistentVolume（PV）和PersistentVolumeClaim（PVC）来进行需要的相关数据的绑定和存储。
persistentVolume是由管理员设置的存储，它是集群的一部分。就像节点时集群中的资源一样，PV也是集群中的资源。PV是Volumes之类的卷插件，但具有独立于使用PV的pod的生命周期。此API对象包含存储实现的细节，即NFS、iSCSI或者特定于云供应商的存储系统
peresistentVolumeClaim是用户存储的请求。它与pod相似，pod消耗节点资源，PVC消耗PV资源。pod可以请求特定级别的资源(CPU和内存)。盛名可以请求特定的大小和访问模式。例如：可以以读/写一次或者 只读多次模式挂载。
这里我们使用NFS来做练习，k8s集群所有节点都需要安装NFS服务。我们选用k8s的master节点（192.168.1.201）作为NFS服务的server端。
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 # yum install -y nfs-utils rpcbind # 启动 systemctl start rpcbind systemctl start nfs # 设置开启启动 systemctl enable rpcbind systemctl enable nfs # # 创建共享目录并设置权限 mkdir -p /nfs/data chmod 777 /nfs/data chown -R nfsnobody:nfsnobody /nfs/data # 修改配置 vim /etc/exports # 添加如下内容 /nfs/data *(rw,fsid=0,sync,no_wdelay,insecure_locks,no_root_squash) # 保存退出后重启master节点的NFS systemctl restart rpcbind systemctl restart nfs
PV资源文件mysqlpv.yml
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 apiVersion: v1 kind: PersistentVolume metadata: name: data-mysql-pv labels: app: mysql-pv spec: accessModes: - ReadWriteOnce capacity: storage: 10Gi mountOptions: - hard - nfsvers=4.1 nfs: path: /mysql server: 192.168 .1 .201 persistentVolumeReclaimPolicy: Retain storageClassName: standard volumeMode: Filesystem
PVC资源文件mysqlpvc.yml
1 2 3 4 5 6 7 8 9 10 11 12 13 apiVersion: v1 kind: PersistentVolumeClaim metadata: name: mysql-pvc labels: app: mysql-pvc spec: accessModes: - ReadWriteOnce storageClassName: standard resources: requests: storage: 5Gi
2.2.4 Deployment和Service资源文件 Mysql镜像使用我们之前上传到Harbor私服的192.168.1.220:5000/k8s/mysql:5.7.32
首先要使用 Kuberctl 创建 docker registry 认证的 secret,语法规则如下
语法规则： kubectl create secret docker-registry myregistrykey --dockerserver= REGISTRY_SERVER --docker-username=DOCKER_USER --dockerpassword= DOCKER_PASSWORD --docker-email=DOCKER_EMAIL
我这里使用下边这个，注意harbor的地址用户名和密码。另外记住 registrykey my-harbor
1 kubectl create secret docker-registry my-harbor --docker-server=192.168.1.220:5000 --docker-username=admin --docker-password=Harbor12345 --docker-email=myemail@qq.com
在所有k8s节点上配置harbor私服
1 2 3 4 5 6 7 vi /etc/docker/daemon.json # 添加如下内容 "insecure-registries":["192.168.1.220:5000"] # 保存后重启docker systemctl daemon-reload systemctl restart docker
Mysql资源文件mysqlservice.yml如下
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 apiVersion: apps/v1 kind: Deployment metadata: name: mysql-deploy labels: app: mysql-deploy spec: replicas: 1 template: metadata: name: mysql-deploy labels: app: mysql-deploy spec: imagePullSecrets: - name: my-harbor containers: - name: mysql-deploy image: 192.168 .1 .220 :5000/k8s/mysql:5.7.32 imagePullPolicy: IfNotPresent ports: - containerPort: 3306 env: - name: MYSQL_ROOT_PASSWORD valueFrom: secretKeyRef: key: password name: mysqlsecret - name: TZ value: Asia/Shanghai args: - "--character-set-server=utf8mb4" - "--collation-server=utf8mb4_unicode_ci" volumeMounts: - mountPath: /etc/mysql/mysql.conf.d/ name: k8s-mysql - mountPath: /var/lib/mysql name: volume-mysql restartPolicy: Always volumes: - name: k8s-mysql configMap: name: mysqlconfigmap - name: volume-mysql persistentVolumeClaim: claimName: mysql-pvc selector: matchLabels: app: mysql-deploy --- apiVersion: v1 kind: Service metadata: name: mysql-svc spec: selector: app: mysql-deploy ports: - port: 3306 targetPort: 3306 nodePort: 30036 type: NodePort
2.2.5 创建Mysql服务 将上边编写好的五个资源文件上传到k8s Master节点上。如下图，放到/data/mysql目录下。
执行如下命令:
1 2 3 4 5 6 7 8 9 [root@k8s-master01 mysql]# cd /data/mysql [root@k8s-master01 mysql]# kubectl create -f . configmap/mysqlconfigmap created persistentvolume/data-mysql-pv created persistentvolumeclaim/mysql-pvc created secret/mysqlsecret created deployment.apps/mysql-deploy created service/mysql-svc created [root@k8s-master01 mysql]#
如果没有异常可以看到，pv,pvc,configmap,secret,deployment,service资源全部创建完成。
我们可以查看pod和service查看Mysql是否正常运行
1 2 3 4 5 6 7 8 [root@k8s-master01 mysql]# kubectl get pods NAME READY STATUS RESTARTS AGE mysql-deploy-7d565f7849-fwbrr 1/1 Running 0 4m44s [root@k8s-master01 mysql]# kubectl get service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.1.0.1 <none> 443/TCP 2d11h mysql-svc NodePort 10.1.250.18 <none> 3306:30036/TCP 4m52s [root@k8s-master01 mysql]#
然后既可以通过Navicat远程连接容器内的Mysql服务了。
地址：192.168.1.201 端口：30036 用户名：root 密码：admin
3.Springboot应用镜像构建 要将一个java应用构建成Docker镜像可以使用Dockerfile，也可以使用dockerfile-maven-plugin，jib-maven-plugin等maven插件进行。这里我们介绍使用jib-maven-plugin来构建Docker镜像。
Jib是由Google出品的容器镜像构建类库--Jib, 通过Jib可以非常简单快速的为你的Java应用构建Docker 和 OCI 镜像, 无需编写Dockerfile, 以 Maven插件、Gradle插件和Java lib的形式提供。
3.1 准备基础镜像 准备openjdk11的镜像，推送到220Harbor私服上
1 2 3 4 5 docker pull openjdk:11-jre-slim docker tag openjdk:11-jre-slim 192.168.1.220:5000/k8s/openjdk:11-jre-slim docker push 192.168.1.220:5000/k8s/openjdk:11-jre-slim docker rmi 192.168.1.220:5000/k8s/openjdk:11-jre-slim docker rmi openjdk:11-jre-slim
3.2 Jib配置 首先在Springboot项目的pom文件中配置Jib插件，全部配置如下
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 <?xml version="1.0" encoding="UTF-8" ?> <project xmlns ="http://maven.apache.org/POM/4.0.0" xmlns:xsi ="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation ="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd" > <modelVersion > 4.0.0</modelVersion > <parent > <groupId > org.springframework.boot</groupId > <artifactId > spring-boot-starter-parent</artifactId > <version > 2.2.11.RELEASE</version > <relativePath /> </parent > <groupId > com.docker</groupId > <artifactId > spring-boot-k8s</artifactId > <version > 1.0</version > <name > spring-boot-k8s</name > <description > Demo project for Spring Boot</description > <properties > <java.version > 11</java.version > </properties > <dependencies > <dependency > <groupId > org.springframework.boot</groupId > <artifactId > spring-boot-starter-web</artifactId > </dependency > <dependency > <groupId > org.mybatis.spring.boot</groupId > <artifactId > mybatis-spring-boot-starter</artifactId > <version > 2.1.2</version > </dependency > <dependency > <groupId > tk.mybatis</groupId > <artifactId > mapper-spring-boot-starter</artifactId > <version > 2.1.5</version > </dependency > <dependency > <groupId > mysql</groupId > <artifactId > mysql-connector-java</artifactId > </dependency > <dependency > <groupId > org.springframework.boot</groupId > <artifactId > spring-boot-starter-test</artifactId > <scope > test</scope > </dependency > </dependencies > <build > <finalName > springboot-k8s-demo</finalName > <plugins > <plugin > <groupId > org.springframework.boot</groupId > <artifactId > spring-boot-maven-plugin</artifactId > </plugin > <plugin > <groupId > com.google.cloud.tools</groupId > <artifactId > jib-maven-plugin</artifactId > <version > 2.5.2</version > <configuration > <from > <image > 192.168.1.220:5000/k8s/openjdk:11-jre-slim</image > <auth > <username > admin</username > <password > Harbor12345</password > </auth > </from > <to > <image > 192.168.1.220:5000/k8s/${project.build.finalName}:${project.version} </image > <auth > <username > admin</username > <password > Harbor12345</password > </auth > </to > <container > <jvmFlags > <jvmFlag > -Xms512m</jvmFlag > </jvmFlags > <environment > <TZ > Asia/Shanghai</TZ > </environment > <ports > <port > 8080</port > </ports > </container > <allowInsecureRegistries > true</allowInsecureRegistries > </configuration > <executions > <execution > <phase > package</phase > <goals > <goal > build </goal > </goals > </execution > </executions > </plugin > </plugins > </build > </project >
3.3 Springboot配置文件准备 Springboot应用我们要打包成Docker镜像部署到k8s集群，那么一些参数配置就不能写死到配置文件，否则修改一些配置参数就需要重新构建镜像。我们可以把一些配置参数写到资源文件的env里边。这里我们修改数据库连接的url,用户名，密码
完整的application.yml如下
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 spring: datasource: url: ${db_url} username: ${db_username} password: ${db_password} driver-class-name: com.mysql.cj.jdbc.Driver sql-script-encoding: UTF-8 hikari: connection-timeout: 30000 idle-timeout: 600000 max-lifetime: 1800000 maximum-pool-size: 60 minimum-idle: 2 mybatis: configuration: map-underscore-to-camel-case: true type-aliases-package: com.k8s.entity mapper-locations: classpath:mapper/*.xml logging: level: com.k8s.dao: debug
3.4 构建镜像 3.2中配置已经将jib与mvn的生命周期绑定，这里我们直接执行package即可。另外，因为我们Harbor私服使用的是http,为了能正常推送镜像到私服仓库，需要构建命令需要添加参数: -DsendCredentialsOverHttp=true
1 mvn clean package -DsendCredentialsOverHttp=true
构建完成后我们可以登陆Harbor仓库查看，可以看到springboot-k8s-demo镜像已经被推送到私服。
3.5 部署到K8S集群 这里我们使用Sercet配置数据库的用户名密码,springbootsecret.yml完整配置如下
1 2 3 4 5 6 7 8 apiVersion: v1 kind: Secret metadata: name: springbootsecret type: Opaque data: password: YWRtaW4= username: cm9vdA==
Springboot应用部署到k8s完整的资源文件springbootservice.yml如下
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 apiVersion: apps/v1 kind: Deployment metadata: name: springboot-k8s-demo labels: app: springboot-k8s-demo spec: replicas: 1 template: metadata: name: springboot-k8s-demo labels: app: springboot-k8s-demo spec: containers: - name: springboot-k8s-demo image: 192.168 .1 .220 :5000/k8s/springboot-k8s-demo:1.0 imagePullPolicy: IfNotPresent ports: - containerPort: 8080 env: - name: db_username valueFrom: secretKeyRef: key: username name: springbootsecret - name: db_password valueFrom: secretKeyRef: key: password name: springbootsecret - name: db_url value: jdbc:mysql://mysql-svc:3306/k8s restartPolicy: Always selector: matchLabels: app: springboot-k8s-demo --- apiVersion: v1 kind: Service metadata: name: springboot-k8s-demo-svc spec: selector: app: springboot-k8s-demo ports: - port: 8080 nodePort: 30080 type: NodePort
将两个资源文件上传到k8s的master节点的/data/app目录下。然后执行create命令创建即可
1 2 3 4 5 6 7 8 [root@k8s-master01 app]# cd /data/app/ [root@k8s-master01 app]# ls springbootsecret.yml springbootservice.yml [root@k8s-master01 app]# kubectl create -f . secret/springbootsecret created deployment.apps/springboot-k8s-demo created service/springboot-k8s-demo-svc created [root@k8s-master01 app]#
然后可以查看pod是否正常
1 2 3 4 5 [root@k8s-master01 app]# kubectl get pods NAME READY STATUS RESTARTS AGE mysql-deploy-7d565f7849-fwbrr 1/1 Running 0 3h3m springboot-k8s-demo-68555bf745-m42fz 1/1 Running 0 62s [root@k8s-master01 app]#
如果已经是running状态，可以通过浏览器或者postman测试接口，看是不是能正常访问数据库即可。
