使用 kind 进行 e2e 测试

Kind 简介

• Kind 是 Kubernetes In Docker 的缩写，顾名思义是使用 Docker 容器作为 Node 并将 Kubernetes 部署至其中的一个工具。
• Kind 可以作为一种本地集群搭建的工具进
• 由于便于快速搭建环境，经常被用来做 k8s 相关测试准备

流程

1. 构建环境
2. 测试

手动搭建单节点环境测试

• 关闭 kind 自带的 kindnet 网络插件,并将其设置为 Calico 的默认子网，编写一个 kind-calico.yaml 文件

  kind: Cluster
  apiVersion: kind.sigs.k8s.io/v1alpha3
  networking:
  disableDefaultCNI: true # disable kindnet
  podSubnet: 192.168.0.0/16 # set to Calico's default subnet
  

• 使用该文件创建 不带网络插件的集群

  [root@k8s-master01 kind]# kind create cluster --config kind-calico.yaml
  Creating cluster "kind" ...
  ✓ Ensuring node image (kindest/node:v1.16.3) 🖼
  ✓ Preparing nodes 📦
  ✓ Writing configuration 📜
  ✓ Starting control-plane 🕹️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️️ 
  ✓ Installing StorageClass 💾
  Set kubectl context to "kind-kind"
  You can now use your cluster with:
  
  kubectl cluster-info --context kind-kind
  
  Thanks for using kind! 😊
  
  

• 导入 calico 镜像 并部署

  # 将calico 镜像打包
  docker save calico/kube-controllers:v3.10.1 calico/node:v3.10.1 calico/pod2daemon-flexvol:v3.10.1  calico/cni:v3.10.1 -o calico.tar
  
  # 导入 calico 相关镜像
  kind load image-archive calico.tar
  
  # 部署 calico 网络
  wget https://docs.projectcalico.org/v3.10/manifests/calico.yaml
  kubectl create -f calico.yaml
  

看到 calico 相关的所有 pod 正常启动运行

NAMESPACE        NAME                                         READY   STATUS             RESTARTS   AGE   IP             NODE                 NOMINATED NODE   READINESS GATES
kube-system      calico-kube-controllers-6b64bcd855-xmvfb     1/1     Running            0          22m   192.168.82.1   kind-control-plane   <none>           <none>
kube-system      calico-node-5lk9m                            1/1     Running            0          22m   172.17.0.2     kind-control-plane   <none>           <none>
kube-system      coredns-5644d7b6d9-6l2w5                     1/1     Running            0          28m   192.168.82.2   kind-control-plane   <none>           <none>
kube-system      coredns-5644d7b6d9-rzqfn                     1/1     Running            0          28m   192.168.82.3   kind-control-plane   <none>           <none>
kube-system      etcd-kind-control-plane                      1/1     Running            0          27m   172.17.0.2     kind-control-plane   <none>           <none>
kube-system      kube-apiserver-kind-control-plane            1/1     Running            0          27m   172.17.0.2     kind-control-plane   <none>           <none>
kube-system      kube-controller-manager-kind-control-plane   1/1     Running            0          27m   172.17.0.2     kind-control-plane   <none>           <none>
kube-system      kube-proxy-srbjr                             1/1     Running            0          28m   172.17.0.2     kind-control-plane   <none>           <none>
kube-system      kube-scheduler-kind-control-plane            1/1     Running            0          27m   172.17.0.2     kind-control-plane   <none>           <none>

• 导入 e2e 测试所需镜像

  # server 镜像
  kind load docker-image beyond.io:5000/kubernetes-e2e-test-images/agnhost:2.6
  
  # client 镜像
  kind load docker-image beyond.io:5000/busybox:1.29
  

• 使用准备好的 e2e 二进制进行测试

  ./e2e.test --ginkgo.focus="NetworkPolicy"
  

手动搭建多节点环境测试

• 多节点配置 muilt-nodes.yaml

  kind: Cluster
  apiVersion: kind.sigs.k8s.io/v1alpha3
  nodes:
  - role: control-plane
  - role: worker
  - role: worker
  networking:
  disableDefaultCNI: true
  serviceSubnet: "10.96.0.1/12"
  podSubnet: "192.168.0.0/16"
  

后续安装 calico 及 测试和单节点一致

构建脚本测试

脚本 模板

#!/usr/bin/env bash

NAME="test-cluster"

echo "+++ Cleaning up old test artifacts"
kind delete cluster --name "${NAME}" > /dev/null 2>&1 || true
rm sample-controller > /dev/null 2>&1 || true

echo "+++ Creating test cluster"
kind create cluster --name "${NAME}" --config config.yaml
export KUBECONFIG="$(kind get kubeconfig-path --name="{$NAME}")"

echo "+++ Building sample-controller"
CGO_ENABLED=0 GOARCH=amd64 GOOS=linux go build -o sample-controller

echo "+++ Building Docker image"
docker build -t sample-controller:test .

echo "+++ Loading image into kind container"
kind load docker-image --name "{$NAME}" sample-controller:test

echo "+++ Deploying sample-controller"
kubectl apply -f sample-manifest.yaml

echo "+++ Deploying example Foo resource"
kubectl apply -f example-foo.yaml

高可用配置

kind: Cluster
apiVersion: kind.sigs.k8s.io/v1alpha3
kubeadmConfigPatches:
- |
  apiVersion: kubeadm.k8s.io/v1beta1
  kind: ClusterConfiguration
  metadata:
    name: config
  networking:
    serviceSubnet: 10.0.0.0/16
  imageRepository: registry.aliyuncs.com/google_containers
  nodeRegistration:
    kubeletExtraArgs:
      pod-infra-container-image: registry.aliyuncs.com/google_containers/pause:3.1
- |
  apiVersion: kubeadm.k8s.io/v1beta1
  kind: InitConfiguration
  metadata:
    name: config
  networking:
    serviceSubnet: 10.0.0.0/16
  imageRepository: registry.aliyuncs.com/google_containers
nodes:
- role: control-plane
- role: control-plane
- role: control-plane
- role: worker
- role: worker
- role: worker