环境配置
# 1、前提条件
`OS：CentOS Stream 8`
`Kubernetes：1.27.3`
`使用kubeadm部署集群`
`高层运行时：Containerd`
`低层运行时：Kata Containers、runc`
`Master节点不能有Taint`
`SELinux需设置为disabled`
# 2、配置虚拟机
**修改服务器版本的虚拟机**
```
ls /etc/sysconfig/network-scripts/
ONBOOT=yes
IPADDR=192.168.89.134
GATEWAY=192.168.89.2
DNS1=119.29.29.29
BOOTROTO=static
```
**更改主机名**
```
hostnamectl set-hostname coco
uname -n
```
**映射主机地址**
```
vim /etc/hosts
192.168.89.134 coco
```
**修改静态ip**
```
cd /etc/sysconfig/network-scripts/
```
**关闭防火墙**
```
systemctl disable firewalld
systemctl stop firewalld
firewall-cmd --state
not running
```
**关闭SELinux**
```
sestatus
sudo vim /etc/selinux/config
disabled
```
**时间同步**
```
crontab -e
0 */1 * * * ntpdate timel.aliyun.com
```
**网桥过滤**
```
cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
vm.swappiness = 0
EOF
```
**开启 be_netfilter**
```
modprobe br_netfilter
sysctl -p /etc/sysctl.d/k8s.conf
lsmod | grep br_netfilter
#自启动
touch /etc/modules-load.d/modules.conf
vim /etc/modules-load.d/modules.conf
添加"br_netfilter"
```
**安装**
```
yum -y install ipset ipvsadm
cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack
EOF
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack
```
**关闭内存交换**
```
#临时关闭
swapoff -a
free -m
#永久关闭
vim /etc/fstab
注释掉 #/dev/mapper/cs-swap none swap defaults 0 0
重启
```
**修改代理**
```
vim .bashrc
export http_proxy="http://172.24.39.216:7890"
export https_proxy="http://172.24.39.216:7890"
source ~/.bashrc
```
**修改containerd代理**
```
vim /etc/systemd/system/containerd.service
[Service]
Environment="HTTP_PROXY=http://172.24.39.216:7890"
Environment="HTTPS_PROXY=http://172.24.39.216:7890"
systemctl daemon-reload
systemctl restart containerd
systemctl restart kubelet
```
**修改kubelet代理**
```
vim /usr/lib/systemd/system/kubelet.service
[Service]
Environment="HTTP_PROXY=http://172.24.39.216:7890"
Environment="HTTPS_PROXY=http://172.24.39.216:7890"
systemctl daemon-reload
systemctl restart kubelet
```
**更新内核**
```
rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
#yum -y install https://www.elrepo.org/elrepo-release-7.0-4.el7.elrepo.noarch.rpm
yum install https://www.elrepo.org/elrepo-release-8.el8.elrepo.noarch.rpm
yum --enablerepo="elrepo-kernel" -y install kernel-lt.x86_64
grub2-set-default 0
grub2-mkconfig -o /boot/grub2/grub.cfg
```
# 3、安装containerd
**下载并解压**
```
wget https://github.com/containerd/containerd/releases/download/v1.7.11/cri-containerd-cni-1.7.11-linux-amd64.tar.gz
tar xf cri-containerd-cni-1.7.11-linux-amd64.tar.gz -C /
```
**删除文件**
```
ls /etc/cni/net.d
mv /etc/cni/net.d/10-containerd-net.conflist /home
```
**配置containerd**
```
mkdir /etc/containerd
containerd config default > /etc/containerd/config.toml
vim /etc/containerd/config.toml
3.8 -> 3.9
```
**开机自启动**
```
systemctl enable --now containerd
containerd –version
```
# 4、kata containers安装(Linux)：需开启嵌套虚拟化 （无需安装）
**下载**
```
wget https://github.com/kata-containers/kata-containers/releases/download/3.2.0/kata-static-3.2.0-amd64.tar.xz
```
**解压**
```
tar xf kata-static-3.2.0-amd64.tar.xz
```
**移动文件**
```
sudo mv opt/kata /opt
```
**创建文件夹**
```
sudo mkdir /etc/kata-containers
```
**移动文件**
```
sudo cp /opt/kata/share/defaults/kata-containers/configuration.toml /etc/kata-containers/configuration.toml
```
**创建软链接**
```
ln -sf /opt/kata/bin/containerd-shim-kata-v2 /usr/local/bin/containerd-shim-kata-v2
ln -sf /opt/kata/bin/kata-runtime /usr/local/bin/kata-runtime
ln -sf /opt/kata/bin/kata-monitor /usr/local/bin/kata-monitor
```
**解决VMW冲突问题**
```
cat > /etc/modprobe.d/blacklist-vmware.conf << EOF
blacklist vmw_vsock_virtio_transport_common
blacklist vmw_vsock_vmci_transport
EOF
```
**重启**
```
init 6 或 sudo reboot
```
**测试是否安装成功**
```
kata-runtime check --no-network-checks
```
**启动br_netfilter**
```
modprobe br_netfilter
```
# 5、修改runc到kata containers
**修改配置文件**
```
vim /etc/containerd/config.toml
registry.k8s.io/pause:3.9
registry.aliyuncs.com/google_containers/pause:3.9
112行 runc -- kata
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata]
runtime_type = "io.containerd.kata.v2"
```
**设置默认为kata**
```
Default
Runtime_type=kata.v2
```
**重启**
```
systemctl daemon-reload
systemctl restart containerd
systemctl restart kubelet
```
# 6、kubeadm、kubelet、kubectl安装
**下载源**
```
cat > /etc/yum.repos.d/k8s.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_ gpgcheck=1
gpgkey= https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
```
**下载Kubeadm**
```
yum -y install kubeadm kubelet kubectl
yum -y install kubeadm-1.27.3-0 kubelet-1.27.3-0 kubectl-1.27.3-0
yum list kubeadm.x86_64 –showduplicates | sort -r
```
**修改配置文件**
```
vim /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS="--cgroup-driver=systemd"
```
**自启动**
```
systemctl enable kubelet
```
**卸载kubeadm**
```
sudo yum remove -y kubeadm kubectl kubelet kubernetes-cni kube*
sudo yum autoremove -y
```
# 7、部署集群
==modprobe br_netfilter==
**集群初始化**
```
kubeadm init --kubernetes-version=v1.27.3 --image-repository=registry.aliyuncs.com/google_containers --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.89.134 --cri-socket unix:///var/run/containerd/containerd.sock
```
**或**
```
kubeadm init --kubernetes-version=v1.27.3 --image-repository=registry.aliyuncs.com/google_containers --pod-network-cidr=10.0.0.0/16 --apiserver-advertise-address=192.168.89.134 --cri-socket unix:///var/run/containerd/containerd.sock
```
**创建文件**
```
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
```
**配置flannel**
```
kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
或
kubectl apply -f kube-flannel.yml
```
**修改子网**
```yaml
net-conf.json: |
{
"Network": "10.0.0.0/16",
"Backend": {
"Type": "vxlan"
}
}
```
**配置calico**
```
v3.26.4
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.26.4/manifests/tigera-operator.yaml
v3.27.2
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.27.2/manifests/tigera-operator.yaml
v3.26.4
wget https://raw.githubusercontent.com/projectcalico/calico/v3.26.4/manifests/custom-resources.yaml
v3.27.2
wget https://raw.githubusercontent.com/projectcalico/calico/v3.27.2/manifests/custom-resources.yaml
raw.githubusercontent.com 无法访问
vim /etc/hosts
加入185.199.108.133 raw.githubusercontent.com
修改子网
vim custom-resources.yaml
cidr=10.244.0.0/16
配置
kubectl create -f custom-resources.yaml
删除
kubectl delete -f custom-resources.yaml
kubectl get ns 命名空间
kubectl get pods -n calico-system
watch kubectl get pods -n calico-system
kubectl get nodes
```
**去Taints（使pod能在master节点运行）**
```
kubectl describe nodes coco |grep Taints
如果是pending状态：
Taints: node-role.kubernetes.io/control-plane:NoSchedule
kubectl taint node coco node-role.kubernetes.io/control-plane-
```
**重置集群**
```
kubeadm reset
rm -rf $HOME/.kube
rm -rf /var/lib/calico
```
# 8、执行Confidential Containers（CoCo）
**添加 label**
```
kubectl get nodes
kubectl label node "coco" "node-role.kubernetes.io/worker="
```
**安装operator controller**
```
setenforce 0
export KUBECONFIG=/etc/kubernetes/admin.conf
kubectl apply -k github.com/confidential-containers/operator/config/release?ref=v0.2.0
报错信息：没有git
sudo yum install git
git --version
```
**查看confidential-containers-system内的pod**
```
kubectl get pods -n confidential-containers-system
```
**#安装ccruntime-ssh-demo 运行时**
```
kubectl apply -f https://raw.githubusercontent.com/confidential-containers/operator/v0.2.0/config/samples/ccruntime-ssh-demo.yaml
```
**#查看confidential-containers-system内的pod**
```
kubectl get pods -n confidential-containers-system --watch
```
**第一个问题：镜像无法拉取**
```
kubectl describe pods cc-operator-controller-manager-86d9c5b45c-bvfdg -n confidential-containers-system
```
```
#修改containerd代理
vim /etc/systemd/system/containerd.service
[Service]
Environment="HTTP_PROXY=http://172.24.58.120:7890"
Environment="HTTPS_PROXY=http://172.24.58.120:7890"
systemctl daemon-reload
systemctl restart containerd
systemctl restart kubelet
#修改kubelet代理
vim /usr/lib/systemd/system/kubelet.service
[Service]
Environment="HTTP_PROXY=http://172.24.58.120:7890"
Environment="HTTPS_PROXY=http://172.24.58.120:7890"
systemctl daemon-reload
systemctl restart kubelet
```
**第二个问题：Kata Containers运行很慢**
```
测试Kata Containers：
kata-runtime check --no-network-checks
```
```
ctr image pull ccr.ccs.tencentyun.com/k8s-tutorial/test-k8s:v1
ctr run --runtime --rm -t ccr.ccs.tencentyun.com/k8s-tutorial/test-k8s:v1 test-kata
运行正常
ctr run --runtime "io.containerd.kata.v2" --rm -t ccr.ccs.tencentyun.com/k8s-tutorial/test-k8s:v1 test-kata
运行超时、死机
ctr image pull docker.io/library/busybox:latest
ctr run --runtime "io.containerd.kata.v2" --rm -t docker.io/library/busybox:latest test-kata
Linux内核版本太低，升级一下内核即可。
4.18-->5.4
```
**重启dns pod**
```
kubectl delete pod -n kube-system -l k8s-app=kube-dns
```
```
kubectl create secret docker-registry dockerhub-secret \
--docker-server=https://index.docker.io/v1/ \
--docker-username=2055331743 \
--docker-password=2002030209 \
--docker-email=2055331605@qq.com
```
**创建confidential pod**
```
vim coco-demo.yaml
```
```
kind: Service
apiVersion: v1
metadata:
name: coco-demo
spec:
selector:
app: coco-demo
ports:
- port: 22
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: coco-demo
spec:
selector:
matchLabels:
app: coco-demo
template:
metadata:
labels:
app: coco-demo
spec:
runtimeClassName: kata #改为kata-clh
containers:
- name: coco-demo
image: docker.io/katadocker/ccv0-ssh
imagePullPolicy: Always
```
**执行yaml文件**
```
kubectl apply -f coco-demo.yaml
```
**查看pod**
```
kubectl get pods
```
**第三个问题：无法拉取加密的镜像**
```
kubectl delete service coco-demo
kubectl delete deployment coco-demo
```
```
kubectl delete pod --force --grace-period=0
```
```
kubectl delete -f https://raw.githubusercontent.com/confidential-containers/operator/v0.2.0/config/samples/ccruntime-ssh-demo.yaml
find / -type d -name "keys"
kubectl exec -it cc-operator-daemon-install-9jwpg -n confidential-containers-system -- /bin/sh
Warning Failed 16s kubelet Failed to pull image "docker.io/katadocker/ccv0-ssh": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/katadocker/ccv0-ssh:latest": failed to extract layer sha256:1a058d5342cc722ad5439cacae4b2b4eedde51d8fe8800fcf28444302355c16d: Unable to get decryption keys in provided key path: lstat /etc/containerd/ocicrypt/keys/key.pem: no such file or directory
Warning Failed 2m52s (x4 over 4m28s) kubelet Failed to pull image "docker.io/katadocker/ccv0-ssh": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/katadocker/ccv0-ssh:latest": failed to extract layer sha256:1a058d5342cc722ad5439cacae4b2b4eedde51d8fe8800fcf28444302355c16d: Unable to get decryption keys in provided key path: Unknown decryption key type
```
**获取pod的ip地址**
```
echo $(kubectl get service coco-demo -o jsonpath="{.spec.clusterIP}")
```
**下载SSH公钥**
```
curl -Lo ccv0-ssh https://raw.githubusercontent.com/confidential-containers/documentation/main/demos/ssh-demo/ccv0-ssh
chmod 600 ccv0-ssh
```
**连接容器**
```
ssh -i ccv0-ssh root@10.111.147.157
```
**验证容器运行在虚拟机内：**
```
ps aux | grep /opt/confidential-containers/bin/qemu-system-x86_64
```
**验证主机访问不到容器的镜像**
```
sudo ctr -n "k8s.io" image check name==docker.io/katadocker/ccv0-ssh
```
```
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: busybox
spec:
containers:
- name: busybox
image: quay.io/prometheus/busybox:latest
command: ['tail', '-f', '/dev/null']
EOF
sudo ctr -n "k8s.io" image check name==quay.io/prometheus/busybox:latest
```
**验证容器的镜像是加密的**
```
#下载skopeo
dnf -y install skopeo-1.10.0
```
```
#查看镜像信息
skopeo inspect docker://docker.io/katadocker/ccv0-ssh:latest
```
```
#安装Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain stable
source "$HOME/.cargo/env"
```
```
#安装attestation-agent
git clone https://github.com/confidential-containers/attestation-agent
```
```
#启动attestation-agent
cd attestation-agent
git checkout v0.2.0 -b v0.2.0
```
```
sudo dnf install -y cmake gcc gcc-c++
```
```
make KBC=offline_fs_kbc
```
```
./app/target/x86_64-unknown-linux-gnu/release/attestation-agent --keyprovider_sock 127.0.0.1:50000 --getresource_sock 127.0.0.1:50001 &
```
```
cat /tmp/ocicrypt_config.json
```
```
cat /etc/aa-offline_fs_kbc-keys.json
```
```
#解密镜像
OCICRYPT_KEYPROVIDER_CONFIG=/tmp/ocicrypt_config.json skopeo copy "docker://docker.io/katadocker/ccv0-ssh:latest" "oci:ccv0-ssh_decrypted:latest" --remove-signatures --insecure-policy --decryption-key "provider:attestation-agent:offline_fs_kbc::null"
```
```
#查看解密之后的镜像
skopeo inspect "oci:ccv0-ssh_decrypted:latest"
```
# 9、解决集群不稳定
**不稳定的原因**
**pod有三个作用**
**第一个作用：在/opt目录下创建confidential-containers**
**第二个作用：在/etc/containerd/config.toml文件下写入以下内容**
```toml
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata]
cri_handler = "cc"
runtime_type = "io.containerd.kata.v2"
privileged_without_host_devices = true
pod_annotations = ["io.katacontainers.*"]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata.options]
ConfigPath = "/opt/confidential-containers/share/defaults/kata-containers/configuration.toml"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-qemu]
cri_handler = "cc"
runtime_type = "io.containerd.kata-qemu.v2"
privileged_without_host_devices = true
pod_annotations = ["io.katacontainers.*"]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-qemu.options]
ConfigPath = "/opt/confidential-containers/share/defaults/kata-containers/configuration-qemu.toml"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-qemu-tdx]
cri_handler = "cc"
runtime_type = "io.containerd.kata-qemu-tdx.v2"
privileged_without_host_devices = true
pod_annotations = ["io.katacontainers.*"]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-qemu-tdx.options]
ConfigPath = "/opt/confidential-containers/share/defaults/kata-containers/configuration-qemu-tdx.toml"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-qemu-tdx-eaa-kbc]
cri_handler = "cc"
runtime_type = "io.containerd.kata-qemu-tdx-eaa-kbc.v2"
privileged_without_host_devices = true
pod_annotations = ["io.katacontainers.*"]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-qemu-tdx-eaa-kbc.options]
ConfigPath = "/opt/confidential-containers/share/defaults/kata-containers/configuration-qemu-tdx-eaa-kbc.toml"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-qemu-sev]
cri_handler = "cc"
runtime_type = "io.containerd.kata-qemu-sev.v2"
privileged_without_host_devices = true
pod_annotations = ["io.katacontainers.*"]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-qemu-sev.options]
ConfigPath = "/opt/confidential-containers/share/defaults/kata-containers/configuration-qemu-sev.toml"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-clh]
cri_handler = "cc"
runtime_type = "io.containerd.kata-clh.v2"
privileged_without_host_devices = true
pod_annotations = ["io.katacontainers.*"]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-clh.options]
ConfigPath = "/opt/confidential-containers/share/defaults/kata-containers/configuration-clh.toml"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-clh-tdx]
cri_handler = "cc"
runtime_type = "io.containerd.kata-clh-tdx.v2"
privileged_without_host_devices = true
pod_annotations = ["io.katacontainers.*"]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-clh-tdx.options]
ConfigPath = "/opt/confidential-containers/share/defaults/kata-containers/configuration-clh-tdx.toml"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-clh-tdx-eaa-kbc]
cri_handler = "cc"
runtime_type = "io.containerd.kata-clh-tdx-eaa-kbc.v2"
privileged_without_host_devices = true
pod_annotations = ["io.katacontainers.*"]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-clh-tdx-eaa-kbc.options]
ConfigPath = "/opt/confidential-containers/share/defaults/kata-containers/configuration-clh-tdx-eaa-kbc.toml"
```
**第三个作用：创建runtimeclass**
**解决办法**
**复制文件**
```
cp -r /opt/confidential-containers /home
```
**删除ccruntime运行时**
```
kubectl delete -f https://raw.githubusercontent.com/confidential-containers/operator/v0.2.0/config/samples/ccruntime-ssh-demo.yaml
```
**删除operator**
```
kubectl delete -k github.com/confidential-containers/operator/config/release?ref=v0.2.0
```
**移动confidential-containers到/opt下**
```
mv /home/confidential-containers /opt
```
**在config.toml文件下添加配置**
```
vim /etc/containerd/config.toml
```
```toml
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata]
cri_handler = "cc"
runtime_type = "io.containerd.kata.v2"
privileged_without_host_devices = true
pod_annotations = ["io.katacontainers.*"]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata.options]
ConfigPath = "/opt/confidential-containers/share/defaults/kata-containers/configuration.toml"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-qemu]
cri_handler = "cc"
runtime_type = "io.containerd.kata-qemu.v2"
privileged_without_host_devices = true
pod_annotations = ["io.katacontainers.*"]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-qemu.options]
ConfigPath = "/opt/confidential-containers/share/defaults/kata-containers/configuration-qemu.toml"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-qemu-tdx]
cri_handler = "cc"
runtime_type = "io.containerd.kata-qemu-tdx.v2"
privileged_without_host_devices = true
pod_annotations = ["io.katacontainers.*"]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-qemu-tdx.options]
ConfigPath = "/opt/confidential-containers/share/defaults/kata-containers/configuration-qemu-tdx.toml"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-qemu-tdx-eaa-kbc]
cri_handler = "cc"
runtime_type = "io.containerd.kata-qemu-tdx-eaa-kbc.v2"
privileged_without_host_devices = true
pod_annotations = ["io.katacontainers.*"]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-qemu-tdx-eaa-kbc.options]
ConfigPath = "/opt/confidential-containers/share/defaults/kata-containers/configuration-qemu-tdx-eaa-kbc.toml"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-qemu-sev]
cri_handler = "cc"
runtime_type = "io.containerd.kata-qemu-sev.v2"
privileged_without_host_devices = true
pod_annotations = ["io.katacontainers.*"]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-qemu-sev.options]
ConfigPath = "/opt/confidential-containers/share/defaults/kata-containers/configuration-qemu-sev.toml"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-clh]
cri_handler = "cc"
runtime_type = "io.containerd.kata-clh.v2"
privileged_without_host_devices = true
pod_annotations = ["io.katacontainers.*"]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-clh.options]
ConfigPath = "/opt/confidential-containers/share/defaults/kata-containers/configuration-clh.toml"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-clh-tdx]
cri_handler = "cc"
runtime_type = "io.containerd.kata-clh-tdx.v2"
privileged_without_host_devices = true
pod_annotations = ["io.katacontainers.*"]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-clh-tdx.options]
ConfigPath = "/opt/confidential-containers/share/defaults/kata-containers/configuration-clh-tdx.toml"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-clh-tdx-eaa-kbc]
cri_handler = "cc"
runtime_type = "io.containerd.kata-clh-tdx-eaa-kbc.v2"
privileged_without_host_devices = true
pod_annotations = ["io.katacontainers.*"]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-clh-tdx-eaa-kbc.options]
ConfigPath = "/opt/confidential-containers/share/defaults/kata-containers/configuration-clh-tdx-eaa-kbc.toml"
```
**创建kata-qemu运行时**
```
vim kata-qemu.yaml
```
**写入以下内容**
```yaml
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
name: kata-qemu
handler: kata-qemu
```
**运行**
```
kubectl apply -f kata-qemu.yaml
```
**另一个问题：以下两个Pod重启之后异常**
**解决办法**
**重启dns pod**
```
kubectl delete pod -n kube-system -l k8s-app=kube-dns
```
**拉取机密容器镜像时显示Not authorized**
**可能是网络问题，第二天重试即可**
```
kubectl apply -f coco-demo.yaml
```
```
kubectl delete service coco-demo
kubectl delete deployment coco-demo
```
```
ssh -i ccv0-ssh root@$(kubectl get service coco-demo -o jsonpath="{.spec.clusterIP}")
```