构建高可用Kubernetes集群：GCP与AWS实战指南

AI 协助说明： 本页由 Codex 协助核验和修复站内链接；正文的技术事实和适用范围未因本次修改而改变，仍以文中来源及当前官方文档为准。

适用范围与更新提示： 通用的 kubeadm、PKI 与节点 join 步骤以合并后的主文为准。本文仅作为 GCP/AWS 基础设施实现参考；云厂商命令、API 和产品行为会变化，发布或执行前应按官方当前文档复核。

1. 高可用Kubernetes架构概览#

在开始搭建前，明确定义高可用Kubernetes集群的核心架构：

graph TD
    subgraph "区域 1 (us-central1-a)"
        CP1[控制平面节点 1] --> ETCD1[etcd 1]
    end
    
    subgraph "区域 2 (us-central1-b)"
        CP2[控制平面节点 2] --> ETCD2[etcd 2]
    end
    
    subgraph "区域 3 (us-central1-c)"
        CP3[控制平面节点 3] --> ETCD3[etcd 3]
    end
    
    subgraph "工作节点池"
        WP1[工作节点 1] --> WP11[Pods]
        WP2[工作节点 2] --> WP22[Pods]
        WP3[工作节点 3] --> WP33[Pods]
    end
    
    LB[外部负载均衡器] --> CP1
    LB --> CP2
    LB --> CP3
    
    INTLB[内部负载均衡器] --> ETCD1
    INTLB --> ETCD2
    INTLB --> ETCD3
    
    LB --> WP1
    LB --> WP2
    LB --> WP3
    
    LB[外部负载均衡器] -->|API Server| Users[用户/客户端]
    LB -->|Ingress流量| Applications[应用流量]

高可用关键要素：

控制平面冗余：至少3个控制平面节点，分布在不同可用区
etcd 高可用：etcd 集群同样跨可用区部署，使用奇数节点（3或5）
外部负载均衡：前端负载均衡器处理API服务器和Ingress流量
内部负载均衡：内部负载均衡器处理etcd集群通信
工作节点多AZ：工作节点池分布在多个可用区
网络分区容忍：配置适当的超时和重试策略
数据备份：定期备份etcd数据

2. GCP 上搭建高可用 Kubernetes 集群#

2.1 前期准备与环境配置#

# 1. 安装 Google Cloud SDK
# Ubuntu/Debian
sudo apt-get update && sudo apt-get install -y ca-certificates curl gnupg
sudo mkdir -p /etc/apt/keyrings
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /etc/apt/keyrings/google-cloud-sdk.gpg
echo "deb [signed-by=/etc/apt/keyrings/google-cloud-sdk.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
sudo apt-get update && sudo apt-get install -y google-cloud-sdk

# 2. 配置 gcloud
gcloud init  # 交互式初始化，登录并选择项目
gcloud config set compute/zone us-central1-a  # 设置默认区域
gcloud config set compute/region us-central1  # 设置默认地区

# 3. 启用必要API
gcloud services enable compute.googleapis.com
gcloud services enable container.googleapis.com
gcloud services enable servicenetworking.googleapis.com
gcloud services enable sqladmin.googleapis.com  # 用于etcd备份

# 4. 创建专用VPC网络
gcloud compute networks create k8s-ha-vpc \
    --subnet-mode=custom \
    --mtu=1460 \
    --bgp-routing-mode=regional

# 5. 创建子网（跨区域）
gcloud compute networks subnets create k8s-subnet-a \
    --network=k8s-ha-vpc \
    --region=us-central1 \
    --range=10.0.1.0/24 \
    --zone=us-central1-a

gcloud compute networks subnets create k8s-subnet-b \
    --network=k8s-ha-vpc \
    --region=us-central1 \
    --range=10.0.2.0/24 \
    --zone=us-central1-b

gcloud compute networks subnets create k8s-subnet-c \
    --network=k8s-ha-vpc \
    --region=us-central1 \
    --range=10.0.3.0/24 \
    --zone=us-central1-c

# 6. 配置防火墙规则
# 控制平面内部通信
gcloud compute firewall-rules create k8s-control-plane-internal \
    --network=k8s-ha-vpc \
    --allow=tcp:6443,tcp:2379-2380,tcp:10250-10252 \
    --source-ranges=10.0.0.0/16 \
    --target-tags=k8s-control-plane

# API Server 外部访问
gcloud compute firewall-rules create k8s-api-external \
    --network=k8s-ha-vpc \
    --allow=tcp:6443 \
    --source-ranges=0.0.0.0/0 \
    --target-tags=k8s-control-plane

# 节点通信
gcloud compute firewall-rules create k8s-node-internal \
    --network=k8s-ha-vpc \
    --allow=tcp:10250,tcp:30000-32767,udp:30000-32767 \
    --source-ranges=10.0.0.0/16 \
    --target-tags=k8s-node

# 7. 创建Cloud Storage桶用于etcd备份
gcloud storage buckets create gs://k8s-ha-etcd-backups-$(date +%s) \
    --location=us-central1 \
    --uniform-bucket-level-access

关键点解释：

VPC和子网设计：创建了专用VPC并在三个不同区域创建子网，为跨可用区部署奠定基础
防火墙规则：精细控制流量，遵循最小权限原则
- 6443端口：API Server
- 2379-2380：etcd客户端和对等通信
- 10250-10252：kubelet和控制器管理器端口
Cloud Storage桶：使用统一访问控制，为etcd备份提供持久化存储

2.2 创建负载均衡器和健康检查#

# 1. 创建健康检查
gcloud compute health-checks create tcp k8s-api-health-check \
    --port=6443 \
    --check-interval=5s \
    --timeout=3s \
    --healthy-threshold=2 \
    --unhealthy-threshold=2

gcloud compute health-checks create tcp k8s-etcd-health-check \
    --port=2379 \
    --check-interval=5s \
    --timeout=3s \
    --healthy-threshold=2 \
    --unhealthy-threshold=2

# 2. 创建后端服务
gcloud compute backend-services create k8s-api-backend \
    --protocol=TCP \
    --port-name=https \
    --health-checks=k8s-api-health-check \
    --global

gcloud compute backend-services create k8s-etcd-backend \
    --protocol=TCP \
    --port-name=etcd \
    --health-checks=k8s-etcd-health-check \
    --global

# 3. 创建实例组并添加到后端
# 首先创建未托管实例组（稍后添加实例）
gcloud compute instance-groups unmanaged create k8s-api-ig-a \
    --zone=us-central1-a
gcloud compute instance-groups unmanaged create k8s-api-ig-b \
    --zone=us-central1-b
gcloud compute instance-groups unmanaged create k8s-api-ig-c \
    --zone=us-central1-c

# 将实例组添加到后端服务
gcloud compute backend-services add-backend k8s-api-backend \
    --instance-group=k8s-api-ig-a \
    --instance-group-zone=us-central1-a \
    --balancing-mode=CONNECTION \
    --max-connections=1000 \
    --global

gcloud compute backend-services add-backend k8s-api-backend \
    --instance-group=k8s-api-ig-b \
    --instance-group-zone=us-central1-b \
    --balancing-mode=CONNECTION \
    --max-connections=1000 \
    --global

gcloud compute backend-services add-backend k8s-api-backend \
    --instance-group=k8s-api-ig-c \
    --instance-group-zone=us-central1-c \
    --balancing-mode=CONNECTION \
    --max-connections=1000 \
    --global

# 4. 创建全局转发规则和IP地址
gcloud compute addresses create k8s-api-ip --global
API_IP=$(gcloud compute addresses describe k8s-api-ip --global --format='value(address)')

gcloud compute forwarding-rules create k8s-api-fr \
    --global \
    --target-tcp-proxy=k8s-api-tcp-proxy \
    --ports=6443 \
    --address=$API_IP

# 5. 创建TCP代理
gcloud compute target-tcp-proxies create k8s-api-tcp-proxy \
    --backend-service=k8s-api-backend

echo "API Server LB IP: $API_IP"

关键点解释：

健康检查：配置积极的健康检查参数（5秒间隔，3秒超时），确保快速检测故障
后端服务：使用TCP协议，因为Kubernetes API是TLS加密的
实例组：创建未托管实例组，稍后在创建VM实例时将它们添加到这些组
全局负载均衡：使用全球负载均衡器，提供单一入口点，跨区域分发流量
静态IP：分配静态IP以便集群端点保持稳定

2.3 创建控制平面节点#

# 1. 生成集群CA证书和密钥
mkdir -p certs && cd certs
openssl genrsa -out ca.key 2048
openssl req -x509 -new -nodes -key ca.key -subj "/CN=kubernetes" -days 3650 -out ca.crt

# 2. 生成etcd证书
cat > etcd-csr.json <<EOF
{
  "CN": "etcd",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "CA",
      "O": "Kubernetes",
      "OU": "etcd",
      "ST": "San Francisco"
    }
  ]
}
EOF

cfssl gencert -initca etcd-csr.json | cfssljson -bare etcd
cfssl gencert \
  -ca=etcd.pem \
  -ca-key=etcd-key.pem \
  -config=ca-config.json \
  -hostname=127.0.0.1,$ETCD_LB_IP \
  -profile=server \
  etcd-csr.json | cfssljson -bare etcd-server

# 3. 创建启动脚本模板
cat > control-plane-startup.sh <<'EOF'
#!/bin/bash
# 安装基础依赖
apt-get update
apt-get install -y apt-transport-https ca-certificates curl software-properties-common gnupg lsb-release

# 安装containerd
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null
apt-get update
apt-get install -y containerd.io
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml
# 确保使用systemd cgroup驱动
sed -i 's/SystemdCgroup \= false/SystemdCgroup \= true/g' /etc/containerd/config.toml
systemctl restart containerd

# 安装kubeadm、kubelet和kubectl
curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | tee /etc/apt/sources.list.d/kubernetes.list
apt-get update
apt-get install -y kubelet=1.26.0-00 kubeadm=1.26.0-00 kubectl=1.26.0-00
apt-mark hold kubelet kubeadm kubectl

# 配置kubelet
cat <<EOF | tee /etc/default/kubelet
KUBELET_EXTRA_ARGS=--cloud-provider=gce
EOF

systemctl daemon-reload
systemctl enable kubelet
systemctl start kubelet

# 仅在第一个节点上初始化控制平面
if [ "$IS_FIRST_CONTROL_PLANE" = "true" ]; then
  # 创建kubeadm配置
  cat > kubeadm-config.yaml <<EOF2
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
kubernetesVersion: v1.26.0
controlPlaneEndpoint: "${API_LB_IP}:6443"
etcd:
  external:
    endpoints:
    - https://${ETCD_LB_IP}:2379
    caFile: /etc/kubernetes/pki/etcd/ca.crt
    certFile: /etc/kubernetes/pki/etcd/server.crt
    keyFile: /etc/kubernetes/pki/etcd/server.key
networking:
  podSubnet: 192.168.0.0/16
  serviceSubnet: 10.96.0.0/12
apiServer:
  certSANs:
  - "${API_LB_IP}"
  - "${INTERNAL_API_LB_IP}"
  extraArgs:
    cloud-provider: gce
controllerManager:
  extraArgs:
    cloud-provider: gce
EOF2

  # 初始化控制平面
  kubeadm init --config=kubeadm-config.yaml
  
  # 设置kubectl
  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config
  
  # 安装Calico CNI
  kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/calico.yaml
  
  # 保存加入命令
  kubeadm token create --print-join-command > /root/join-command.sh
  chmod +x /root/join-command.sh
fi
EOF

# 4. 创建控制平面实例
# 获取负载均衡IP
API_LB_IP=$API_IP
ETCD_LB_IP=$API_IP  # 在测试环境中可以共用，生产环境建议分开

# 创建第一个控制平面节点
gcloud compute instances create k8s-cp-1 \
    --zone=us-central1-a \
    --machine-type=e2-medium \
    --subnet=k8s-subnet-a \
    --tags=k8s-control-plane \
    --metadata-from-file=startup-script=control-plane-startup.sh \
    --metadata=IS_FIRST_CONTROL_PLANE=true,API_LB_IP=$API_LB_IP,ETCD_LB_IP=$ETCD_LB_IP \
    --scopes=cloud-platform \
    --image-family=ubuntu-2204-lts \
    --image-project=ubuntu-os-cloud \
    --boot-disk-size=50GB \
    --boot-disk-type=pd-ssd

# 创建第二个控制平面节点
gcloud compute instances create k8s-cp-2 \
    --zone=us-central1-b \
    --machine-type=e2-medium \
    --subnet=k8s-subnet-b \
    --tags=k8s-control-plane \
    --metadata-from-file=startup-script=control-plane-startup.sh \
    --metadata=IS_FIRST_CONTROL_PLANE=false,API_LB_IP=$API_LB_IP,ETCD_LB_IP=$ETCD_LB_IP \
    --scopes=cloud-platform \
    --image-family=ubuntu-2204-lts \
    --image-project=ubuntu-os-cloud \
    --boot-disk-size=50GB \
    --boot-disk-type=pd-ssd

# 创建第三个控制平面节点
gcloud compute instances create k8s-cp-3 \
    --zone=us-central1-c \
    --machine-type=e2-medium \
    --subnet=k8s-subnet-c \
    --tags=k8s-control-plane \
    --metadata-from-file=startup-script=control-plane-startup.sh \
    --metadata=IS_FIRST_CONTROL_PLANE=false,API_LB_IP=$API_LB_IP,ETCD_LB_IP=$ETCD_LB_IP \
    --scopes=cloud-platform \
    --image-family=ubuntu-2204-lts \
    --image-project=ubuntu-os-cloud \
    --boot-disk-size=50GB \
    --boot-disk-type=pd-ssd

# 5. 将实例添加到实例组
for instance in k8s-cp-1 k8s-cp-2 k8s-cp-3; do
  zone_suffix=$(echo $instance | cut -d'-' -f3)
  case $zone_suffix in
    1) zone=us-central1-a; ig=k8s-api-ig-a ;;
    2) zone=us-central1-b; ig=k8s-api-ig-b ;;
    3) zone=us-central1-c; ig=k8s-api-ig-c ;;
  esac
  
  gcloud compute instance-groups unmanaged add-instances $ig \
    --instances=$instance \
    --zone=$zone
done

关键点解释：

证书管理：
- 为集群生成CA证书，这是所有其他证书的基础
- etcd证书需要包含负载均衡IP，以支持外部访问
容器运行时：
- 使用containerd而非Docker，遵循Kubernetes推荐
- 配置systemd cgroup驱动，确保与kubelet兼容
kubeadm配置：
- controlPlaneEndpoint指向负载均衡IP，这是高可用的关键
- 外部etcd配置允许独立管理etcd集群
- 网络配置指定Pod和服务CIDR范围
- 云提供商设置为gce，启用GCP集成功能
CNI插件：
- Calico提供网络策略和BGP路由，适合生产环境
- 其他选项包括Cilium（eBPF）或Flannel（简单）
节点分布：
- 控制平面节点分布在三个不同可用区，确保区域故障不会导致整个控制平面宕机
- 每个节点使用SSD磁盘，为etcd提供良好I/O性能
启动脚本：
- 条件逻辑确保只有第一个节点执行kubeadm init
- 其他节点等待加入命令，实现自动化集群扩展

2.4 设置工作节点#

# 1. 创建工作节点启动脚本
cat > worker-node-startup.sh <<'EOF'
#!/bin/bash
# 安装基础依赖
apt-get update
apt-get install -y apt-transport-https ca-certificates curl software-properties-common gnupg lsb-release

# 安装containerd
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null
apt-get update
apt-get install -y containerd.io
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml
# 确保使用systemd cgroup驱动
sed -i 's/SystemdCgroup \= false/SystemdCgroup \= true/g' /etc/containerd/config.toml
systemctl restart containerd

# 安装kubeadm、kubelet和kubectl
curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | tee /etc/apt/sources.list.d/kubernetes.list
apt-get update
apt-get install -y kubelet=1.26.0-00 kubeadm=1.26.0-00 kubectl=1.26.0-00
apt-mark hold kubelet kubeadm kubectl

# 配置kubelet
cat <<EOF | tee /etc/default/kubelet
KUBELET_EXTRA_ARGS=--cloud-provider=gce
EOF

systemctl daemon-reload
systemctl enable kubelet
systemctl start kubelet

# 从第一个控制平面节点获取加入命令
# 在实际部署中，这应该通过安全的方式实现，如Cloud Storage或Secret Manager
until curl -f http://k8s-cp-1/join-command.sh -o /root/join-command.sh; do
  echo "等待加入命令可用..."
  sleep 10
done

chmod +x /root/join-command.sh
/root/join-command.sh
EOF

# 2. 创建实例模板
gcloud compute instance-templates create k8s-worker-template \
    --machine-type=e2-medium \
    --subnet=k8s-subnet-a \
    --tags=k8s-node \
    --metadata-from-file=startup-script=worker-node-startup.sh \
    --scopes=cloud-platform \
    --image-family=ubuntu-2204-lts \
    --image-project=ubuntu-os-cloud \
    --boot-disk-size=50GB \
    --boot-disk-type=pd-standard

# 3. 创建实例组
gcloud compute instance-groups managed create k8s-worker-ig-a \
    --zone=us-central1-a \
    --base-instance-name=k8s-worker-a \
    --template=k8s-worker-template \
    --size=2

gcloud compute instance-groups managed create k8s-worker-ig-b \
    --zone=us-central1-b \
    --base-instance-name=k8s-worker-b \
    --template=k8s-worker-template \
    --size=2

gcloud compute instance-groups managed create k8s-worker-ig-c \
    --zone=us-central1-c \
    --base-instance-name=k8s-worker-c \
    --template=k8s-worker-template \
    --size=2

# 4. 配置自动扩缩容
gcloud compute instance-groups managed set-autoscaling k8s-worker-ig-a \
    --zone=us-central1-a \
    --max-num-replicas=10 \
    --min-num-replicas=2 \
    --target-cpu-utilization=0.6 \
    --cool-down-period=60

gcloud compute instance-groups managed set-autoscaling k8s-worker-ig-b \
    --zone=us-central1-b \
    --max-num-replicas=10 \
    --min-num-replicas=2 \
    --target-cpu-utilization=0.6 \
    --cool-down-period=60

gcloud compute instance-groups managed set-autoscaling k8s-worker-ig-c \
    --zone=us-central1-c \
    --max-num-replicas=10 \
    --min-num-replicas=2 \
    --target-cpu-utilization=0.6 \
    --cool-down-period=60

关键点解释：

实例模板：
- 使用托管实例组(Managed Instance Groups)，提供自动修复和扩缩容能力
- 启动脚本从第一个控制平面节点获取加入命令，实现自动化
节点分布：
- 工作节点同样分布在三个可用区，确保应用高可用
- 每个区域至少2个节点，防止单节点故障影响服务
自动扩缩容：
- 基于CPU利用率(60%)触发扩缩容
- 冷却期设置为60秒，避免频繁扩缩容
- 每个区域最大10个节点，根据需求调整
云提供商集成：
- 启用GCE云提供商，支持自动创建负载均衡器、持久卷等
- 实例具有cloud-platform范围，允许访问GCP API

2.5 验证和配置集群#

# 1. 等待集群就绪
echo "等待集群初始化完成..."
sleep 300  # 等待5分钟

# 2. 从第一个控制平面节点获取kubeconfig
gcloud compute scp k8s-cp-1:~/.kube/config ~/.kube/config --zone=us-central1-a --quiet
sed -i "s/server:.*/server: https:\/\/$API_LB_IP:6443/g" ~/.kube/config

# 3. 验证集群状态
kubectl get nodes -o wide
# 预期输出: 3个控制平面节点和6个工作节点，全部为Ready状态

# 4. 验证核心组件
kubectl get pods -n kube-system
# 确保coredns, calico-node, kube-proxy等组件正常运行

# 5. 配置etcd备份
cat > etcd-backup-cronjob.yaml <<EOF
apiVersion: batch/v1
kind: CronJob
metadata:
  name: etcd-backup
  namespace: kube-system
spec:
  schedule: "0 */6 * * *"  # 每6小时
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: etcd-backup-sa
          nodeSelector:
            node-role.kubernetes.io/control-plane: ""
          hostNetwork: true
          containers:
          - name: etcd-backup
            image: bitnami/etcd:3.5
            command: ["/bin/sh", "-c"]
            args:
            - |
              ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt \
              --cert=/etc/kubernetes/pki/etcd/server.crt \
              --key=/etc/kubernetes/pki/etcd/server.key \
              --endpoints=https://127.0.0.1:2379 \
              snapshot save /backup/etcd-snapshot-\$(date +%Y-%m-%d_%H-%M-%S).db
              gsutil cp /backup/etcd-snapshot-\$(date +%Y-%m-%d_%H-%M-%S).db gs://k8s-ha-etcd-backups/
            volumeMounts:
            - name: backup-dir
              mountPath: /backup
            - name: etcd-certs
              mountPath: /etc/kubernetes/pki/etcd
          volumes:
          - name: backup-dir
            emptyDir: {}
          - name: etcd-certs
            hostPath:
              path: /etc/kubernetes/pki/etcd
          restartPolicy: OnFailure
EOF

# 6. 创建etcd备份服务账户
cat > etcd-backup-rbac.yaml <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: etcd-backup-sa
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: etcd-backup-role
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: etcd-backup-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: etcd-backup-role
subjects:
- kind: ServiceAccount
  name: etcd-backup-sa
  namespace: kube-system
EOF

kubectl apply -f etcd-backup-rbac.yaml
kubectl apply -f etcd-backup-cronjob.yaml

# 7. 配置Metrics Server (用于HPA)
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

# 8. 配置集群自动扩缩容
# 创建集群自动扩缩容服务账户
gcloud iam service-accounts create cluster-autoscaler --display-name="Cluster Autoscaler SA"
gcloud projects add-iam-policy-binding $(gcloud config get-value project) \
    --member="serviceAccount:cluster-autoscaler@$(gcloud config get-value project).iam.gserviceaccount.com" \
    --role="roles/compute.instanceAdmin.v1"
gcloud projects add-iam-policy-binding $(gcloud config get-value project) \
    --member="serviceAccount:cluster-autoscaler@$(gcloud config get-value project).iam.gserviceaccount.com" \
    --role="roles/monitoring.viewer"
gcloud projects add-iam-policy-binding $(gcloud config get-value project) \
    --member="serviceAccount:cluster-autoscaler@$(gcloud config get-value project).iam.gserviceaccount.com" \
    --role="roles/logging.logWriter"

# 获取服务账户密钥
gcloud iam service-accounts keys create cluster-autoscaler-key.json \
    --iam-account=cluster-autoscaler@$(gcloud config get-value project).iam.gserviceaccount.com

# 创建集群自动扩缩容部署
cat > cluster-autoscaler.yaml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cluster-autoscaler
  namespace: kube-system
  labels:
    app: cluster-autoscaler
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cluster-autoscaler
  template:
    metadata:
      labels:
        app: cluster-autoscaler
    spec:
      serviceAccountName: cluster-autoscaler
      containers:
      - name: cluster-autoscaler
        image: k8s.gcr.io/autoscaling/cluster-autoscaler:v1.26.0
        resources:
          limits:
            cpu: 100m
            memory: 300Mi
          requests:
            cpu: 100m
            memory: 300Mi
        command:
        - ./cluster-autoscaler
        - --v=4
        - --stderrthreshold=info
        - --cloud-provider=gce
        - --skip-nodes-with-local-storage=false
        - --expander=least-waste
        - --node-group-auto-discovery=instance-groups:gce-zone=us-central1,instance-groups=k8s-worker-ig-a,k8s-worker-ig-b,k8s-worker-ig-c
        - --scale-down-enabled
        - --scale-down-utilization-threshold=0.5
        - --scale-down-delay-after-add=10m
        - --scale-down-delay-after-delete=1m
        - --scale-down-delay-after-failure=3m
        - --scale-down-unneeded-time=10m
        env:
        - name: GOOGLE_APPLICATION_CREDENTIALS
          value: /var/run/secrets/gcp/service-account.json
        volumeMounts:
        - name: service-account-volume
          mountPath: /var/run/secrets/gcp
      volumes:
      - name: service-account-volume
        secret:
          secretName: cluster-autoscaler-secret
      tolerations:
      - key: "node-role.kubernetes.io/control-plane"
        operator: "Exists"
        effect: "NoSchedule"
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 1
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: "node-role.kubernetes.io/control-plane"
                  operator: "Exists"
              topologyKey: "kubernetes.io/hostname"
EOF

# 创建secret和部署
kubectl create secret generic cluster-autoscaler-secret \
    --namespace=kube-system \
    --from-file=service-account.json=cluster-autoscaler-key.json

kubectl apply -f cluster-autoscaler.yaml

# 9. 验证完整集群功能
echo "创建测试部署和Service"
cat > test-app.yaml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.25
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  type: LoadBalancer
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: nginx
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx
spec:
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx
            port:
              number: 80
EOF

kubectl apply -f test-app.yaml

echo "等待测试应用部署完成..."
sleep 60

kubectl get pods,svc,ingress -o wide
echo "集群部署完成！API Server地址: https://$API_LB_IP:6443"

关键点解释：

etcd备份：
- 使用CronJob定期备份etcd
- 通过Cloud Storage持久化备份
- 使用专用服务账户，遵循最小权限原则
集群自动扩缩容：
- 集成GCP IAM权限
- 配置缩容策略，平衡成本和性能
- 优先将自动扩缩容Pod调度到控制平面节点
高可用验证：
- 部署测试应用、Service和Ingress
- 验证跨区域部署和负载均衡
- 确保Ingress控制器正常工作
可观测性基础：
- 部署Metrics Server，为HPA和kubectl top提供支持
- 为后续监控集成奠定基础

3. AWS 上搭建高可用 Kubernetes 集群#

3.1 前期准备与环境配置#

# 1. 安装 AWS CLI
# Ubuntu/Debian
sudo apt-get update
sudo apt-get install -y awscli

# 配置 AWS CLI (交互式)
aws configure
# 输入您的访问密钥、秘密密钥、区域(us-east-1)和输出格式(json)

# 2. 创建专用VPC
VPC_ID=$(aws ec2 create-vpc \
    --cidr-block 10.0.0.0/16 \
    --query 'Vpc.VpcId' \
    --output text)
aws ec2 modify-vpc-attribute \
    --vpc-id $VPC_ID \
    --enable-dns-support '{"Value":true}'
aws ec2 modify-vpc-attribute \
    --vpc-id $VPC_ID \
    --enable-dns-hostnames '{"Value":true}'
aws ec2 create-tags \
    --resources $VPC_ID \
    --tags Key=Name,Value=k8s-ha-vpc

# 3. 创建子网（跨可用区）
SUBNET_A=$(aws ec2 create-subnet \
    --vpc-id $VPC_ID \
    --cidr-block 10.0.1.0/24 \
    --availability-zone us-east-1a \
    --query 'Subnet.SubnetId' \
    --output text)
SUBNET_B=$(aws ec2 create-subnet \
    --vpc-id $VPC_ID \
    --cidr-block 10.0.2.0/24 \
    --availability-zone us-east-1b \
    --query 'Subnet.SubnetId' \
    --output text)
SUBNET_C=$(aws ec2 create-subnet \
    --vpc-id $VPC_ID \
    --cidr-block 10.0.3.0/24 \
    --availability-zone us-east-1c \
    --query 'Subnet.SubnetId' \
    --output text)

aws ec2 create-tags --resources $SUBNET_A --tags Key=Name,Value=k8s-subnet-a
aws ec2 create-tags --resources $SUBNET_B --tags Key=Name,Value=k8s-subnet-b
aws ec2 create-tags --resources $SUBNET_C --tags Key=Name,Value=k8s-subnet-c

# 4. 创建互联网网关
IGW_ID=$(aws ec2 create-internet-gateway \
    --query 'InternetGateway.InternetGatewayId' \
    --output text)
aws ec2 attach-internet-gateway \
    --vpc-id $VPC_ID \
    --internet-gateway-id $IGW_ID
aws ec2 create-tags \
    --resources $IGW_ID \
    --tags Key=Name,Value=k8s-igw

# 5. 创建路由表和路由
RT_ID=$(aws ec2 create-route-table \
    --vpc-id $VPC_ID \
    --query 'RouteTable.RouteTableId' \
    --output text)
aws ec2 create-route \
    --route-table-id $RT_ID \
    --destination-cidr-block 0.0.0.0/0 \
    --gateway-id $IGW_ID
aws ec2 associate-route-table \
    --subnet-id $SUBNET_A \
    --route-table-id $RT_ID
aws ec2 associate-route-table \
    --subnet-id $SUBNET_B \
    --route-table-id $RT_ID
aws ec2 associate-route-table \
    --subnet-id $SUBNET_C \
    --route-table-id $RT_ID
aws ec2 create-tags \
    --resources $RT_ID \
    --tags Key=Name,Value=k8s-public-rt

# 6. 创建安全组
SG_ID=$(aws ec2 create-security-group \
    --group-name k8s-ha-sg \
    --description "Kubernetes HA Security Group" \
    --vpc-id $VPC_ID \
    --query 'GroupId' \
    --output text)

# 允许控制平面内部通信
aws ec2 authorize-security-group-ingress \
    --group-id $SG_ID \
    --protocol tcp \
    --port 6443 \
    --cidr 10.0.0.0/16
aws ec2 authorize-security-group-ingress \
    --group-id $SG_ID \
    --protocol tcp \
    --port 2379-2380 \
    --cidr 10.0.0.0/16
aws ec2 authorize-security-group-ingress \
    --group-id $SG_ID \
    --protocol tcp \
    --port 10250-10252 \
    --cidr 10.0.0.0/16

# 允许API Server外部访问
aws ec2 authorize-security-group-ingress \
    --group-id $SG_ID \
    --protocol tcp \
    --port 6443 \
    --cidr 0.0.0.0/0

# 允许节点通信
aws ec2 authorize-security-group-ingress \
    --group-id $SG_ID \
    --protocol tcp \
    --port 10250 \
    --cidr 10.0.0.0/16
aws ec2 authorize-security-group-ingress \
    --group-id $SG_ID \
    --protocol tcp \
    --port 30000-32767 \
    --cidr 0.0.0.0/0
aws ec2 authorize-security-group-ingress \
    --group-id $SG_ID \
    --protocol udp \
    --port 30000-32767 \
    --cidr 0.0.0.0/0

# 允许SSH访问（仅管理使用）
aws ec2 authorize-security-group-ingress \
    --group-id $SG_ID \
    --protocol tcp \
    --port 22 \
    --cidr $(curl -s ifconfig.me)/32

# 7. 创建S3桶用于etcd备份
BUCKET_NAME="k8s-ha-etcd-backups-$(date +%s)"
aws s3api create-bucket \
    --bucket $BUCKET_NAME \
    --region us-east-1 \
    --create-bucket-configuration LocationConstraint=us-east-1

# 8. 创建IAM角色
# 创建控制平面实例角色
cat > control-plane-trust-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

aws iam create-role \
    --role-name k8s-control-plane-role \
    --assume-role-policy-document file://control-plane-trust-policy.json

# 附加策略
cat > control-plane-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:*",
        "elasticloadbalancing:*",
        "ecr:GetAuthorizationToken",
        "ecr:BatchCheckLayerAvailability",
        "ecr:GetDownloadUrlForLayer",
        "ecr:GetRepositoryPolicy",
        "ecr:DescribeRepositories",
        "ecr:ListImages",
        "ecr:DescribeImages",
        "ecr:BatchGetImage",
        "autoscaling:*",
        "s3:*",
        "cloudwatch:*",
        "logs:*"
      ],
      "Resource": "*"
    }
  ]
}
EOF

aws iam put-role-policy \
    --role-name k8s-control-plane-role \
    --policy-name k8s-control-plane-policy \
    --policy-document file://control-plane-policy.json

# 为工作节点创建类似角色
aws iam create-role \
    --role-name k8s-worker-role \
    --assume-role-policy-document file://control-plane-trust-policy.json

aws iam put-role-policy \
    --role-name k8s-worker-role \
    --policy-name k8s-worker-policy \
    --policy-document file://control-plane-policy.json

# 创建实例配置文件
aws iam create-instance-profile --instance-profile-name k8s-control-plane-profile
aws iam add-role-to-instance-profile \
    --instance-profile-name k8s-control-plane-profile \
    --role-name k8s-control-plane-role

aws iam create-instance-profile --instance-profile-name k8s-worker-profile
aws iam add-role-to-instance-profile \
    --instance-profile-name k8s-worker-profile \
    --role-name k8s-worker-role

echo "VPC ID: $VPC_ID"
echo "Subnet A: $SUBNET_A"
echo "Subnet B: $SUBNET_B"
echo "Subnet C: $SUBNET_C"
echo "Security Group: $SG_ID"
echo "S3 Bucket: $BUCKET_NAME"

关键点解释：

VPC设计：
- 启用DNS支持和主机名，确保内部服务发现
- 子网分布在三个不同可用区(us-east-1a/b/c)
- 互联网网关提供外部访问能力
安全组：
- 遵循最小权限原则，仅开放必要的端口
- 内部通信限制在VPC CIDR范围内
- SSH访问限制到当前IP，提高安全性
IAM角色：
- 为控制平面和工作节点创建专用角色
- 授予管理EC2、ELB、ECR、S3和CloudWatch的权限
- 使用实例配置文件将角色附加到EC2实例
S3桶：
- 创建专用S3桶存储etcd备份
- 位于同一区域，减少延迟和跨区费用

3.2 创建负载均衡器#

# 1. 创建网络负载均衡器 (NLB) 用于 API Server
# 创建目标组
API_TG_ARN=$(aws elbv2 create-target-group \
    --name k8s-api-tg \
    --protocol TCP \
    --port 6443 \
    --vpc-id $VPC_ID \
    --target-type instance \
    --health-check-path "/" \
    --health-check-protocol TCP \
    --health-check-interval-seconds 10 \
    --health-check-timeout-seconds 6 \
    --healthy-threshold-count 3 \
    --unhealthy-threshold-count 3 \
    --query 'TargetGroups[0].TargetGroupArn' \
    --output text)

# 创建负载均衡器
LB_ARN=$(aws elbv2 create-load-balancer \
    --name k8s-api-lb \
    --subnets $SUBNET_A $SUBNET_B $SUBNET_C \
    --security-groups $SG_ID \
    --scheme internet-facing \
    --type network \
    --query 'LoadBalancers[0].LoadBalancerArn' \
    --output text)

# 获取DNS名称
LB_DNS=$(aws elbv2 describe-load-balancers \
    --load-balancer-arns $LB_ARN \
    --query 'LoadBalancers[0].DNSName' \
    --output text)

# 创建监听器
aws elbv2 create-listener \
    --load-balancer-arn $LB_ARN \
    --protocol TCP \
    --port 6443 \
    --default-actions Type=forward,TargetGroupArn=$API_TG_ARN

# 2. 为etcd创建内部负载均衡器
ETCD_TG_ARN=$(aws elbv2 create-target-group \
    --name k8s-etcd-tg \
    --protocol TCP \
    --port 2379 \
    --vpc-id $VPC_ID \
    --target-type instance \
    --health-check-path "/" \
    --health-check-protocol TCP \
    --health-check-interval-seconds 10 \
    --health-check-timeout-seconds 6 \
    --healthy-threshold-count 3 \
    --unhealthy-threshold-count 3 \
    --query 'TargetGroups[0].TargetGroupArn' \
    --output text)

ETCD_LB_ARN=$(aws elbv2 create-load-balancer \
    --name k8s-etcd-lb \
    --subnets $SUBNET_A $SUBNET_B $SUBNET_C \
    --security-groups $SG_ID \
    --scheme internal \
    --type network \
    --query 'LoadBalancers[0].LoadBalancerArn' \
    --output text)

ETCD_LB_DNS=$(aws elbv2 describe-load-balancers \
    --load-balancer-arns $ETCD_LB_ARN \
    --query 'LoadBalancers[0].DNSName' \
    --output text)

aws elbv2 create-listener \
    --load-balancer-arn $ETCD_LB_ARN \
    --protocol TCP \
    --port 2379 \
    --default-actions Type=forward,TargetGroupArn=$ETCD_TG_ARN

echo "API Load Balancer DNS: $LB_DNS"
echo "etcd Load Balancer DNS: $ETCD_LB_DNS"

关键点解释：

网络负载均衡器(NLB)：
- 选择NLB而非ALB，因为API Server使用TLS且需要传输层负载均衡
- 配置积极的健康检查(10秒间隔，6秒超时)
- 跨三个可用区部署，确保高可用
内部etcd负载均衡器：
- 使用内部NLB，确保etcd通信不暴露到公网
- 同样跨三个可用区，提高etcd集群的可用性
健康检查配置：
- TCP健康检查适合加密服务
- 健康/不健康阈值设置为3，避免网络抖动导致的误判

3.3 创建控制平面节点#

# 1. 创建密钥对（用于SSH访问）
aws ec2 create-key-pair \
    --key-name k8s-ha-key \
    --query 'KeyMaterial' \
    --output text > k8s-ha-key.pem
chmod 400 k8s-ha-key.pem

# 2. 生成集群证书
# 在本地执行以下命令：
mkdir -p certs && cd certs

# CA配置
cat > ca-config.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
        "usages": ["signing", "key encipherment", "server auth", "client auth"],
        "expiry": "87600h"
      }
    }
  }
}
EOF

# CA CSR
cat > ca-csr.json <<EOF
{
  "CN": "Kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "CA",
      "O": "Kubernetes",
      "OU": "CA",
      "ST": "San Francisco"
    }
  ]
}
EOF

# 生成CA
cfssl gencert -initca ca-csr.json | cfssljson -bare ca

# 3. 创建控制平面启动脚本
cat > control-plane-userdata.sh <<EOF
#!/bin/bash
# 安装基础依赖
apt-get update
apt-get install -y apt-transport-https ca-certificates curl software-properties-common gnupg lsb-release

# 安装containerd
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=\$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \$(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null
apt-get update
apt-get install -y containerd.io
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml
# 确保使用systemd cgroup驱动
sed -i 's/SystemdCgroup \= false/SystemdCgroup \= true/g' /etc/containerd/config.toml
systemctl restart containerd

# 安装kubeadm、kubelet和kubectl
curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | tee /etc/apt/sources.list.d/kubernetes.list
apt-get update
apt-get install -y kubelet=1.26.0-00 kubeadm=1.26.0-00 kubectl=1.26.0-00
apt-mark hold kubelet kubeadm kubectl

# 配置kubelet
cat <<EOF2 | tee /etc/default/kubelet
KUBELET_EXTRA_ARGS=--cloud-provider=aws
EOF2

systemctl daemon-reload
systemctl enable kubelet
systemctl start kubelet

# 等待实例元数据服务可用
sleep 10

# 获取实例信息
INSTANCE_ID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
AVAILABILITY_ZONE=$(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone)
REGION=\${AVAILABILITY_ZONE::-1}

# 配置AWS云提供商
cat > /etc/kubernetes/aws.yaml <<EOF3
apiVersion: cloud-provider-aws/v1alpha1
kind: ClusterConfig
clusterName: k8s-ha-cluster
region: \$REGION
EOF3

# 仅在第一个节点上初始化控制平面
if [ "\$IS_FIRST_CONTROL_PLANE" = "true" ]; then
  # 创建kubeadm配置
  cat > kubeadm-config.yaml <<EOF4
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
kubernetesVersion: v1.26.0
controlPlaneEndpoint: "${LB_DNS}:6443"
etcd:
  external:
    endpoints:
    - https://${ETCD_LB_DNS}:2379
    caFile: /etc/kubernetes/pki/etcd/ca.crt
    certFile: /etc/kubernetes/pki/etcd/server.crt
    keyFile: /etc/kubernetes/pki/etcd/server.key
networking:
  podSubnet: 192.168.0.0/16
  serviceSubnet: 10.96.0.0/12
apiServer:
  certSANs:
  - "${LB_DNS}"
  - "${ETCD_LB_DNS}"
  extraArgs:
    cloud-provider: aws
controllerManager:
  extraArgs:
    cloud-provider: aws
EOF4

  # 初始化控制平面
  kubeadm init --config=kubeadm-config.yaml --upload-certs
  
  # 设置kubectl
  mkdir -p \$HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf \$HOME/.kube/config
  sudo chown \$(id -u):\$(id -g) \$HOME/.kube/config
  
  # 安装Calico CNI
  kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/calico.yaml
  
  # 保存加入命令和token
  kubeadm token create --print-join-command > /root/join-command.sh
  chmod +x /root/join-command.sh
  
  # 保存证书密钥
  cert_key=\$(kubeadm init phase upload-certs --upload-certs | grep -A1 'certificate key:' | tail -n1 | tr -d '[:space:]')
  echo "CERT_KEY=\$cert_key" >> /root/join-command.sh
fi
EOF

# 4. 创建控制平面实例
# 获取第一个可用区的子网ID
SUBNET_IDS=($SUBNET_A $SUBNET_B $SUBNET_C)
AVAILABILITY_ZONES=(us-east-1a us-east-1b us-east-1c)

# 创建第一个控制平面节点
INSTANCE_ID_1=$(aws ec2 run-instances \
    --image-id ami-0d527b8c289b4af0f \  # Ubuntu 22.04 LTS us-east-1
    --instance-type t3.medium \
    --key-name k8s-ha-key \
    --security-group-ids $SG_ID \
    --subnet-id ${SUBNET_IDS[0]} \
    --iam-instance-profile Name=k8s-control-plane-profile \
    --user-data file://control-plane-userdata.sh \
    --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=k8s-cp-1},{Key=Role,Value=control-plane}]' \
    --query 'Instances[0].InstanceId' \
    --output text)

# 等待实例运行
aws ec2 wait instance-running --instance-ids $INSTANCE_ID_1

# 为第一个节点添加到目标组的脚本
INSTANCE_IP_1=$(aws ec2 describe-instances \
    --instance-ids $INSTANCE_ID_1 \
    --query 'Reservations[0].Instances[0].PrivateIpAddress' \
    --output text)

aws elbv2 register-targets \
    --target-group-arn $API_TG_ARN \
    --targets Id=$INSTANCE_ID_1,Port=6443
aws elbv2 register-targets \
    --target-group-arn $ETCD_TG_ARN \
    --targets Id=$INSTANCE_ID_1,Port=2379

# 创建第二个控制平面节点
INSTANCE_ID_2=$(aws ec2 run-instances \
    --image-id ami-0d527b8c289b4af0f \
    --instance-type t3.medium \
    --key-name k8s-ha-key \
    --security-group-ids $SG_ID \
    --subnet-id ${SUBNET_IDS[1]} \
    --iam-instance-profile Name=k8s-control-plane-profile \
    --user-data file://control-plane-userdata.sh \
    --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=k8s-cp-2},{Key=Role,Value=control-plane}]' \
    --query 'Instances[0].InstanceId' \
    --output text)

aws ec2 wait instance-running --instance-ids $INSTANCE_ID_2

INSTANCE_IP_2=$(aws ec2 describe-instances \
    --instance-ids $INSTANCE_ID_2 \
    --query 'Reservations[0].Instances[0].PrivateIpAddress' \
    --output text)

aws elbv2 register-targets \
    --target-group-arn $API_TG_ARN \
    --targets Id=$INSTANCE_ID_2,Port=6443
aws elbv2 register-targets \
    --target-group-arn $ETCD_TG_ARN \
    --targets Id=$INSTANCE_ID_2,Port=2379

# 创建第三个控制平面节点
INSTANCE_ID_3=$(aws ec2 run-instances \
    --image-id ami-0d527b8c289b4af0f \
    --instance-type t3.medium \
    --key-name k8s-ha-key \
    --security-group-ids $SG_ID \
    --subnet-id ${SUBNET_IDS[2]} \
    --iam-instance-profile Name=k8s-control-plane-profile \
    --user-data file://control-plane-userdata.sh \
    --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=k8s-cp-3},{Key=Role,Value=control-plane}]' \
    --query 'Instances[0].InstanceId' \
    --output text)

aws ec2 wait instance-running --instance-ids $INSTANCE_ID_3

INSTANCE_IP_3=$(aws ec2 describe-instances \
    --instance-ids $INSTANCE_ID_3 \
    --query 'Reservations[0].Instances[0].PrivateIpAddress' \
    --output text)

aws elbv2 register-targets \
    --target-group-arn $API_TG_ARN \
    --targets Id=$INSTANCE_ID_3,Port=6443
aws elbv2 register-targets \
    --target-group-arn $ETCD_TG_ARN \
    --targets Id=$INSTANCE_ID_3,Port=2379

echo "控制平面节点创建完成"
echo "节点1 ID: $INSTANCE_ID_1 IP: $INSTANCE_IP_1"
echo "节点2 ID: $INSTANCE_ID_2 IP: $INSTANCE_IP_2"
echo "节点3 ID: $INSTANCE_ID_3 IP: $INSTANCE_IP_3"

关键点解释：

云提供商集成：
- 配置AWS云提供商，启用ELB、EBS和自动扩展组集成
- 使用实例元数据服务获取区域和实例ID
证书管理：
- 使用--upload-certs将控制平面证书上传到集群
- 为其他控制平面节点生成证书密钥
负载均衡器集成：
- 将实例注册到目标组，实现自动健康检查
- 同时注册到API Server和etcd负载均衡器
区域分布：
- 控制平面节点分布在三个不同的AWS可用区
- 每个节点位于不同的子网，提高可用性

3.4 设置工作节点#

# 1. 创建工作节点用户数据脚本
cat > worker-node-userdata.sh <<EOF
#!/bin/bash
# 安装基础依赖
apt-get update
apt-get install -y apt-transport-https ca-certificates curl software-properties-common gnupg lsb-release

# 安装containerd
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=\$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \$(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null
apt-get update
apt-get install -y containerd.io
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml
# 确保使用systemd cgroup驱动
sed -i 's/SystemdCgroup \= false/SystemdCgroup \= true/g' /etc/containerd/config.toml
systemctl restart containerd

# 安装kubeadm、kubelet和kubectl
curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | tee /etc/apt/sources.list.d/kubernetes.list
apt-get update
apt-get install -y kubelet=1.26.0-00 kubeadm=1.26.0-00 kubectl=1.26.0-00
apt-mark hold kubelet kubeadm kubectl

# 配置kubelet
cat <<EOF2 | tee /etc/default/kubelet
KUBELET_EXTRA_ARGS=--cloud-provider=aws
EOF2

systemctl daemon-reload
systemctl enable kubelet
systemctl start kubelet

# 等待实例元数据服务可用
sleep 10

# 获取实例信息
INSTANCE_ID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
AVAILABILITY_ZONE=$(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone)
REGION=\${AVAILABILITY_ZONE::-1}

# 配置AWS云提供商
cat > /etc/kubernetes/aws.yaml <<EOF3
apiVersion: cloud-provider-aws/v1alpha1
kind: ClusterConfig
clusterName: k8s-ha-cluster
region: \$REGION
EOF3

# 从第一个控制平面节点获取加入命令
# 在实际生产环境中，这应该通过安全的方式实现，如S3或Secrets Manager
until curl -f http://$INSTANCE_IP_1/join-command.sh -o /root/join-command.sh; do
  echo "等待加入命令可用..."
  sleep 10
done

chmod +x /root/join-command.sh
CERT_KEY=\$(grep CERT_KEY /root/join-command.sh | cut -d= -f2)
/root/join-command.sh --certificate-key \$CERT_KEY
EOF

# 2. 创建启动模板
LAUNCH_TEMPLATE_ID=$(aws ec2 create-launch-template \
    --launch-template-name k8s-worker-template \
    --version-description "Initial version" \
    --launch-template-data "{
        \"ImageId\": \"ami-0d527b8c289b4af0f\",
        \"InstanceType\": \"t3.medium\",
        \"KeyName\": \"k8s-ha-key\",
        \"SecurityGroupIds\": [\"$SG_ID\"],
        \"IamInstanceProfile\": {\"Name\": \"k8s-worker-profile\"},
        \"UserData\": \"$(base64 -w0 worker-node-userdata.sh)\",
        \"TagSpecifications\": [{
            \"ResourceType\": \"instance\",
            \"Tags\": [{
                \"Key\": \"Name\",
                \"Value\": \"k8s-worker\"
            },{
                \"Key\": \"Role\",
                \"Value\": \"worker\"
            }]
        }]
    }" \
    --query 'LaunchTemplate.LaunchTemplateId' \
    --output text)

# 3. 创建自动扩展组
# 在可用区A创建ASG
aws autoscaling create-auto-scaling-group \
    --auto-scaling-group-name k8s-worker-asg-a \
    --launch-template "LaunchTemplateId=$LAUNCH_TEMPLATE_ID,Version=1" \
    --min-size 2 \
    --max-size 10 \
    --desired-capacity 2 \
    --vpc-zone-identifier ${SUBNET_IDS[0]} \
    --target-group-arns $API_TG_ARN \
    --health-check-type EC2 \
    --health-check-grace-period 300 \
    --tags "Key=Name,Value=k8s-worker-a" "Key=k8s.io/cluster-autoscaler/enabled,Value=true" "Key=k8s.io/cluster-autoscaler/k8s-ha-cluster,Value=true"

# 在可用区B创建ASG
aws autoscaling create-auto-scaling-group \
    --auto-scaling-group-name k8s-worker-asg-b \
    --launch-template "LaunchTemplateId=$LAUNCH_TEMPLATE_ID,Version=1" \
    --min-size 2 \
    --max-size 10 \
    --desired-capacity 2 \
    --vpc-zone-identifier ${SUBNET_IDS[1]} \
    --target-group-arns $API_TG_ARN \
    --health-check-type EC2 \
    --health-check-grace-period 300 \
    --tags "Key=Name,Value=k8s-worker-b" "Key=k8s.io/cluster-autoscaler/enabled,Value=true" "Key=k8s.io/cluster-autoscaler/k8s-ha-cluster,Value=true"

# 在可用区C创建ASG
aws autoscaling create-auto-scaling-group \
    --auto-scaling-group-name k8s-worker-asg-c \
    --launch-template "LaunchTemplateId=$LAUNCH_TEMPLATE_ID,Version=1" \
    --min-size 2 \
    --max-size 10 \
    --desired-capacity 2 \
    --vpc-zone-identifier ${SUBNET_IDS[2]} \
    --target-group-arns $API_TG_ARN \
    --health-check-type EC2 \
    --health-check-grace-period 300 \
    --tags "Key=Name,Value=k8s-worker-c" "Key=k8s.io/cluster-autoscaler/enabled,Value=true" "Key=k8s.io/cluster-autoscaler/k8s-ha-cluster,Value=true"

# 4. 配置集群自动扩缩容
# 创建集群自动扩缩容IAM角色
cat > autoscaler-trust-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

aws iam create-role \
    --role-name k8s-cluster-autoscaler-role \
    --assume-role-policy-document file://autoscaler-trust-policy.json

cat > autoscaler-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "autoscaling:DescribeAutoScalingGroups",
        "autoscaling:DescribeAutoScalingInstances",
        "autoscaling:DescribeLaunchConfigurations",
        "autoscaling:DescribeTags",
        "autoscaling:SetDesiredCapacity",
        "autoscaling:TerminateInstanceInAutoScalingGroup",
        "ec2:DescribeLaunchTemplateVersions"
      ],
      "Resource": "*"
    }
  ]
}
EOF

aws iam put-role-policy \
    --role-name k8s-cluster-autoscaler-role \
    --policy-name k8s-cluster-autoscaler-policy \
    --policy-document file://autoscaler-policy.json

# 创建集群自动扩缩容部署
echo "等待集群初始化完成..."
sleep 300  # 等待5分钟

# 从第一个控制平面节点获取kubeconfig
aws ec2 wait instance-status-ok --instance-ids $INSTANCE_ID_1
scp -i k8s-ha-key.pem -o StrictHostKeyChecking=no ubuntu@$INSTANCE_IP_1:~/.kube/config ~/.kube/config
sed -i "s/server:.*/server: https:\/\/$LB_DNS:6443/g" ~/.kube/config

# 创建集群自动扩缩容部署
cat > cluster-autoscaler.yaml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cluster-autoscaler
  namespace: kube-system
  labels:
    app: cluster-autoscaler
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cluster-autoscaler
  template:
    metadata:
      labels:
        app: cluster-autoscaler
    spec:
      serviceAccountName: cluster-autoscaler
      containers:
      - name: cluster-autoscaler
        image: k8s.gcr.io/autoscaling/cluster-autoscaler:v1.26.0
        resources:
          limits:
            cpu: 100m
            memory: 300Mi
          requests:
            cpu: 100m
            memory: 300Mi
        command:
        - ./cluster-autoscaler
        - --v=4
        - --stderrthreshold=info
        - --cloud-provider=aws
        - --skip-nodes-with-local-storage=false
        - --expander=least-waste
        - --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/k8s-ha-cluster
        - --scale-down-enabled
        - --scale-down-utilization-threshold=0.5
        - --scale-down-delay-after-add=10m
        - --scale-down-delay-after-delete=1m
        - --scale-down-delay-after-failure=3m
        - --scale-down-unneeded-time=10m
        env:
        - name: AWS_REGION
          value: us-east-1
      tolerations:
      - key: "node-role.kubernetes.io/control-plane"
        operator: "Exists"
        effect: "NoSchedule"
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 1
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: "node-role.kubernetes.io/control-plane"
                  operator: "Exists"
              topologyKey: "kubernetes.io/hostname"
EOF

kubectl apply -f cluster-autoscaler.yaml

# 5. 验证集群
kubectl get nodes -o wide
kubectl get pods -n kube-system

echo "集群部署完成！API Server地址: https://$LB_DNS:6443"

关键点解释：

自动扩展组(ASG)：
- 为每个可用区创建专用ASG，确保跨区可用性
- 配置最小2个节点，防止单点故障
- 标记ASG以便集群自动扩缩容识别
集群自动扩缩容：
- 配置AWS特定参数，如区域和发现标签
- 设置合理的缩容延迟，避免频繁扩缩容
- 通过节点亲和性确保调度到控制平面节点
云提供商集成：
- 每个工作节点配置AWS云提供商
- 启用EBS卷、ELB和自动扩展集成
- 健康检查宽限期(300秒)确保节点完全初始化
安全考虑：
- 使用专用IAM角色，遵循最小权限原则
- 为不同组件创建专用角色（控制平面、工作节点、自动扩缩容）
- 使用kubeconfig文件代替不安全的加入命令分发

4. 高可用集群关键设计原则总结#

控制平面冗余:
- 至少3个控制平面节点，分布在不同物理位置
- 奇数节点etcd集群，确保在分区时保持多数
- 实例类型应足够强大(T3.medium或更高)
负载均衡策略:
- API Server: 外部负载均衡器，提供单一入口点
- etcd: 内部负载均衡器，保护敏感数据
- 健康检查应积极但不过于敏感(10秒间隔，3秒超时)
- 使用TCP健康检查，适合加密服务
数据持久性和备份:
- 定期etcd备份(每6小时)到对象存储
- 自动化恢复流程
- 保留多个备份版本(每日、每周、每月)
- 定期测试恢复流程
跨区域容错:
- 工作节点分布至少2-3个可用区
- 应用使用Pod反亲和性，防止单点故障
- 数据库和服务设计为区域故障容忍
自动修复和扩缩容:
- 节点自动修复(实例组自愈)
- 基于CPU/内存和自定义指标的Pod自动扩缩容
- 基于调度失败和资源利用率的集群自动扩缩容
- 冷却期设置防止抖动(10分钟)
可观测性和监控:
- 集群级指标收集(Metrics Server)
- 节点和Pod日志聚合
- 关键组件健康检查(控制平面、etcd、网络)
- 设置告警(控制平面不可用、节点不健康、磁盘空间)
安全最佳实践:
- 最小权限IAM角色
- 网络隔离(安全组/防火墙规则)
- 加密所有通信(TLS 1.2+)
- 定期轮换证书和密钥
- 审计日志
灾难恢复计划:
- 文档化的恢复步骤
- 定期演练灾难恢复
- 跨区域备份
- 备用集群配置

5. 后续优化和扩展#

增强安全:
- 实施Pod安全策略
- 配置网络策略
- 启用审计日志
- 集成外部密钥管理服务(Vault, AWS KMS)
性能优化:
- 调整etcd参数(内存、快照)
- 优化kubelet参数
- 启用CoreDNS缓存
- 配置节点本地DNS缓存
多集群策略:
- 部署集群联邦
- 实施GitOps多集群管理
- 设置跨集群服务发现
- 实现集群级故障转移
GitOps和CI/CD:
- 集成Argo CD或Flux
- 配置自动部署流水线
- 实现金丝雀发布
- 配置自动化测试
成本优化:
- 实施Spot实例混合策略
- 配置垂直Pod自动扩缩容
- 优化资源请求和限制
- 实现非生产环境自动缩容(夜间/周末)

6. 故障排除和维护指南#

控制平面故障:

# 检查API Server
kubectl get --raw='/healthz?verbose'

# 检查etcd健康
ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  --endpoints=https://127.0.0.1:2379 endpoint health

# 检查调度器和控制器管理器
kubectl get pods -n kube-system | grep -E 'scheduler|controller-manager'

网络问题:

# 检查CoreDNS
kubectl get pods -n kube-system -l k8s-app=kube-dns

# 测试DNS解析
kubectl run -it --rm --restart=Never --image=busybox:1.28 dns-test -- nslookup kubernetes.default

# 检查kube-proxy
kubectl get pods -n kube-system -l k8s-app=kube-proxy

节点问题:

# 查看节点状态
kubectl get nodes -o wide

# 检查节点详细信息
kubectl describe node <node-name>

# 检查kubelet日志
journalctl -u kubelet -f

维护操作:

# 安全驱逐节点
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

# 恢复节点
kubectl uncordon <node-name>

# 证书轮换
kubeadm certs renew all
systemctl restart kubelet

7. 结论#

构建高可用Kubernetes集群需要全面考虑架构设计、组件配置和运维策略。无论是GCP还是AWS，核心原则相同：控制平面冗余、跨区域部署、自动化恢复和严格的安全策略。

通过本指南，您已经了解了：

如何设计高可用Kubernetes架构
GCP和AWS上具体实现步骤
关键组件配置和集成
自动扩缩容和自我修复策略
监控和维护最佳实践

生产环境中，建议：

从小规模开始，逐步扩展
建立自动化测试和验证流程
文档化所有自定义配置和流程
定期演练灾难恢复
保持组件版本最新，关注安全公告

高可用Kubernetes集群是现代云原生应用的基石，投资于其稳健性将为业务提供持续、可靠的服务能力。

构建高可用Kubernetes集群：GCP与AWS实战指南

1. 高可用Kubernetes架构概览#

2. GCP 上搭建高可用 Kubernetes 集群#

2.1 前期准备与环境配置#

2.2 创建负载均衡器和健康检查#

2.3 创建控制平面节点#

2.4 设置工作节点#

2.5 验证和配置集群#

3. AWS 上搭建高可用 Kubernetes 集群#

3.1 前期准备与环境配置#

3.2 创建负载均衡器#

3.3 创建控制平面节点#

3.4 设置工作节点#

4. 高可用集群关键设计原则总结#

5. 后续优化和扩展#

6. 故障排除和维护指南#

7. 结论#

本文共 9518 字，创建于 Dec 31, 2025