K3s 配置 Envoy Gateway：结合 Cloudflare Tunnel 与 Tailscale 实现内外网服务隔离

#K8s#Config#Cloudflare#Note

✨ 笔记记录了在 K3s 中通过 Envoy Gateway 实现服务内外网隔离的运维配置过程，包括利用 Tailscale 接入 LoadBalancer 暴露私有服务、通过 Cloudflare Tunnel 暴露公开服务，并结合 cert-manager 自动签发 Let's Encrypt TLS 证书以及配合 external-dns 实现 Cloudflare DNS 记录的自动化管理。

这几天重装迁移了一台线路较普通的服务器，正好当实验环境试试 Envoy Gateway，顺便记录下大致配置过程。

在 AI Agent 大行其道的今天，这种操作方式多少显得有点“古法运维”。AI 固然能几秒钟把服务拉起来，但如果不细致审视底层的组装细节，配置次数多了时间长了，整个就容易变成“黑盒”。按需配置、理清关键组件的流量走向，这种清晰的掌控感还是不错的。（也可能是”登味儿/守旧派“，或者说单纯烧不起 Token 🤣

配置概览

目标效果：在同一个 K3s 集群内，将公开服务与私有服务从入口层进行彻底隔离：

  • 私有服务：通过 Envoy Gateway 暴露为 LoadBalancer 类型的 Service，配合集群内 K3s 自带的 ServiceLB（原 Klipper LoadBalancer），借助 Tailscale 节点网络进行内网访问。
  • 公开服务：通过 Envoy Gateway 暴露为 ClusterIP 类型的 Service，流量仅在集群内部流转，前端由部署在集群内的 cloudflared 容器以 Cloudflare Tunnel 方式反向代理出去。
[GatewayClass: envoy-gateway] (控制面控制器)
       │
       ├──> [Gateway: envoy-gateway-tailscale]  ──> Service类型: LoadBalancer (Tailscale 内网接入)
       │
       └──> [Gateway: envoy-gateway-cloudflare] ──> Service类型: ClusterIP    (Cloudflare Tunnel 接入)

此外，配合 cert-manager 完成 Let's Encrypt TLS 证书自动签发，并利用 external-dns 根据网关类型自动同步 DNS 记录（Tailscale 自动建 A 记录，Cloudflare Tunnel 自动建 CNAME 记录）。

K3s 配置安装

https://k3s.io/

基础的系统配置和 Tailscale 安装此处略去不表。

配置文件 /etc/rancher/k3s/config.yaml 
write-kubeconfig-mode: "0600"
# token: "" # 强随机字符串 (openssl rand -hex 16)
token-file: /etc/rancher/k3s/token

bind-address: "0.0.0.0"
node-ip: "<Tailscale-IP>"
node-external-ip: "<Tailscale-IP>"
flannel-external-ip: true

tls-san:
  - "127.0.0.1"
  - "localhost"
  - "<Tailscale-IP>"
  - "<HostName>"
  - "<Tailscale-MagicDNS>"
  - "<个人域名>"

disable:
  - traefik

kube-proxy-arg:
  - "proxy-mode=nftables"

# 基础的 kubelet 参数就放在这，官方建议是使用专门的 kubelet 配置文件
kubelet-arg:
  - "max-pods=110"
  - "image-gc-high-threshold=80"
  - "image-gc-low-threshold=60"
  - "container-log-max-size=10Mi"
  - "container-log-max-files=3"
  - "eviction-hard=memory.available<300Mi,nodefs.available<2Gi"
  - "serialize-image-pulls=false"

# 需调整内核参数
protect-kernel-defaults: true
secrets-encryption: true
token-file 生成
mkdir -p /etc/rancher/k3s
TOKEN_FILE="/etc/rancher/k3s/token"
if [[ ! -f "${TOKEN_FILE}" ]]; then
    if command -v openssl >/dev/null 2>&1; then
        openssl rand -hex 16 > "${TOKEN_FILE}"
    else
        head -c 32 /dev/urandom \
            | base64 \
            | tr -dc 'a-zA-Z0-9' \
            | head -c 32 > "${TOKEN_FILE}"
    fi
    chmod 600 "${TOKEN_FILE}"
else
    log "检测到已有 token，跳过生成"
fi
内核参数配置
cat >/etc/sysctl.d/99-k3s-kubelet.conf <<EOF
# Kubernetes network forwarding
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
# Bridge netfilter
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
# Protect kernel defaults requirements
vm.overcommit_memory = 1
kernel.panic = 10
kernel.panic_on_oops = 1
EOF

sysctl --system >/dev/null

安装命令 curl -sfL https://get.k3s.io | sh -

面板用的是 Radar 项目，本机启动，CLI + Web 形式，提供 MCP Server

https://github.com/skyhook-io/radar

Envoy 网关部署

https://gateway.envoyproxy.io/
直接用 Helm 安装 Envoy Gateway，可配置项：https://gateway.envoyproxy.io/docs/install/gateway-helm-api/
# 安装
helm install eg oci://docker.io/envoyproxy/gateway-helm --version v1.9.1 -n envoy-gateway-system --create-namespace --set config.envoyGateway.logging.format=json

# 等待 Envoy Gateway 启动就绪
kubectl wait --timeout=5m -n envoy-gateway-system deployment/envoy-gateway --for=condition=Available

控制面安装好了，接下来就是 GatewayClass、Gateway、HTTPRoute 以及数据面的网络/路由发布。

创建 Gateway Class
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: eg
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
创建 EnvoyProxy 配置，覆写 Envoy Gateway Controller 之后生成的 Service 名称，
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: cloudflare-proxy-config
  namespace: envoy-gateway-system
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyService:
        type: ClusterIP
        name: envoy-svc-fixed
创建 Gateway，其中 Cloudflare Tunnel 对应引入上面的 Envoy 自定义覆写配置，Tailscale 不用
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: cloudflare-gateway
  namespace: envoy-gateway-system
spec:
  gatewayClassName: eg
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: All
	infrastructure:
    parametersRef:
      group: gateway.envoyproxy.io
      kind: EnvoyProxy
      name: cloudflare-proxy-config 
      
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: tailscale-gateway
  namespace: envoy-gateway-system
spec:
  gatewayClassName: eg
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: All
  • Tailscale Gateway：Service 为默认的 LoadBalancer。K3s 的 ServiceLB 检测到后，会在节点启动 Pod 进行端口转发与流量透传。
  • Cloudflare Gateway：引入自定义 EnvoyProxy 配置，指定 Service 为 ClusterIP，仅限 K3s 内网访问。

Tailscale 链路测试

用 Kubernetes 的 httpbin 测试服务

Deployment + Service
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
  template:
    metadata:
      labels:
        app: httpbin
    spec:
      containers:
      - name: httpbin
        image: kennethreitz/httpbin
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
  namespace: default
spec:
  selector:
    app: httpbin
  hostnames:
	  - httpbin.okhk.net
  ports:
  - port: 80
    targetPort: 80
HTTPRoute
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: httpbin
  namespace: default
spec:
  parentRefs:
  - name: tailscale-gateway
    namespace: envoy-gateway-system
		sectionName: http
  
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /

    backendRefs:
    - name: httpbin
      port: 80
本地可以通过修改 hosts 或在内网 DNS 中配置域名指向 Tailscale 节点 IP，即可成功访问。

Cloudflare Tunnel 接入

上面验证了 Tailscale 的接入，下面开始接入 Cloudflare Tunnel，将其运行在 K3s 内部，直接对接 Envoy。

Namespace
apiVersion: v1
kind: Namespace
metadata:
  name: cloudflare
Secret

在 Cloudflare Zero Trust 处创建得到 Tunnel Token，或者调用 API 也行，不做赘述

apiVersion: v1
kind: Secret
metadata:
  name: cf-tunnel-secret
  namespace: cloudflare
type: Opaque
stringData:
  tunnel-token: "<Tunnel Token>"
Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cloudflared
  namespace: cloudflare
  labels:
    app.kubernetes.io/name: cloudflared
spec:
  replicas: 1
  revisionHistoryLimit: 2

  selector:
    matchLabels:
      app.kubernetes.io/name: cloudflared

  template:
    metadata:
      labels:
        app.kubernetes.io/name: cloudflared

    spec:
      containers:
        - name: cloudflared
          image: cloudflare/cloudflared:2026.9.3
          imagePullPolicy: IfNotPresent
          args:
            - tunnel
            - --output
            - json
            - --loglevel
            - debug
            - --metrics
            - 0.0.0.0:2000
            - --no-autoupdate
            - run

          env:
            - name: TUNNEL_TOKEN
              valueFrom:
                secretKeyRef:
                  name: cf-tunnel-secret
                  key: tunnel-token

          ports:
            - name: metrics
              containerPort: 2000
              protocol: TCP

          resources:
            requests:
              cpu: 10m
              memory: 32Mi
            limits:
              cpu: 100m
              memory: 128Mi
          livenessProbe:
            httpGet:
              path: /ready
              port: metrics
            initialDelaySeconds: 15
            periodSeconds: 30
            timeoutSeconds: 5
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /ready
              port: metrics
            initialDelaySeconds: 5
            periodSeconds: 10
            timeoutSeconds: 3
            failureThreshold: 3

          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            runAsNonRoot: true
            runAsUser: 65532
            runAsGroup: 65532
            capabilities:
              drop:
                - ALL
            seccompProfile:
              type: RuntimeDefault

          volumeMounts:
            - name: tmp
              mountPath: /tmp

      volumes:
        - name: tmp
          emptyDir: {}
部署完成后，Cloudflare Dashboard 显示隧道连接已建立。配置一条 Published application routes ，Service 指向 K3s 中 Envoy Gateway Service。也就是：envoy-svc-fixed.envoy-gateway-system。此时公网流量经过 Cloudflare CDN 边缘节点，由 Tunnel 穿透注入集群，再传递给 Envoy Gateway 进行路由分发。

至此 Tailscale 和 Cloudflare Tunnel 访问 K3s 服务的基本链路已经测试通过。

cert-manager 自动签发 Let's Encrypt 证书

Helm 部署
helm install \
  cert-manager oci://quay.io/jetstack/charts/cert-manager \
  --version v1.21.2 \
  --namespace cert-manager \
  --create-namespace \
  --set crds.enabled=true
Cloudflare API Key 创建

登录 Cloudflare Dashboard，进入 My Profile -> API Tokens，创建 Token：
• Permissions：
◦ Zone - DNS - Edit
◦ Zone - Zone - Read
• Zone Resources：选择包含您域名的 Zone（或 Include - All zones）

将 Token 存储为 Kubernetes Secret，后续要使用 ClusterIssuer ，必须存储在 cert-manager 相同 namespace 下
kubectl create secret generic cloudflare-api-token-secret \
  --namespace cert-manager \
  --from-literal=api-token=<CLOUDFLARE_API_TOKEN>
配置 cert-manager ClusterIssuer 开启 DNS-01 Challenge 认证
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-cloudflare-dns
spec:
  acme:
    # 邮箱，用于接收 LetsEncrypt 证书到期提醒
    email: your-email@example.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-cloudflare-account-key
    solvers:
    - dns01:
        cloudflare:
          apiTokenSecretRef:
            name: cloudflare-api-token-secret
            key: api-token
申请证书（Envoy Gateway Namespace 下）
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: okhk-tls
  namespace: envoy-gateway-system
spec:
  secretName: okhk-tls-secret
  issuerRef:
    name: letsencrypt-cloudflare-dns
    kind: ClusterIssuer
  dnsNames:
  - "*.okhk.net"
在 Envoy Gateway 的 Listener 中引用挂载 TLS 证书
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: envoy-gateway
  namespace: envoy-gateway-system
spec:
  gatewayClassName: eg
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: All
    - name: https
      protocol: HTTPS
      port: 443
      hostname: "*.okhk.net"
      tls:
        mode: Terminate
        certificateRefs:
          - kind: Secret
            name: okhk-tls-secret
      allowedRoutes:
        namespaces:
          from: All
  infrastructure:
    parametersRef:
      group: gateway.envoyproxy.io
      kind: EnvoyProxy
      name: custom-proxy-config

至此 TLS 证书配置完成，后续域名只需要配置 Tailscale 的 A 记录或者 Cloudflare Tunnel 的 CNAME 记录即可完成私有服务和公开服务的接入。

external-dns 配置

为了避免手动维护 DNS 记录，通过部署 external-dns 实现 HTTPRoute 与 DNS 的自动同步：
helm repo add external-dns https://kubernetes-sigs.github.io/external-dns/
helm repo update

helm install external-dns external-dns/external-dns \
  -namespace external-dns \
  --create-namespace \
  -f external-dns-values.yaml
domainFilters:
  - okhk.net
env:
  - name: CF_API_TOKEN
    valueFrom:
      secretKeyRef:
        key: apiKey
        name: external-dns
extraArgs:
  - --cloudflare-dns-records-per-page=5000
  - --cloudflare-record-comment=provisioned by external-dns
  - --events
  - --min-event-sync-interval=5s
  - --batch-change-size=200
  - --batch-change-interval=1s
  - --metrics-address=:7979
  - --events-emit=RecordError
interval: 1m
logFormat: json
logLevel: debug
policy: upsert
provider:
  name: cloudflare
registry: txt
secretConfiguration:
  data:
    apiKey: 
  enabled: true
  mountPath: /etc/secrets
sources:
  - gateway-httproute
txtOwnerId: edns
txtPrefix: _externaldns.
  • Tailscale 路由：external-dns 默认解析 Gateway 的外部 IP 并生成 A 记录，正好契合 Tailscale LoadBalancer 的 IP 地址。
  • Cloudflare Tunnel 路由：在 Gateway 资源上添加注解 external-dns.kubernetes.io/target 指向 Tunnel CNAME，并在 HTTPRoute 上添加注解以自动开启 Cloudflare Proxied：
Gateway 中配置 CNAME 记录指向目标：external-dns.kubernetes.io/target 
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  annotations:
    external-dns.kubernetes.io/target: <Tunnel-ID>.cfargotunnel.com
  name: cloudflare-gateway
  namespace: envoy-gateway-system
spec:
  gatewayClassName: eg
  infrastructure:
    parametersRef:
      group: gateway.envoyproxy.io
      kind: EnvoyProxy
      name: cf-proxy-config
  listeners:
    - allowedRoutes:
        namespaces:
          from: All
      name: http
      port: 80
      protocol: HTTP
    - allowedRoutes:
        namespaces:
          from: All
      hostname: "*.okhk.net"
      name: https
      port: 443
      protocol: HTTPS
      tls:
        certificateRefs:
          - kind: Secret
            name: okhk-tls-secret
        mode: Terminate
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: tailscale-gateway
  namespace: envoy-gateway-system
spec:
  gatewayClassName: eg
  listeners:
    - allowedRoutes:
        namespaces:
          from: All
      hostname: "*.okhk.net"
      name: https
      port: 443
      protocol: HTTPS
      tls:
        certificateRefs:
          - kind: Secret
            name: okhk-tls-secret
        mode: Terminate
HTTPRoute 中配置开启 Cloudflare 代理： external-dns.kubernetes.io/cloudflare-proxied
metadata:
  annotations:
    external-dns.kubernetes.io/cloudflare-proxied: "true"

下图中 RemoteAddr10.42.0.28 和 10.42.0.29 分别是两个 Gateway 实例的 Pod IP，可见确实是两个 Gateway 分别转发的。

完成上述架构配置后，日后新增服务只需编写对应的 HTTPRoute 并绑定到不同的 Gateway，即可无缝完成私有服务与公开服务的隔离发布。

记录的有些杂，本来想用 AI 跑一轮优化下行文结构，看了下还是算了，就先到这。