我正在使用GKE（版本1.21.11-gke.1100）测试Gateway API。我正在使用gatewayClassName: gke-l7-rilb作为客户端和网关之间的TLS网关。HTTPS在客户端和负载均衡器之间使用托管区域SSL证书完美工作。
我有2个Httproutes引用2个kube服务（backendRefs）。一个服务可以通过HTTP访问，另一个可以通过HTTPS访问（来自Argo Workflows项目的argo服务器服务，如果可能的话）。
当我使用HTTP创建引用服务的httproute时，GCP负载均衡器后端服务被创建并且工作没有任何问题（健康）。
但是，当我创建引用argo-service的httproute时，创建了一个GCP负载均衡器后端服务，但在端点协议设置为HTTP而不是HTTPS的情况下无法工作（不健康）。您应该知道，我确保向argo-server服务添加了注解cloud.google.com/app-protocols: '{"web":"HTTPS"}'，以在负载均衡器和argo-server应用程序之间启用HTTPS。
如果我使用ingress资源和相同的argo服务定义创建相同的geatway API配置，端点协议（GCP负载均衡器后端服务的）被正确设置为HTTPS，并且非常健康和工作。
就像网关API的httproute一样，GKE网关控制器没有考虑cloud.google.com/app-protocols服务注解，尽管这里提到它与网关API相关。
编辑1：添加yaml文件

网关：

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: Gateway
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"gateway.networking.k8s.io/v1alpha2","kind":"Gateway","metadata":{"annotations":{},"labels":{"app.kubernetes.io/managed-by":"gcp-cloud-build-deploy"},"name":"regional-internal-https","namespace":"exposition"},"spec":{"addresses":[{"type":"NamedAddress","value":"dev-gateway-internal-lb-static-ip"}],"gatewayClassName":"gke-l7-rilb","listeners":[{"allowedRoutes":{"kinds":[{"kind":"HTTPRoute"}],"namespaces":{"from":"Selector","selector":{"matchLabels":{"exposed":"true"}}}},"name":"https","port":443,"protocol":"HTTPS","tls":{"mode":"Terminate","options":{"networking.gke.io/pre-shared-certs":"plat-dev-europe-west1"}}}]}}
    networking.gke.io/addresses: ""
    networking.gke.io/backend-services: gkegw1-bkib-argo-argo-server-2746-8ktcvo8d0ktp,
      gkegw1-bkib-demo-application-demo-service-80-y5bgcnm71kjv, gkegw1-bkib-exposition-gw-serve404-80-pciznuyt569p
    networking.gke.io/firewalls: ""
    networking.gke.io/forwarding-rules: gkegw1-bkib-exposition-regional-internal-https-tqsh4njw7io8
    networking.gke.io/health-checks: gkegw1-bkib-argo-argo-server-2746-8ktcvo8d0ktp,
      gkegw1-bkib-demo-application-demo-service-80-y5bgcnm71kjv, gkegw1-bkib-exposition-gw-serve404-80-pciznuyt569p
    networking.gke.io/last-reconcile-time: "2022-06-16T15:57:45Z"
    networking.gke.io/ssl-certificates: ""
    networking.gke.io/target-proxies: gkegw1-bkib-exposition-regional-internal-https-tqsh4njw7io8
    networking.gke.io/url-maps: gkegw1-bkib-exposition-regional-internal-https-tqsh4njw7io8
  creationTimestamp: "2022-06-15T08:28:20Z"
  finalizers:
  - gateway.finalizer.networking.gke.io
  generation: 1
  labels:
    app.kubernetes.io/managed-by: gcp-cloud-build-deploy
  managedFields:
  - apiVersion: gateway.networking.k8s.io/v1alpha2
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:kubectl.kubernetes.io/last-applied-configuration: {}
        f:labels:
          .: {}
          f:app.kubernetes.io/managed-by: {}
      f:spec:
        .: {}
        f:addresses: {}
        f:gatewayClassName: {}
        f:listeners:
          .: {}
          k:{"name":"https"}:
            .: {}
            f:allowedRoutes:
              .: {}
              f:kinds: {}
              f:namespaces:
                .: {}
                f:from: {}
                f:selector:
                  .: {}
                  f:matchLabels:
                    .: {}
                    f:exposed: {}
            f:name: {}
            f:port: {}
            f:protocol: {}
            f:tls:
              .: {}
              f:mode: {}
              f:options:
                .: {}
                f:networking.gke.io/pre-shared-certs: {}
    manager: kubectl-client-side-apply
    operation: Update
    time: "2022-06-15T08:28:20Z"
  - apiVersion: gateway.networking.k8s.io/v1alpha2
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:networking.gke.io/addresses: {}
          f:networking.gke.io/backend-services: {}
          f:networking.gke.io/firewalls: {}
          f:networking.gke.io/forwarding-rules: {}
          f:networking.gke.io/health-checks: {}
          f:networking.gke.io/last-reconcile-time: {}
          f:networking.gke.io/ssl-certificates: {}
          f:networking.gke.io/target-proxies: {}
          f:networking.gke.io/url-maps: {}
        f:finalizers:
          .: {}
          v:"gateway.finalizer.networking.gke.io": {}
      f:status:
        f:addresses: {}
    manager: GoogleGKEGatewayController
    operation: Update
    time: "2022-06-15T08:30:16Z"
  name: regional-internal-https
  namespace: exposition
  resourceVersion: "42337844"
  uid: 59333aea-1a79-4e9b-afbc-595ae9ccdfd7
spec:
  addresses:
  - type: NamedAddress
    value: dev-gateway-internal-lb-static-ip
  gatewayClassName: gke-l7-rilb
  listeners:
  - allowedRoutes:
      kinds:
      - group: gateway.networking.k8s.io
        kind: HTTPRoute
      namespaces:
        from: Selector
        selector:
          matchLabels:
            exposed: "true"
    name: https
    port: 443
    protocol: HTTPS
    tls:
      mode: Terminate
      options:
        networking.gke.io/pre-shared-certs: plat-dev-europe-west1
status:
  addresses:
  - type: IPAddress
    value: 10.163.112.28
  conditions:
  - lastTransitionTime: "1970-01-01T00:00:00Z"
    message: Waiting for controller
    reason: NotReconciled
    status: Unknown
    type: Scheduled

字符串

Httproute：

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"gateway.networking.k8s.io/v1alpha2","kind":"HTTPRoute","metadata":{"annotations":{},"labels":{"app.kubernetes.io/managed-by":"gcp-cloud-build-deploy"},"name":"argo-server","namespace":"argo"},"spec":{"hostnames":["argo-server.plat.dev.df.gcp.corp.modified.com"],"parentRefs":[{"kind":"Gateway","name":"regional-internal-https","namespace":"exposition"}],"rules":[{"backendRefs":[{"name":"argo-server","port":2746}]}]}}
  creationTimestamp: "2022-06-15T12:27:04Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: gcp-cloud-build-deploy
  managedFields:
    - apiVersion: gateway.networking.k8s.io/v1alpha2
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:kubectl.kubernetes.io/last-applied-configuration: {}
          f:labels:
            .: {}
            f:app.kubernetes.io/managed-by: {}
        f:spec:
          .: {}
          f:hostnames: {}
          f:parentRefs: {}
          f:rules: {}
      manager: kubectl-client-side-apply
      operation: Update
      time: "2022-06-15T12:27:04Z"
    - apiVersion: gateway.networking.k8s.io/v1alpha2
      fieldsType: FieldsV1
      fieldsV1:
        f:status:
          .: {}
          f:parents: {}
      manager: GoogleGKEGatewayController
      operation: Update
      time: "2022-06-15T12:29:02Z"
  name: argo-server
  namespace: argo
  resourceVersion: "42362026"
  uid: 981ce997-c574-4878-bec1-b03c7707838c
spec:
  hostnames:
    - argo-server.plat.dev.df.gcp.corp.modified.com
  parentRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: regional-internal-https
      namespace: exposition
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: argo-server
          port: 2746
          weight: 1
      matches:
        - path:
            type: PathPrefix
            value: /
status:
  parents:
    - conditions:
        - lastTransitionTime: "2022-06-16T17:00:11Z"
          message: ""
          reason: RouteAccepted
          status: "True"
          type: Accepted
        - lastTransitionTime: "2022-06-16T17:00:11Z"
          message: ""
          reason: ReconciliationSucceeded
          status: "True"
          type: Reconciled
      controllerName: networking.gke.io/gateway
      parentRef:
        group: gateway.networking.k8s.io
        kind: Gateway
        name: regional-internal-https
        namespace: exposition

型

售后服务：

apiVersion: v1
kind: Service
metadata:
  annotations:
    cloud.google.com/app-protocols: '{"web":"HTTPS"}'
    cloud.google.com/backend-config: '{"default": "argo-server-backendconfig"}'
    cloud.google.com/neg: '{"exposed_ports":{"2746":{}}}'
    cloud.google.com/neg-status: '{"network_endpoint_groups":{"2746":"k8s1-f83345f9-argo-argo-server-2746-4d39c835"},"zones":["europe-west1-c"]}'
    cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/app-protocols":"{\"web\":\"HTTPS\"}","cloud.google.com/backend-config":"{\"default\": \"argo-server-backendconfig\"}","cloud.google.com/neg":"{\"ingress\": true}","cluster-autoscaler.kubernetes.io/safe-to-evict":"true"},"labels":{"app.kubernetes.io/managed-by":"gcp-cloud-build-deploy"},"name":"argo-server","namespace":"argo"},"spec":{"ports":[{"name":"web","port":2746,"targetPort":2746}],"selector":{"app":"argo-server"}}}
  creationTimestamp: "2022-06-15T11:44:07Z"
  labels:
    app.kubernetes.io/managed-by: gcp-cloud-build-deploy
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:cloud.google.com/app-protocols: {}
          f:cloud.google.com/backend-config: {}
          f:cluster-autoscaler.kubernetes.io/safe-to-evict: {}
          f:kubectl.kubernetes.io/last-applied-configuration: {}
        f:labels:
          .: {}
          f:app.kubernetes.io/managed-by: {}
      f:spec:
        f:ports:
          .: {}
          k:{"port":2746,"protocol":"TCP"}:
            .: {}
            f:name: {}
            f:port: {}
            f:protocol: {}
            f:targetPort: {}
        f:selector:
          .: {}
          f:app: {}
        f:sessionAffinity: {}
        f:type: {}
    manager: kubectl-client-side-apply
    operation: Update
    time: "2022-06-15T12:27:23Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:cloud.google.com/neg: {}
    manager: GoogleGKEGatewayController
    operation: Update
    time: "2022-06-15T12:28:06Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:cloud.google.com/neg-status: {}
    manager: glbc
    operation: Update
    time: "2022-06-15T12:28:06Z"
  name: argo-server
  namespace: argo
  resourceVersion: "41692832"
  uid: 25024d53-1d31-4165-8033-1843ec5d72ec
spec:
  clusterIP: 10.163.247.121
  clusterIPs:
  - 10.163.247.121
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: web
    port: 2746
    protocol: TCP
    targetPort: 2746
  selector:
    app: argo-server
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

型

2条答案

按热度按时间

krugob8w1#

我找到了一个解决方案，我认为这是一个变通办法。
1.使用networking.gke.io/app-protocols: '{"web":"HTTPS"}' annotation而不是cloud.google.com/app-protocols: '{"web":"HTTPS"}'。此annotation将在服务级别使用，其中web是端口的名称。这将在负载均衡器和应用程序之间启用HTTPS （为指定的HTTP路由创建的后端服务的端点协议）。这与gatewayClassName: gke-l7-rilb区域内部负载均衡器完美配合。
1.使用cloud.google.com/v1 BackendConfig创建自定义健康检查，其中将类型设置为HTTPS，将端口设置为2746。更多详细信息请参见https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features#direct_healthWith入口，GCE入口控制器会从应用程序就绪探测器自动创建此健康检查，但显然GKE网关控制器尚未实现此功能。*
1.确保您有防火墙规则允许入口流量在2746端口上进行Google Cloud健康检查 * 通过入口，GCE入口控制器会自动创建所需的防火墙规则，但显然GKE网关控制器中尚未实现此功能。*
最后，我说这是一个解决方案，因为我想象并希望GKE网关控制器的未来版本将修复我上面提到的3个问题或点。

赞(0）回复(0）举报 5个月前

m0rkklqb2#

对我来说也是同样的问题：如何在GKE集群中使用Kuberenetes资源HTTPRoute（使用gatewayClass：gke-l7-gxlb）实现负载均衡和应用/k8s-pod之间的HTTPS通信：
解决方案：需要在提到的kubernetesservice中设置字段“appProtocol：HTTPS”：

apiVersion: v1
kind: Service
metadata:
name: your-service
labels:
    app.kubernetes.io/name: your-service
spec:
type: ClusterIP
ports:
   - name: port-https
     port: 443
     targetPort: 8080
     protocol: TCP
     appProtocol: HTTPS
 selector:
    app.kubernetes.io/name: your-app

字符串
之后，您可以通过后端服务在GCP控制台/负载均衡详细信息中看到“端点协议”为“HTTPS”：
x1c 0d1x的数据

kubernetes GKE网关API httproute不适用于负载均衡器和应用程序之间的https

2条答案

相关问题

热门标签

最新问答