Skip to content

Conversation

@skrobul
Copy link
Collaborator

@skrobul skrobul commented Dec 9, 2025

Summary

This PR migrates UnderStack from the legacy Ingress API with ingress-nginx controller to the Kubernetes Gateway API with Envoy Gateway. This migration is necessary due to the ingress-nginx project retirement with a March 2026 deadline.

Motivation

  • ingress-nginx is being retired by the Kubernetes community
  • Gateway API is the modern, standardized approach for ingress traffic management
  • Better separation of concerns between cluster operators (Gateway) and application developers (Routes)
  • More expressive routing capabilities and native support for multiple protocols

Changes

Infrastructure Components

Envoy Gateway Deployment (components/envoy-gateway/)

  • New component for Envoy Gateway controller
  • Configured with Backend API and EnvoyPatchPolicy support
  • Deployed via ArgoCD in apps/infra/envoy-gateway.yaml

envoy-configs Helm Chart (components/envoy-configs/)

  • Centralized configuration for Gateway and Route resources
  • Templates for Gateway (external/internal), HTTPRoute, and TLSRoute
  • JSON schema validation for configuration
  • Supports TLS termination, passthrough, and HTTPS backends
  • Cross-namespace route references with label selectors

cert-manager Updates

  • Enabled Gateway API support via Helm installation
  • Bootstrap script now installs cert-manager with enableGatewayApi: true
  • Removed kustomize-based installation in favor of Helm

OpenStack Helm Charts

Updated all OpenStack service components to disable Ingress generation:

  • network.use_external_ingress_controller: true
  • manifests.ingress_api: false
  • manifests.service_ingress_api: false

Application Updates

ArgoCD, Argo Workflows, Dex

  • Removed standalone Ingress manifests
  • Routing now managed via envoy-configs

external-dns

  • Upgraded to support Gateway API resources

Architecture

The new architecture uses:

  • Shared Gateways: Two cluster-wide Gateways (external, internal) in envoy-gateway namespace
  • Distributed Routes: HTTPRoute/TLSRoute resources in application namespaces
  • Centralized Configuration: All routes defined in site-specific envoy-configs.yaml
  • Automatic TLS: cert-manager provisions certificates at Gateway level

Documentation

Migration Path for Deployments

Existing deployments need to update their deploy/<site>/ repository:

  1. Add skip: true for ingress-nginx in apps.yaml
  2. Create GatewayClass resource in manifests/envoy-gateway/
  3. Create helm-configs/envoy-configs.yaml with Gateway and route definitions
  4. Remove old Ingress manifests
  5. Disable Ingress in Helm charts (monitoring, etc.)
  6. Update Cilium L2 policies to match Envoy Gateway
  7. Update load balancer IP sharing annotations
  8. Manually remove old ingress-nginx namespace if it blocks Service IP assignment

See docs/operator-guide/gateway-api.md for detailed instructions.

Breaking Changes

⚠️ This is a breaking change for existing deployments

Deployments must update their site-specific configuration to:

  • Define routes and gateways iin helm-configs/envoy-configs.yaml
  • Remove old Ingress resources
  • Potentially manually clean up ingress-nginx namespace

Related Links

@skrobul skrobul requested a review from a team December 9, 2025 12:05
@skrobul skrobul changed the title Migrate from ingress-nginx to Kubernetes Gateway API feat: Migrate from ingress-nginx to Kubernetes Gateway API Dec 9, 2025
We no longer do "static" install of the cert-manager, because custom
configuration needs to be provided to enable the Gateway API
functionality.
Includes cert-manager integration.
For HTTPRoute, if the name is not explicitly set, default to first part
of the hostname/FQDN.
Technically, this gives user a way to configure which routes is the
listener going to accept.
Some of the services require passthrough TLS instead of termination.
This adds ability to direct traffic over https without doing cert
verification.
So that we can disable TLS validation...
These have been replaced by HTTPRoutes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants