Implement e2e test for ingress2gateway

[rikatz]

What would you like to be added:
Implement e2e test for ingress2gateway

We don't have as of today any sort of e2e that can get some Ingress resource and turn into a GatewayAPI resource, to later apply on a cluster and test its behavior.

This can cause situations where an implementation is not correct, or doesn't reflect the real user experience

As a proposal, we can implement some e2e tests that:

• Bootstrap a KinD cluster
• Install some GatewayAPI controller - Suggestion is to use envoy gateway
• Has a set of tests for the desired providers that represent an ingress resource. As an example, we can have yaml manifests that represents Ingress NGINX with its features (server-alias, canary, etc).
• Run ingress2gateway on these files, convert to GatewayAPI manifests and apply to cluster
• Run a simple "curl" to guarantee that the expected behavior is working

As this is a cli, we want to reproduce the user behavior, so my suggestion is to use bats for these tests.

[rikatz]

/kind test

[k8s-ci-robot]

@rikatz: The label(s) kind/test cannot be applied, because the repository doesn't have them.

In response to this:

/kind test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[carmal891]

@rikatz Hi,
I’ve gone through the initial e2e structure you’ve set up. I noticed that the mentioned KinD cluster bootstrap and the pre_check() function still need some work. I'd be happy to collaborate and help complete these pending parts. My plan is to use your PR as a base and contribute additional changes in a separate commit.

Please let me know if this is fine by you and if there’s anything else you’d like me to focus on regarding this ticket.

[rikatz]

@carmal891 totally fine, go for it. I am not sure we do have an agreement on my structure, but I think that using bats reflects the best user experience for e2e tests (we are fully reproducing what users do on kubectl, not what a program would do using go libraries)

One note: I wanted to discuss with @LiorLieberman first (and you are welcome to contribute!) about #244

I think part of the experience that needs to be improved on i2g is the fact that we assume that a Gateway will need to be created, and that we want to attach routes to it, while we may have already a Gateway or other parent (like a ListenerSet) and just want to create routes that attaches to it.

But for now, as we don't have any e2e test I would love to see something, so feel free to contribute over my PR, opening yours :)

[carmal891]

@rikatz Apologies for the delay from my side. I got caught up with other project priorities. I ran the initial e2e structure that you set up using Bats but I encountered an issue

When executing the tests, I hit the following error:

carmalshaktan@Carmals-MacBook-Pro` ingress2gateway % make e2e

I2G=/Users/carmalshaktan/repos/k8s-sigs/ingress2gateway/ingress2gateway hack/verify-e2e.sh
Client Version: v1.31.1
Kustomize Version: v5.4.2
Server Version: v1.30.0
   LOG: Installing envoy gateway
   LOG: Creating GatewayClass
   LOG: Creating Gateway
   LOG: Starting port-forwarding to gateway
   LOG: Setting the sample application
   LOG: Test is ready to roll
e2e_simple.bats
 ✗ Simple Ingress manifest test
   LOG: Running i2g on e2e/manifests/simple/simple.yaml
   (from function `do_request' in file e2e/tests/functions.bash, line 15,
    in test file e2e/tests/e2e_simple.bats, line 9)
     `do_request' failed
   apiVersion: gateway.networking.k8s.io/v1
   kind: Gateway
   metadata:
     annotations:
       gateway.networking.k8s.io/generator: ingress2gateway-4ceeccb-dirty
     name: nginx
   spec:
     gatewayClassName: nginx
     listeners:
     - hostname: some.test
       name: some-test-http
       port: 80
       protocol: HTTP
   status: {}
   ---
   apiVersion: gateway.networking.k8s.io/v1
   kind: HTTPRoute
   metadata:
     annotations:
       gateway.networking.k8s.io/generator: ingress2gateway-4ceeccb-dirty
     name: simple-some-test
   spec:
     hostnames:
     - some.test
     parentRefs:
     - name: nginx
     rules:
     - backendRefs:
       - name: backend
         port: 3000
       matches:
       - path:
           type: Exact
           value: /
   status:
     parents: []
   Error from server (NotFound): httproutes.gateway.networking.k8s.io "simple-some-test" not found
   LOG: Killed port-forwarding
   LOG: Removing the sample application
   LOG: Cleaning gateway and gatewayclass resources

1 test, 1 failure

make: *** [e2e] Error

Observation

While the generated HTTPRoute exists in YAML, its status.parents field is empty, so I assume it may not have attached to the Gateway yet. I also tried running kubectl get httproute simple-some-test before the curl step, and it returns NotFound.

Could you please advise on how to ensure the HTTPRoute attaches properly so that the e2e test can proceed?

[Stevenjin8]

Ok, with #265, I think doing the tests in bash is not a good idea. Here is my idea of e2e tests

The integration tests assume a running cluster exists (we can pass k--ubeconfig/--context if needed), a load balancer is available (MetalLB, cloud-provider-kind.), and ideally all Gateway API implementations are installed. I’m not 100% sure about the last one yet.

Each test case describes:

• An Ingress resource, plus any extra resources.
• The flags we’ll pass into ingress2gateway.
• Dummy services/backends so there’s something for the ingresses/gateways to route to.
• An ingress provider, with an Deploy function that:
  1. Deploys the Ingress and waits for it to be ready.
  2. Returns a public IP of the Ingress.
  3. Provides a cleanup hook.
• A list of Gateway emitters, each with the same interface as the ingress provider.
• A verify function that takes an IP and checks behavior (likely sends HTTP requests).

The test case will run as follows

1. Deploy the dummy services/backends.
2. Deploy the Ingress and its supporting resources.
3. Run the verify function with the
4. For each Gateway implementation:
   1. Run ingress2gateway with the input flags and corresponding emitter
   2. Deploy the gwapi resources
   3. Run the same verification logic
5. Clean up

Some other things to keep in mind.

• It should be easy to do a kind cluster setup, run integration tests, teardown in a single command for CI
• We should have an easy way to setup a kind cluster with metallb
• It should be easy to run tests for only one provider and/or emitter
• We should provide ways of skipping setup/cleanup so its easy to iterate on the tests themselves

[johananl]

Hey people! I'm interested in this effort and would love to help move the e2e tests forward. I hope it's OK that I chime in out of the blue. Happy to join any relevant meetings and introduce myself properly 🙂

Looks like there is already initial work on this and I don't want to step on anybody's toes. @carmal891 are you actively working on this at the moment? If so, care to share the current status of the WIP?

Regarding implementing the tests in Bash, my two cents:

It certainly makes sense to replicate the user behavior as closely as possible so in that regard using Bash to execute the CLI is very reasonable.

However, I'm counting 9 GWAPI implementations under pkg/i2gw/providers and I assume there will be more in the future. The testing setup discussed here will have to wait for resources to converge in the k8s API, send network requests (with potential timeouts) and clean things up for every test case for every provider. Doing all of this sequentially will likely be very slow with a large number of test cases, and doing this in parallel (asynchronously) using Bash can be a mess, especially when trying to hunt down test failures.

Go has the testing package which gives us the following for free:

• Parallel and isolated test execution using t.Parallel()
• The ability to organize tests into a hierarchy and execute subsets using go test -run
• Easier debugging thanks to Delve

Given that the bulk of the tricky logic sits in Go libraries under pkg/i2gw, AFAICT the only extra things Bash testing would add are whether Cobra works and whether stdin/stdout/stderr works. The former is a Go framework that's not a part of ingress2gateway and the latter is generic Linux behavior. IMHO we can simply assume these two layers work, or we could have 1-3 smoke tests in Bash just to ensure correct CLI behavior and still do the bulk of the tricky testing bits in Go.

To conclude, IMO sacrificing easy parallelism, test organization and debugging to include a very thin extra layer of Go code isn't a great trade off.

Happy to hear any thoughts! 🙏

[johananl]

AFAICT the only extra things Bash testing would add are whether Cobra works and whether stdin/stdout/stderr works

On second look I see that there is quite a bit of logic in cmd/print.go. I'd say that's still not a strong argument against Go tests because IMO we can simply separate constructing strings from printing them, thus keeping the cmd package very lean which is probably a good idea anyway.

[carmal891]

@johananl Hi @johananl , Please feel free to contribute here. I was testing the trying to enhance the initial e2e Initial e2e structure #240 but hit some issues. I have not assigned it to myself for the same reason