feat: improve e2e testing tooling and documentation

Major improvements to the end-to-end testing infrastructure:

Testing Infrastructure:
- Add automatic payment method ID fetching from RedisCloud API
- Create templated test examples with dynamic payment method injection
- Add cluster/test/setup.sh script for comprehensive test environment setup
- Add datasource.yaml for dynamic value injection in tests
- Support environment variable configuration via .envrc

Documentation Updates:
- Restructure TESTING.md with clear quick-start instructions
- Add comprehensive payment method configuration section
- Document environment variable setup with .envrc.example
- Update CI-TESTING.md to reflect current CI/CD workflows
- Clean up references to removed scripts in all docs

Cleanup:
- Remove obsolete scripts (quick-start.sh, setup-podman.sh, test-provider.sh)
- Consolidate functionality into Makefile targets
- Add proper .gitignore entries for test artifacts
- Remove untracked files (provider-aws/, kubeconfig, etc.)

Developer Experience:
- Add devenv.nix configuration for development environment
- Add rediscloud-payment-methods and rediscloud-first-payment-method-id make targets
- Improve error handling and user feedback in setup scripts
- Simplify credential management with environment variables

This refactoring makes the e2e testing process more robust and user-friendly,
particularly for testing Pro subscription features that require payment methods.

Author: rawkode

.claude/settings.local.json

@@ -1,3 +1,5 @@
+source_env_if_exists .envrc.local
+
 export DIRENV_WARN_TIMEOUT=20s
 
 if has devenv; then
@@ -7,16 +9,14 @@ fi
 
 # RedisCloud API Credentials
 # Get these from: RedisCloud Console → Account Settings → API Keys
-export REDISCLOUD_API_KEY="your-api-key-here"
-export REDISCLOUD_SECRET_KEY="your-secret-key-here"
+#export REDISCLOUD_API_KEY="YOUR_API_KEY_HERE"
+#export REDISCLOUD_SECRET_KEY="YOUR_SECRET_KEY_HERE"
 export REDISCLOUD_URL="https://api.redislabs.com/v1"
 
 # Kubernetes/Crossplane Configuration
 export CROSSPLANE_NAMESPACE="crossplane-system"
 export PROVIDER_NAMESPACE="crossplane-system"
 
 # Development Settings
-export DOCKER_HOST="unix:///run/user/$(id -u)/podman/podman.sock"
 export KIND_CLUSTER_NAME="rediscloud-test"
-export KIND_EXPERIMENTAL_PROVIDER=podman                                                                             │ │
 export KUBECONFIG_PATH="${PWD}/.kube/config"

.envrc

@@ -0,0 +1,20 @@
+# RedisCloud API Credentials
+# Get these from: RedisCloud Console → Account Settings → API Keys
+export REDISCLOUD_API_KEY="YOUR_API_KEY_HERE"
+export REDISCLOUD_SECRET_KEY="YOUR_SECRET_KEY_HERE"
+export REDISCLOUD_URL="https://api.redislabs.com/v1"
+
+# Optional: Set a specific payment method ID (otherwise will fetch the first one)
+# export REDISCLOUD_PAYMENT_METHOD_ID="12345"
+
+# Kubernetes/Crossplane Configuration
+export CROSSPLANE_NAMESPACE="crossplane-system"
+export PROVIDER_NAMESPACE="crossplane-system"
+
+# Development Settings
+export KIND_CLUSTER_NAME="rediscloud-test"
+export KUBECONFIG_PATH="${PWD}/.kube/config"
+
+# For 1Password users, you can override in .envrc.local:
+# export REDISCLOUD_API_KEY="op://Private/Redis Cloud/api-tokens/api-account-key"
+# export REDISCLOUD_SECRET_KEY="op://Private/Redis Cloud/api-tokens/api-user-key"

.envrc.example

@@ -290,6 +290,62 @@ jobs:
         run: |
           ./scripts/check-examples.py package/crds examples
 
+  e2e-tests:
+    runs-on: ubuntu-24.04
+    needs: detect-noop
+    if: ${{ needs.detect-noop.outputs.noop != 'true' }}
+
+    steps:
+      - name: Setup QEMU
+        uses: docker/setup-qemu-action@49b3bc8e6bdd4a60e6116a5414239cba5943d3cf # v3
+        with:
+          platforms: all
+
+      - name: Setup Docker Buildx
+        uses: docker/setup-buildx-action@v3
+        with:
+          version: ${{ env.DOCKER_BUILDX_VERSION }}
+          install: true
+
+      - name: Checkout
+        uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4
+        with:
+          submodules: true
+
+      - name: Setup Go
+        uses: actions/setup-go@0a12ed9d6a96ab950c8f026ed9f722fe0da7ef32 # v5
+        with:
+          go-version: ${{ env.GO_VERSION }}
+
+      - name: Find the Go Build Cache
+        id: go
+        run: echo "cache=$(make go.cachedir)" >> $GITHUB_OUTPUT
+
+      - name: Cache the Go Build Cache
+        uses: actions/cache@v4
+        with:
+          path: ${{ steps.go.outputs.cache }}
+          key: ${{ runner.os }}-build-e2e-tests-${{ hashFiles('**/go.sum') }}
+          restore-keys: ${{ runner.os }}-build-e2e-tests-
+
+      - name: Cache Go Dependencies
+        uses: actions/cache@v4
+        with:
+          path: .work/pkg
+          key: ${{ runner.os }}-pkg-${{ hashFiles('**/go.sum') }}
+          restore-keys: ${{ runner.os }}-pkg-
+
+      - name: Vendor Dependencies
+        run: make vendor vendor.check
+
+      - name: Build Helm Chart
+        run: make -j2 build.all
+        env:
+          BUILD_ARGS: "--load"
+
+      - name: Run E2E Tests
+        run: make e2e USE_HELM3=true
+
   publish-artifacts:
     runs-on: ubuntu-24.04
     needs:
@@ -300,6 +356,7 @@ jobs:
       - unit-tests
       - local-deploy
       - check-examples
+      - e2e-tests
     if: needs.detect-noop.outputs.noop != 'true'
 
     steps:

.github/workflows/ci.yml

@@ -9,6 +9,10 @@ cover.out
 # ignore IDE folders
 .vscode/
 .idea/
+.claude/
+
+# Secrets
+.envrc.local
 
 # Devenv
 .devenv*
@@ -22,3 +26,6 @@ devenv.local.nix
 
 # Testing
 .kube/
+kubeconfig
+cluster/test/processed-examples/
+cluster/test/datasource.yaml

.gitignore

@@ -27,8 +27,46 @@ See the [README.md](README.md#developing) for development setup instructions.
 Before submitting a pull request:
 
 1. Run the full test suite: `make test`
-2. Test your changes locally: `./scripts/test-provider.sh`
-3. Ensure all CRDs generate properly: `make generate`
+2. Test your changes locally: `make local-deploy`
+3. Run end-to-end tests: `make e2e`
+4. Ensure all CRDs generate properly: `make generate`
+5. Run all pre-submit checks: `make reviewable`
+
+### Testing with RedisCloud Pro Subscriptions
+
+For testing Pro subscription features that require payment methods:
+
+```bash
+# Set RedisCloud API credentials
+export REDISCLOUD_API_KEY="your-api-key"
+export REDISCLOUD_SECRET_KEY="your-secret-key"
+export REDISCLOUD_URL="https://api.redislabs.com/v1"
+
+# List available payment methods
+make rediscloud-payment-methods
+
+# Get first payment method ID
+make rediscloud-first-payment-method-id
+
+# Run e2e tests with automatic payment method fetching
+make e2e
+
+# Or with 1Password integration
+op run -- make e2e
+```
+
+### Useful Make Targets
+
+```bash
+make help                                 # Show all available targets
+make test                                 # Run unit tests
+make e2e                                  # Run end-to-end tests
+make local-deploy                         # Deploy provider to local Kind cluster
+make run                                  # Run provider locally (out of cluster)
+make reviewable                           # Run all checks before submitting PR
+make rediscloud-payment-methods          # List available payment methods from RedisCloud
+make rediscloud-first-payment-method-id  # Get first payment method ID
+```
 
 ## Submitting Changes
 

CONTRIBUTING.md

@@ -175,12 +175,13 @@ run: go.build
 # ====================================================================================
 # End to End Testing
 CROSSPLANE_VERSION = 1.16.0
-CROSSPLANE_NAMESPACE = upbound-system
+CROSSPLANE_NAMESPACE ?= crossplane-system
 -include build/makelib/local.xpkg.mk
 -include build/makelib/controlplane.mk
 
 # This target requires the following environment variables to be set:
 # - UPTEST_EXAMPLE_LIST, a comma-separated list of examples to test
+#   Default: examples/rediscloud/subscription.yaml
 #   To ensure the proper functioning of the end-to-end test resource pre-deletion hook, it is crucial to arrange your resources appropriately.
 #   You can check the basic implementation here: https://github.com/crossplane/uptest/blob/main/internal/templates/03-delete.yaml.tmpl.
 # - UPTEST_CLOUD_CREDENTIALS (optional), multiple sets of AWS IAM User credentials specified as key=value pairs.
@@ -193,15 +194,31 @@ CROSSPLANE_NAMESPACE = upbound-system
 #   aws_secret_access_key = REDACTED'
 #   The associated `ProviderConfig`s will be named as `default` and `peer`.
 # - UPTEST_DATASOURCE_PATH (optional), please see https://github.com/crossplane/uptest#injecting-dynamic-values-and-datasource
+# - REDISCLOUD_PAYMENT_METHOD_ID (optional), the payment method ID to use for subscription examples
+# - REDISCLOUD_API_KEY, REDISCLOUD_SECRET_KEY, REDISCLOUD_URL - RedisCloud API credentials to fetch payment method ID
+UPTEST_EXAMPLE_LIST ?= examples/rediscloud/subscription.yaml
 uptest: $(UPTEST) $(KUBECTL) $(KUTTL)
 	@$(INFO) running automated tests
-	@KUBECTL=$(KUBECTL) KUTTL=$(KUTTL) $(UPTEST) e2e "${UPTEST_EXAMPLE_LIST}" --data-source="${UPTEST_DATASOURCE_PATH}" --setup-script=cluster/test/setup.sh --default-conditions="Test" || $(FAIL)
+	@# Create datasource.yaml file before running uptest (will be updated by setup script)
+	@mkdir -p cluster/test
+	@echo "# Placeholder datasource file - will be populated by setup script" > cluster/test/datasource.yaml
+	@echo "payment_method_id: \"\"" >> cluster/test/datasource.yaml
+	@echo "api_key: \"\"" >> cluster/test/datasource.yaml
+	@echo "secret_key: \"\"" >> cluster/test/datasource.yaml
+	@echo "url: \"\"" >> cluster/test/datasource.yaml
+	@KUBECTL=$(KUBECTL) KUTTL=$(KUTTL) \
+		REDISCLOUD_API_KEY=$${REDISCLOUD_API_KEY:-} \
+		REDISCLOUD_SECRET_KEY=$${REDISCLOUD_SECRET_KEY:-} \
+		REDISCLOUD_URL=$${REDISCLOUD_URL:-} \
+		REDISCLOUD_PAYMENT_METHOD_ID=$${REDISCLOUD_PAYMENT_METHOD_ID:-} \
+		UPTEST_DATASOURCE_PATH="cluster/test/datasource.yaml" \
+		$(UPTEST) e2e "$(UPTEST_EXAMPLE_LIST)" --data-source="cluster/test/datasource.yaml" --setup-script=cluster/test/setup.sh --default-conditions="Test" || $(FAIL)
 	@$(OK) running automated tests
 
 local-deploy: build controlplane.up local.xpkg.deploy.provider.$(PROJECT_NAME)
 	@$(INFO) running locally built provider
 	@$(KUBECTL) wait provider.pkg $(PROJECT_NAME) --for condition=Healthy --timeout 5m
-	@$(KUBECTL) -n upbound-system wait --for=condition=Available deployment --all --timeout=5m
+	@$(KUBECTL) -n $(CROSSPLANE_NAMESPACE) wait --for=condition=Available deployment --all --timeout=5m
 	@$(OK) running locally built provider
 
 e2e: local-deploy uptest
@@ -261,6 +278,40 @@ help-special: crossplane.help
 
 .PHONY: crossplane.help help-special
 
+# ====================================================================================
+# RedisCloud API Testing
+
+# Fetch payment methods from RedisCloud API
+.PHONY: rediscloud-payment-methods
+rediscloud-payment-methods:
+	@$(INFO) Fetching RedisCloud payment methods
+	@curl -s -X GET "$${REDISCLOUD_URL}/payment-methods" \
+		-H "x-api-key: $${REDISCLOUD_API_KEY}" \
+		-H "x-api-secret-key: $${REDISCLOUD_SECRET_KEY}" | jq '.' || $(FAIL)
+	@$(OK) Fetched RedisCloud payment methods
+
+# Get the first payment method ID from RedisCloud API
+.PHONY: rediscloud-first-payment-method-id
+rediscloud-first-payment-method-id:
+	@curl -s -X GET "$${REDISCLOUD_URL}/payment-methods" \
+		-H "x-api-key: $${REDISCLOUD_API_KEY}" \
+		-H "x-api-secret-key: $${REDISCLOUD_SECRET_KEY}" | jq -r '.paymentMethods[0].id // empty'
+
+# Run e2e tests with processed examples
+.PHONY: e2e-with-processed-examples
+e2e-with-processed-examples:
+	@$(INFO) Running e2e tests with processed examples
+	@# The setup.sh script will handle fetching payment method ID and processing templates
+	@# Use processed examples if they exist, otherwise fall back to original examples
+	@if [ -d "cluster/test/processed-examples" ] && [ "$$(ls -A cluster/test/processed-examples/*.yaml 2>/dev/null | wc -l)" -gt 0 ]; then \
+		EXAMPLE_LIST=$$(ls cluster/test/processed-examples/*.yaml | tr '\n' ',' | sed 's/,$$//'); \
+		echo "Using processed examples: $${EXAMPLE_LIST}"; \
+		$(MAKE) uptest UPTEST_EXAMPLE_LIST="$${EXAMPLE_LIST}"; \
+	else \
+		echo "Using default examples"; \
+		$(MAKE) uptest; \
+	fi
+
 # TODO(negz): Update CI to use these targets.
 vendor: modules.download
 vendor.check: modules.check

Makefile

@@ -120,10 +120,37 @@ make build
 
 ### Testing
 
-Run the test harness (uses podman by default):
+Run unit tests:
 
 ```bash
-./scripts/test-provider.sh
+make test
+```
+
+Run end-to-end tests:
+
+```bash
+# Run with default examples (subscription + database)
+make e2e
+
+# Run with custom examples
+UPTEST_EXAMPLE_LIST="examples/rediscloud/acl-user.yaml" make e2e
+```
+
+#### E2E Testing with RedisCloud Pro Subscriptions
+
+Pro subscriptions require a valid payment method ID. The test infrastructure automatically handles this:
+
+```bash
+# Set RedisCloud API credentials
+export REDISCLOUD_API_KEY="your-api-key"
+export REDISCLOUD_SECRET_KEY="your-secret-key"
+export REDISCLOUD_URL="https://api.redislabs.com/v1"
+
+# Run e2e tests (automatically fetches payment method ID)
+make e2e
+
+# Or with 1Password integration
+op run -- make e2e
 ```
 
 For detailed testing instructions, see [TESTING.md](TESTING.md).