Skip to content

Conversation

nogueiraanderson
Copy link
Contributor

@nogueiraanderson nogueiraanderson commented Aug 26, 2025

Automatic SSL Certificate Configuration for OpenShift Clusters

Jira Ticket: PMM-14242 - Add SSL certificate automation for OpenShift clusters

Problem

OpenShift clusters created through Jenkins require manual SSL certificate configuration for both the console and PMM services. This involves:

  • Manually installing cert-manager operator and creating Let's Encrypt issuers
  • Creating certificates and applying them to routes
  • Configuring AWS ACM certificates for LoadBalancer services
  • Creating Route53 DNS records manually
  • Different procedures for different service types (routes vs LoadBalancers)

Solution

Implemented comprehensive SSL automation that:

  • Automatically configures Let's Encrypt certificates for OpenShift console routes
  • Manages AWS ACM certificates for PMM LoadBalancer services
  • Creates DNS validation records for new ACM certificates
  • Auto-switches PMM to LoadBalancer service type when SSL is enabled
  • Handles Route53 DNS record creation automatically
  • Provides unified SSL configuration stage in the pipeline

Changes

New Libraries:

  • vars/openshiftSSL.groovy - Let's Encrypt certificate management via cert-manager
  • vars/awsCertificates.groovy - AWS ACM certificate and Route53 DNS automation

Modified Files:

  • cloud/jenkins/openshift-cluster-create.yml - Added SSL configuration parameters section
  • cloud/jenkins/openshift_cluster_create.groovy - Added SSL configuration stage
  • vars/openshiftCluster.groovy - Enhanced deployPMM() with automatic SSL support

Example

Before:

// Manual SSL configuration required after cluster creation
// No SSL parameters available
stage('Deploy PMM') {
    // PMM deployed with ClusterIP, no SSL
    openshiftCluster.deployPMM(params)
}

After:

// Automatic SSL configuration
stage('Configure SSL Certificates') {
    when {
        expression { params.ENABLE_SSL == true }
    }
    steps {
        script {
            // Automatically configures Let's Encrypt for console
            // Automatically configures ACM for PMM if deployed
            def sslConfig = [
                enableSSL: params.ENABLE_SSL,
                sslMethod: params.SSL_METHOD,
                sslEmail: params.SSL_EMAIL,
                // ... other SSL params
            ]
            
            if (params.SSL_METHOD == 'letsencrypt') {
                openshiftSSL.setupLetsEncrypt(sslConfig)
            }
            
            // PMM automatically gets LoadBalancer + ACM when SSL enabled
            if (params.DEPLOY_PMM) {
                // ACM configured automatically in deployPMM()
            }
        }
    }
}

Notes

  • ACM certificates require DNS validation (now automated)
  • Let's Encrypt certificates auto-renew every 60 days via cert-manager
  • DNS propagation may take up to 5 minutes

Validation

  • Create cluster with SSL enabled (no existing certificates)
  • Verify Let's Encrypt certificates on OpenShift console
  • Verify ACM certificates on PMM LoadBalancer
  • Check Route53 DNS records created correctly
  • Test with staging certificates first
  • Verify HTTPS endpoints are accessible

- Add Let's Encrypt support via cert-manager for OpenShift routes
- Add AWS ACM certificate management for LoadBalancer services
- Implement DNS automation with Route53
- Auto-switch PMM to LoadBalancer when SSL is enabled
- Add ACM certificate creation with DNS validation

Changes:
- New vars/openshiftSSL.groovy library for Let's Encrypt management
- New vars/awsCertificates.groovy library for ACM and Route53
- Enhanced openshift_cluster_create pipeline with SSL parameters
- Modified deployPMM() to support ACM certificates automatically

Related to: PMM-14242
@nogueiraanderson nogueiraanderson force-pushed the feature/openshift-ssl-automation branch from 53b1487 to 0c5b1eb Compare August 26, 2025 06:37
- Add newline to vars/openshiftSSL.groovy
- Add newline to vars/awsCertificates.groovy

Per code style guidelines, all files should end with a newline character.
@nogueiraanderson nogueiraanderson changed the title Implement automatic SSL certificate configuration for OpenShift clusters PMM-14242: Implement automatic SSL certificate configuration for OpenShift clusters Aug 26, 2025
@nogueiraanderson nogueiraanderson marked this pull request as ready for review August 26, 2025 06:44
nogueiraanderson added a commit that referenced this pull request Sep 1, 2025
- Moved all OpenShift cluster management files from cloud/jenkins to pmm/openshift
- Updated script paths in YAML files to reflect new location
- Aligns with organizational structure changes from PR #3504
nogueiraanderson added a commit that referenced this pull request Sep 1, 2025
…e job (#3517)

* Add PMM_IP environment variable to openshift-cluster-create job

- Modified deployPMM function to retrieve PMM service IP address
- Added logic to get IP from monitoring-service LoadBalancer or ClusterIP
- Export PMM_IP as environment variable in the pipeline
- Display PMM IP address in post-creation output
- Store PMM IP in cluster metadata for reference

* Add PMM IP to job description and fix parameter references

- Display PMM IP in the Jenkins job description HTML
- Fix incorrect parameter references (MASTER_NODES -> 3, WORKER_NODES -> WORKER_COUNT, etc.)
- Fix password reference to use env.PMM_PASSWORD instead of env.PMM_ADMIN_PASSWORD

* Move OpenShift cluster management jobs to pmm/openshift directory

- Moved all OpenShift cluster management files from cloud/jenkins to pmm/openshift
- Updated script paths in YAML files to reflect new location
- Aligns with organizational structure changes from PR #3504

* Update Jenkins job configurations to use master branch and official repository

- Changed branch from 'feature/openshift-shared-libraries' to 'master'
- Updated repository URL from personal fork to official Percona-Lab repository

* Fix PMM IP retrieval to use public ingress IP instead of internal ClusterIP

The monitoring-service in PMM namespace is a ClusterIP service (internal only),
not a LoadBalancer. The actual public access goes through the OpenShift ingress
controller (router-default service in openshift-ingress namespace).

This fix:
- Gets the public IP from the ingress controller LoadBalancer
- Resolves the hostname to IP address for external access
- Removes the incorrect fallback to ClusterIP which returns internal IPs

Testing confirmed:
- Old approach returned 172.30.x.x (internal, not accessible)
- New approach returns actual public IP (e.g., 3.129.202.84)
- External tools can now connect to PMM using the correct IP

* Remove trailing whitespace

* Fix fallback logic: only resolve hostname if no direct IP available

The previous logic was flawed - it would try hostname first and only fall back
to IP if hostname wasn't available. But if a provider gives us a direct IP,
we should just use it without DNS resolution.

New logic:
1. First check if LoadBalancer provides direct IP (GCP, some bare metal)
2. If no IP, check for hostname (AWS, Azure) and resolve it
3. If neither, try to get external IP from worker node (on-prem setups)

This ensures we get a working IP address across all cloud providers and
deployment scenarios.

* Simplify PMM IP retrieval for AWS-only environment

Since this is entirely AWS, remove unnecessary fallback logic:
- Remove check for direct IP (AWS always provides hostname)
- Remove worker node external IP fallback (not applicable to AWS)
- Keep only the AWS ELB hostname resolution logic

This makes the code cleaner and more maintainable by focusing only
on the AWS use case that actually applies.

* Use getent for DNS resolution with nslookup fallback

- getent is available by default on Oracle Linux 9 Jenkins agents
- nslookup requires bind-utils package which may not be installed
- Added fallback to nslookup if getent fails for any reason
- Tested with real AWS ELB hostnames and confirmed working

This ensures DNS resolution works reliably on Jenkins agents without requiring additional packages.
@nogueiraanderson nogueiraanderson force-pushed the feature/openshift-ssl-automation branch from 3d5da75 to 458c52d Compare September 2, 2025 10:02
@nogueiraanderson nogueiraanderson merged commit 283a92e into master Sep 2, 2025
@nogueiraanderson nogueiraanderson deleted the feature/openshift-ssl-automation branch September 2, 2025 11:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants