Use PEM certificates loaded from secrets for Kafka #11447

tinaselenge · 2025-05-19T10:41:16Z

Type of change

Refactoring

Description

Use KubernetesSecretConfigProvider to access secrets directly to configure Kafka truststore and keystore used for nodes to authenticate each other and with clients.
Create an internal secret that holds all the trusted certificates for Authorization server so that it can be accessed directly via KubernetesSecretConfigProvider to configure Authorization server's truststore.
Create an internal secret that holds all the trusted certificates under a same key for configured OAuth server of listeners, so that it can be volume mounted as a single cert file. This is used for OAuth server's Jaas configuration of truststore. The reason we cannot directly access this secret to configure ssl.truststore.certificates is because Jaas configuration cannot parse multiline of certificates. (This might get improved in a future PR but for now going with a simpler approach to keep the refactoring minimal).
Remove volume mounts and environment variables for configuring truststore and keystore as they are no longer needed because secrets are directly accessed.
Add auth hash of the internal secrets to the pod annotations, so that when there is a change in these secrets, pods will be triggered to roll.
Remove the script that is used to prepare TLS certificates which created PKCS12 certs based on the volume mounted PEM format certificates.
Refactored KafkaAgent to directly access the cluster CA and node certificates and use them to configure the HTTP server, instead of using PKCS12 certificates generated by the script. Added util class to allow creating JKS keystores from secrets.

Resolves part of #11294

Checklist

Please go through this checklist and make sure all applicable tasks have been done

Write tests
Make sure all tests pass
Update documentation
Check RBAC rights for Kubernetes / OpenShift roles
Try your changes from Pod inside your Kubernetes and OpenShift cluster, not just locally
Reference relevant issue(s) and close them after merging
Update CHANGELOG.md
Supply screenshots for visual changes, such as Grafana dashboards

tinaselenge · 2025-06-10T08:25:39Z

@ppatierno @katheris can you please review this PR when you get a chance? Thank you :)

...er-operator/src/main/java/io/strimzi/operator/cluster/operator/assembly/KafkaReconciler.java

ppatierno · 2025-06-16T07:38:25Z

...er-operator/src/main/java/io/strimzi/operator/cluster/operator/assembly/KafkaReconciler.java

+                                reconciliation.namespace(),
+                                oauthSecret,
+                                kafka.generateSecret(Map.of(oauthSecret + ".crt", mergeAndEncodeCerts(certs)), oauthSecret))
+                        .mapEmpty());


the core part of this method is exactly the same as the authzTrustedCertsSecret. Is there any way to factor out a common method here? wdyt?

They look similar but quite different with the way we join all the secrets. For cluster trusted certs, we just add them under individual keys in the secret, so it's simple. For OAuth, we are collecting the certs as a list of strings first and then decode and merge them into a single string before encoding and putting it under a single key in the secret.

However, these two methods are getting used in several places in KafkaConnect, Kafka and KafkaBridge. So I think if I should move them in AuthenticationUtils, instead of them repeating them in their assembly operator classes.

ppatierno · 2025-06-16T07:40:39Z

kafka-agent/src/main/java/io/strimzi/kafka/agent/KafkaAgent.java

-        sslContextFactory.setTrustStorePath(sslTruststorePath);
-        sslContextFactory.setTrustStorePassword(sslTruststorePassword);
+        sslContextFactory.setKeyStore(KafkaAgentUtils.jksKeyStore(nodeCertSecret));
+        sslContextFactory.setKeyStorePassword("changeit");


what about this "changeit"? We had a random generated password when using the script.

I had followed what KafkaAgentClient had, but you are right, we should do the same as the script. I added a method to generate random password with 32 chars like the script did.

kafka-agent/src/main/java/io/strimzi/kafka/agent/KafkaAgent.java

tinaselenge · 2025-06-18T14:42:29Z

Thank @ppatierno so much for reviewing the PR. I have now addressed your comments.

Could you also please kick off the regression tests?

im-konge · 2025-06-18T16:02:41Z

/azp run regression

azure-pipelines · 2025-06-18T16:02:54Z

Azure Pipelines successfully started running 1 pipeline(s).

im-konge · 2025-06-19T08:35:06Z

/azp run regression

azure-pipelines · 2025-06-19T08:35:19Z

Azure Pipelines successfully started running 1 pipeline(s).

katheris

The changes look pretty good to me, I just had a couple of questions and suggestions that I added.

katheris · 2025-06-19T09:33:09Z

...perator/src/main/java/io/strimzi/operator/cluster/model/KafkaBrokerConfigurationBuilder.java

-            writer.println(CruiseControlConfigurationParameters.METRICS_REPORTER_SSL_TRUSTSTORE_LOCATION + "=/tmp/kafka/cluster.truststore.p12");
-            writer.println(CruiseControlConfigurationParameters.METRICS_REPORTER_SSL_TRUSTSTORE_PASSWORD + "=" + PLACEHOLDER_CERT_STORE_PASSWORD_CONFIG_PROVIDER_ENV_VAR);
+            writer.println(CruiseControlConfigurationParameters.METRICS_REPORTER_SSL_KEYSTORE_TYPE + "=PEM");
+            writer.println(CruiseControlConfigurationParameters.METRICS_REPORTER_SSL_KEYSTORE_CERTIFICATE_CHAIN + "=" + String.format(PLACEHOLDER_SECRET_TEMPLATE_KUBE_CONFIG_PROVIDER, reconciliation.namespace(), node.podName(), node.podName() + ".crt"));


I wonder whether the code would read better if we had a method, e.g.

Suggested change

writer.println(CruiseControlConfigurationParameters.METRICS_REPORTER_SSL_KEYSTORE_CERTIFICATE_CHAIN + "=" + String.format(PLACEHOLDER_SECRET_TEMPLATE_KUBE_CONFIG_PROVIDER, reconciliation.namespace(), node.podName(), node.podName() + ".crt"));

writer.println(CruiseControlConfigurationParameters.METRICS_REPORTER_SSL_KEYSTORE_CERTIFICATE_CHAIN + "=" + secretConfigProvider(reconciliation.namespace(), node.podName(), node.podName() + ".crt"));

I can't remember what we've done for the other similar changes though, so maybe this is something we could look at as a follow up PR

I think that's a good idea but yeah maybe I can do it in a follow up PR.

...er-operator/src/main/java/io/strimzi/operator/cluster/operator/assembly/ReconcilerUtils.java

katheris · 2025-06-19T13:31:31Z

kafka-agent/src/main/java/io/strimzi/kafka/agent/KafkaAgentUtils.java

+/**
+ * Class with various utility methods for generating KeyStore and TrustStore for KafkaAgent
+ */
+public class KafkaAgentUtils {


A lot of these methods are duplicates of ones we have elsewhere, but I assume you put them here so we didn't have to pull in any other Strimzi modules and create cyclic dependencies?

Yes, that was the idea. I didn't think it was good idea to pull in any Strimzi modules into the agent running in the Kafka process.

ppatierno · 2025-06-23T08:09:45Z

@tinaselenge I restarted failed regression tests, not sure if they were related to the PR but there were quite a few. Let's see the next run.

im-konge · 2025-06-23T08:15:14Z

@tinaselenge I restarted failed regression tests, not sure if they were related to the PR but there were quite a few. Let's see the next run.

They failed even for the previous runs, so I guess they are related to the PR.

tinaselenge · 2025-06-24T15:13:25Z

yes, they are definitely related as they failed locally for me as well. I fixed OAuth related failures but still trying to fix some failures in ListenersST that tests listeners with custom certificates. I will update the PR once I have it passing locally.

katheris · 2025-06-30T10:08:03Z

/azp run regression

azure-pipelines · 2025-06-30T10:08:16Z

Azure Pipelines successfully started running 1 pipeline(s).

tinaselenge · 2025-07-24T11:17:44Z

@strimzi/maintainers can someone please kick off the regression tests? I believe I have fixed them now. Thank you.

im-konge · 2025-07-24T11:21:49Z

/azp run regression

azure-pipelines · 2025-07-24T11:22:02Z

Azure Pipelines successfully started running 1 pipeline(s).

tinaselenge · 2025-07-27T09:41:21Z

There is one remaining system test failure:

testKafkaWithVersion Kafka version: KafkaVersion{version='3.9.0', metadataVersion='3.9', isDefault=false, isSupported=true}.version()

testKafkaWithVersion Kafka version: KafkaVersion{version='3.9.1', metadataVersion='3.9', isDefault=false, isSupported=true}.version()

They fail because of the changes I made to KafkaAgent is missing from kafka-agent-3 (agent for Kafka 3.x versions). To fix this test, I would have to duplicate all the changes in kafka-agent-3. However Kafka 4.1.0 is currently in the process of getting released and will soon be supported in the next Strimzi release. This means the next Strimzi version will not support Kafka 3.9.x version, therefore kafka-agent-3 will be removed. Given that, we will likely to remove it in the next 2,3 weeks, I decided not to fix this test and wait for it to get removed before merging this PR.

While we wait for that to happen, I think this PR can be reviewed again since all the review comments were addressed and other system tests were fixed.

ppatierno · 2025-08-01T13:04:57Z

@tinaselenge could you please solve the conflicts we have now :-(

Signed-off-by: Gantigmaa Selenge <[email protected]>

…rets into ReconcilerUtils Signed-off-by: Gantigmaa Selenge <[email protected]>

Signed-off-by: Gantigmaa Selenge <[email protected]>

tinaselenge · 2025-08-01T14:57:06Z

@ppatierno done :)

tinaselenge force-pushed the use-pem-kafka branch from e5eaea1 to eb7a6d6 Compare May 19, 2025 12:06

tinaselenge force-pushed the use-pem-kafka branch from eb7a6d6 to 8ffd37f Compare May 28, 2025 14:09

tinaselenge marked this pull request as ready for review May 28, 2025 14:12

tinaselenge requested review from ppatierno and katheris and removed request for ppatierno May 28, 2025 14:12

ppatierno added this to the 0.47.0 milestone Jun 4, 2025

tinaselenge force-pushed the use-pem-kafka branch from 8ffd37f to 91eaf20 Compare June 12, 2025 13:09

ppatierno requested changes Jun 16, 2025

View reviewed changes

tinaselenge force-pushed the use-pem-kafka branch 2 times, most recently from 3d7de64 to 3363f8a Compare June 18, 2025 09:15

github-advanced-security bot found potential problems Jun 18, 2025

View reviewed changes

kafka-agent/src/main/java/io/strimzi/kafka/agent/KafkaAgent.java Fixed Show fixed Hide fixed

tinaselenge force-pushed the use-pem-kafka branch from f67eec1 to 595b61b Compare June 18, 2025 10:39

katheris reviewed Jun 19, 2025

View reviewed changes

scholzj modified the milestones: 0.47.0, 0.48.0 Jul 10, 2025

tinaselenge force-pushed the use-pem-kafka branch 2 times, most recently from bfede7c to 8ae072a Compare July 23, 2025 14:28

tinaselenge force-pushed the use-pem-kafka branch from b493973 to 6650758 Compare July 27, 2025 09:33

tinaselenge requested review from katheris and ppatierno July 27, 2025 09:41

tinaselenge added 7 commits August 1, 2025 15:52

Use PEM certificates loaded from secrets for Kafka

606faed

Signed-off-by: Gantigmaa Selenge <[email protected]>

Address review comments from Paolo

53df9e4

Signed-off-by: Gantigmaa Selenge <[email protected]>

Refactor out the common methods used for reconciling the internal sec…

7c7ecb9

…rets into ReconcilerUtils Signed-off-by: Gantigmaa Selenge <[email protected]>

Fix spotbugs failure

ffc9be3

Signed-off-by: Gantigmaa Selenge <[email protected]>

Address comments from Kate, fix system tests

0c70628

Signed-off-by: Gantigmaa Selenge <[email protected]>

Fix failing system tests

18e1038

Signed-off-by: Gantigmaa Selenge <[email protected]>

Fix unit tests that failed due to rebasing the main branch

c604c5b

Signed-off-by: Gantigmaa Selenge <[email protected]>

tinaselenge force-pushed the use-pem-kafka branch from 6650758 to c604c5b Compare August 1, 2025 14:52

	writer.println(CruiseControlConfigurationParameters.METRICS_REPORTER_SSL_KEYSTORE_CERTIFICATE_CHAIN + "=" + String.format(PLACEHOLDER_SECRET_TEMPLATE_KUBE_CONFIG_PROVIDER, reconciliation.namespace(), node.podName(), node.podName() + ".crt"));
	writer.println(CruiseControlConfigurationParameters.METRICS_REPORTER_SSL_KEYSTORE_CERTIFICATE_CHAIN + "=" + secretConfigProvider(reconciliation.namespace(), node.podName(), node.podName() + ".crt"));

Use PEM certificates loaded from secrets for Kafka #11447

Are you sure you want to change the base?

Use PEM certificates loaded from secrets for Kafka #11447

Conversation

tinaselenge commented May 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Type of change

Description

Checklist

Uh oh!

tinaselenge commented Jun 10, 2025

Uh oh!

Uh oh!

ppatierno Jun 16, 2025

Choose a reason for hiding this comment

Uh oh!

tinaselenge Jun 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ppatierno Jun 16, 2025

Choose a reason for hiding this comment

Uh oh!

tinaselenge Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tinaselenge commented Jun 18, 2025

Uh oh!

im-konge commented Jun 18, 2025

Uh oh!

azure-pipelines bot commented Jun 18, 2025

Uh oh!

im-konge commented Jun 19, 2025

Uh oh!

azure-pipelines bot commented Jun 19, 2025

Uh oh!

katheris left a comment

Choose a reason for hiding this comment

Uh oh!

katheris Jun 19, 2025

Choose a reason for hiding this comment

Uh oh!

tinaselenge Jun 20, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

katheris Jun 19, 2025

Choose a reason for hiding this comment

Uh oh!

tinaselenge Jun 20, 2025

Choose a reason for hiding this comment

Uh oh!

ppatierno commented Jun 23, 2025

Uh oh!

im-konge commented Jun 23, 2025

Uh oh!

tinaselenge commented Jun 24, 2025

Uh oh!

katheris commented Jun 30, 2025

Uh oh!

azure-pipelines bot commented Jun 30, 2025

Uh oh!

tinaselenge commented Jul 24, 2025

Uh oh!

im-konge commented Jul 24, 2025

Uh oh!

azure-pipelines bot commented Jul 24, 2025

Uh oh!

tinaselenge commented Jul 27, 2025

Uh oh!

ppatierno commented Aug 1, 2025

Uh oh!

tinaselenge commented Aug 1, 2025

Uh oh!

Uh oh!

tinaselenge commented May 19, 2025 •

edited

Loading

tinaselenge Jun 16, 2025 •

edited

Loading