[Bug]: Importing Ca certificates with USE_SYSTEM_CA_CERTS is slower than it used to be

[achimwinter]

Please add the exact image (with tag) that you are using

21-jre-alpine

Please add the version of Docker you are running

Docker version 26.1.3, build b72abbb

What happened?

Since yesterday (23.07.2024) i get a exception connecting to my documentDB Database on AWS.
My setup is like this: in a builder stage i download the global-bundle.pem from aws and copy that to /certificates/aws-ca.crt.
Setting the Environment variable
ENV USE_SYSTEM_CA_CERTS=1
Which worked for me so far, except for images from yesterday on.
Since then i get a whole lot of log statements in the container about all the certificates imported, but the application fails when connecting to the database.

I also tried to use: "JRE_CACERTS_PATH" (from https://github.com/docker-library/docs/pull/2445/files) added to the CMD, but that also didnt work. (I am not using a non-root image)

Using an older image works still fine.

Relevant log output

2024-07-24T13:21:07.052+02:00	Importing keystore /tmp/tmp.KbBNoA to /opt/java/openjdk/lib/security/cacerts...
2024-07-24T13:21:07.754+02:00	Entry for alias naverglobalrootcertificationauthority successfully imported.
2024-07-24T13:21:07.755+02:00	Entry for alias teliasonerarootcav1 successfully imported.
2024-07-24T13:21:07.756+02:00	Entry for alias vtruseccrootca successfully imported.
2024-07-24T13:21:07.757+02:00	Entry for alias emsignrootca-g1 successfully imported.
2024-07-24T13:21:07.757+02:00	Entry for alias quovadisrootca3g3 successfully imported.
2024-07-24T13:21:07.758+02:00	Entry for alias secureglobalca successfully imported.
2024-07-24T13:21:07.758+02:00	Entry for alias microsoftrsarootcertificateauthority2017 successfully imported.
2024-07-24T13:21:07.759+02:00	Entry for alias ssl.comevrootcertificationauthorityecc successfully imported.
2024-07-24T13:21:07.759+02:00	Entry for alias szafirrootca2 successfully imported.
2024-07-24T13:21:07.759+02:00	Entry for alias ssl.comtlsrsarootca2022 successfully imported.
2024-07-24T13:21:07.760+02:00	Entry for alias quovadisrootca1g3 successfully imported.
2024-07-24T13:21:07.760+02:00	Entry for alias atostrustedrootrootcarsatls2021 successfully imported.
2024-07-24T13:21:07.840+02:00	Entry for alias autoridaddecertificacionfirmaprofesionalcifa62634068 successfully imported.
2024-07-24T13:21:07.842+02:00	Entry for alias securesignrootca11 successfully imported.
2024-07-24T13:21:07.843+02:00	Entry for alias telekomsecuritytlsrsaroot2023 successfully imported.
2024-07-24T13:21:07.843+02:00	Entry for alias isrgrootx2 successfully imported.
2024-07-24T13:21:07.843+02:00	Entry for alias isrgrootx1 successfully imported.
2024-07-24T13:21:07.845+02:00	Entry for alias digicertglobalrootca successfully imported.
2024-07-24T13:21:07.845+02:00	Entry for alias sectigopublicserverauthenticationrootr46 successfully imported.
2024-07-24T13:21:07.846+02:00	Entry for alias bjcaglobalrootca2 successfully imported.
2024-07-24T13:21:07.846+02:00	Entry for alias globalsignroote46 successfully imported.
2024-07-24T13:21:07.846+02:00	Entry for alias bjcaglobalrootca1 successfully imported.
2024-07-24T13:21:07.847+02:00	Entry for alias starfieldservicesrootcertificateauthority-g2 successfully imported.
2024-07-24T13:21:07.847+02:00	Entry for alias actalisauthenticationrootca successfully imported.
2024-07-24T13:21:07.848+02:00	Entry for alias tubitakkamusmsslkoksertifikasi-surum1 successfully imported.
2024-07-24T13:21:07.849+02:00	Entry for alias amazonrootca4 successfully imported.
2024-07-24T13:21:07.850+02:00	Entry for alias amazonrootca3 successfully imported.
2024-07-24T13:21:07.851+02:00	Entry for alias amazonrootca2 successfully imported.
2024-07-24T13:21:07.851+02:00	Entry for alias amazonrootca1 successfully imported.
2024-07-24T13:21:07.851+02:00	Entry for alias affirmtrustpremium successfully imported.
2024-07-24T13:21:07.852+02:00	Entry for alias haricatlsrsarootca2021 successfully imported.
2024-07-24T13:21:07.852+02:00	Entry for alias entrustrootcertificationauthority-g4 successfully imported.
2024-07-24T13:21:07.852+02:00	Entry for alias entrustrootcertificationauthority-g2 successfully imported.
2024-07-24T13:21:07.853+02:00	Entry for alias gdcatrustauthr5root successfully imported.
2024-07-24T13:21:07.853+02:00	Entry for alias atostrustedrootrootcaecctls2021 successfully imported.
2024-07-24T13:21:07.854+02:00	Entry for alias telekomsecuritytlseccroot2020 successfully imported.
2024-07-24T13:21:07.854+02:00	Entry for alias emsigneccrootca-g3 successfully imported.
2024-07-24T13:21:07.855+02:00	Entry for alias atostrustedroot2011 successfully imported.
2024-07-24T13:21:07.855+02:00	Entry for alias d-trustevrootca12020 successfully imported.
2024-07-24T13:21:07.855+02:00	Entry for alias anfsecureserverrootca successfully imported.
2024-07-24T13:21:07.856+02:00	Entry for alias certignarootca successfully imported.
2024-07-24T13:21:07.856+02:00	Entry for alias swisssignsilverca-g2 successfully imported.
2024-07-24T13:21:07.857+02:00	Entry for alias vtrusrootca successfully imported.
2024-07-24T13:21:07.857+02:00	Entry for alias digicerttlsrsa4096rootg5 successfully imported.
2024-07-24T13:21:07.859+02:00	Entry for alias comodoecccertificationauthority successfully imported.
2024-07-24T13:21:07.860+02:00	Entry for alias securetrustca successfully imported.
2024-07-24T13:21:07.860+02:00	Entry for alias cadisigrootr2 successfully imported.
2024-07-24T13:21:07.860+02:00	Entry for alias aaacertificateservices successfully imported.
2024-07-24T13:21:07.861+02:00	Entry for alias starfieldrootcertificateauthority-g2 successfully imported.
2024-07-24T13:21:07.861+02:00	Entry for alias buypassclass2rootca successfully imported.
2024-07-24T13:21:07.862+02:00	Entry for alias tuntrustrootca successfully imported.
2024-07-24T13:21:07.939+02:00	Entry for alias buypassclass3rootca successfully imported.
2024-07-24T13:21:07.940+02:00	Entry for alias epkirootcertificationauthority successfully imported.
2024-07-24T13:21:07.940+02:00	Entry for alias entrust.netcertificationauthority(2048) successfully imported.
2024-07-24T13:21:07.940+02:00	Entry for alias certigna successfully imported.
2024-07-24T13:21:07.940+02:00	Entry for alias cfcaevroot successfully imported.
2024-07-24T13:21:07.941+02:00	Entry for alias emsignrootca-c1 successfully imported.
2024-07-24T13:21:07.941+02:00	Entry for alias certumtrustednetworkca successfully imported.
2024-07-24T13:21:07.941+02:00	Entry for alias securitycommunicationrootca3 successfully imported.
2024-07-24T13:21:07.941+02:00	Entry for alias securitycommunicationrootca2 successfully imported.
2024-07-24T13:21:07.941+02:00	Entry for alias oistewisekeyglobalrootgcca successfully imported.
2024-07-24T13:21:07.942+02:00	Entry for alias usertrustrsacertificationauthority successfully imported.
2024-07-24T13:21:07.942+02:00	Entry for alias trustwaveglobaleccp384certificationauthority successfully imported.
2024-07-24T13:21:07.942+02:00	Entry for alias firmaprofesionalcaroot-aweb successfully imported.
2024-07-24T13:21:07.942+02:00	Entry for alias swisssigngoldca-g2 successfully imported.
2024-07-24T13:21:07.942+02:00	Entry for alias globalsign-3 successfully imported.
2024-07-24T13:21:07.943+02:00	Entry for alias certsignrootca successfully imported.
2024-07-24T13:21:07.943+02:00	Entry for alias globalsign-2 successfully imported.
2024-07-24T13:21:07.943+02:00	Entry for alias globalsign-1 successfully imported.
2024-07-24T13:21:07.944+02:00	Entry for alias certumec-384ca successfully imported.
2024-07-24T13:21:07.944+02:00	Entry for alias hipkirootca-g1 successfully imported.
2024-07-24T13:21:07.944+02:00	Entry for alias twcaglobalrootca successfully imported.
2024-07-24T13:21:07.944+02:00	Entry for alias trustwaveglobaleccp256certificationauthority successfully imported.
2024-07-24T13:21:07.944+02:00	Entry for alias globalsignrootr46 successfully imported.
2024-07-24T13:21:07.945+02:00	Entry for alias entrustrootcertificationauthority-ec1 successfully imported.
2024-07-24T13:21:07.945+02:00	Entry for alias emsigneccrootca-c3 successfully imported.
2024-07-24T13:21:07.945+02:00	Entry for alias digicerttrustedrootg4 successfully imported.
2024-07-24T13:21:07.945+02:00	Entry for alias quovadisrootca2g3 successfully imported.
2024-07-24T13:21:07.945+02:00	Entry for alias trustwaveglobalcertificationauthority successfully imported.
2024-07-24T13:21:07.946+02:00	Entry for alias gtsrootr4 successfully imported.
2024-07-24T13:21:07.946+02:00	Entry for alias gtsrootr3 successfully imported.
2024-07-24T13:21:07.946+02:00	Entry for alias gtsrootr2 successfully imported.
2024-07-24T13:21:07.946+02:00	Entry for alias gtsrootr1 successfully imported.
2024-07-24T13:21:07.947+02:00	Entry for alias hellenicacademicandresearchinstitutionseccrootca2015 successfully imported.
2024-07-24T13:21:07.947+02:00	Entry for alias d-trustrootclass3ca22009 successfully imported.
2024-07-24T13:21:07.948+02:00	Entry for alias commscopepublictrustrsaroot-02 successfully imported.
2024-07-24T13:21:07.948+02:00	Entry for alias e-szignorootca2017 successfully imported.
2024-07-24T13:21:07.949+02:00	Entry for alias commscopepublictrustrsaroot-01 successfully imported.
2024-07-24T13:21:07.949+02:00	Entry for alias affirmtrustcommercial successfully imported.
2024-07-24T13:21:07.950+02:00	Entry for alias godaddyclass2certificationauthority successfully imported.
2024-07-24T13:21:07.950+02:00	Entry for alias digicertassuredidrootg3 successfully imported.
2024-07-24T13:21:07.950+02:00	Entry for alias affirmtrustnetworking successfully imported.
2024-07-24T13:21:07.951+02:00	Entry for alias digicertassuredidrootg2 successfully imported.
2024-07-24T13:21:07.951+02:00	Entry for alias d-trustrootclass3ca2ev2009 successfully imported.
2024-07-24T13:21:07.951+02:00	Entry for alias baltimorecybertrustroot successfully imported.
2024-07-24T13:21:07.952+02:00	Entry for alias comodocertificationauthority successfully imported.
2024-07-24T13:21:07.953+02:00	Entry for alias starfieldclass2certificationauthority successfully imported.
2024-07-24T13:21:07.953+02:00	Entry for alias usertrustecccertificationauthority successfully imported.
2024-07-24T13:21:07.953+02:00	Entry for alias quovadisrootca3 successfully imported.
2024-07-24T13:21:07.954+02:00	Entry for alias sectigopublicserverauthenticationroote46 successfully imported.
2024-07-24T13:21:07.955+02:00	Entry for alias quovadisrootca2 successfully imported.
2024-07-24T13:21:07.955+02:00	Entry for alias trustasiaglobalrootcag4 successfully imported.
2024-07-24T13:21:07.956+02:00	Entry for alias trustasiaglobalrootcag3 successfully imported.
2024-07-24T13:21:07.956+02:00	Entry for alias twcarootcertificationauthority successfully imported.
2024-07-24T13:21:07.956+02:00	Entry for alias d-trustbrrootca12020 successfully imported.
2024-07-24T13:21:07.956+02:00	Entry for alias commscopepublictrusteccroot-02 successfully imported.
2024-07-24T13:21:07.957+02:00	Entry for alias commscopepublictrusteccroot-01 successfully imported.
2024-07-24T13:21:07.957+02:00	Entry for alias certumtrustedrootca successfully imported.
2024-07-24T13:21:07.957+02:00	Entry for alias ucaglobalg2root successfully imported.
2024-07-24T13:21:07.957+02:00	Entry for alias ssl.comrootcertificationauthorityecc successfully imported.
2024-07-24T13:21:07.958+02:00	Entry for alias certainlyrootr1 successfully imported.
2024-07-24T13:21:07.958+02:00	Entry for alias identrustcommercialrootca1 successfully imported.
2024-07-24T13:21:07.958+02:00	Entry for alias izenpe.com successfully imported.
2024-07-24T13:21:07.959+02:00	Entry for alias ucaextendedvalidationroot successfully imported.
2024-07-24T13:21:07.960+02:00	Entry for alias microsece-szignorootca2009 successfully imported.
2024-07-24T13:21:07.960+02:00	Entry for alias acraizfnmt-rcmservidoresseguros successfully imported.
2024-07-24T13:21:07.960+02:00	Entry for alias digicerttlseccp384rootg5 successfully imported.
2024-07-24T13:21:07.960+02:00	Entry for alias certsignrootcag2 successfully imported.
2024-07-24T13:21:07.961+02:00	Entry for alias globalsignrootca successfully imported.
2024-07-24T13:21:07.963+02:00	Entry for alias acraizfnmt-rcm successfully imported.
2024-07-24T13:21:07.963+02:00	Entry for alias certainlyroote1 successfully imported.
2024-07-24T13:21:07.963+02:00	Entry for alias affirmtrustpremiumecc successfully imported.
2024-07-24T13:21:07.963+02:00	Entry for alias xrampglobalcertificationauthority successfully imported.
2024-07-24T13:21:07.963+02:00	Entry for alias teliarootcav2 successfully imported.
2024-07-24T13:21:07.963+02:00	Entry for alias netlockarany(classgold)ftanstvny successfully imported.
2024-07-24T13:21:07.963+02:00	Entry for alias ssl.comrootcertificationauthorityrsa successfully imported.
2024-07-24T13:21:07.964+02:00	Entry for alias entrustrootcertificationauthority successfully imported.
2024-07-24T13:21:07.964+02:00	Entry for alias digicertassuredidrootca successfully imported.
2024-07-24T13:21:08.039+02:00	Entry for alias digicertglobalrootg3 successfully imported.
2024-07-24T13:21:08.040+02:00	Entry for alias digicertglobalrootg2 successfully imported.
2024-07-24T13:21:08.040+02:00	Entry for alias certumtrustednetworkca2 successfully imported.
2024-07-24T13:21:08.040+02:00	Entry for alias oistewisekeyglobalrootgbca successfully imported.
2024-07-24T13:21:08.040+02:00	Entry for alias comodorsacertificationauthority successfully imported.
2024-07-24T13:21:08.040+02:00	Entry for alias haricatlseccrootca2021 successfully imported.
2024-07-24T13:21:08.041+02:00	Entry for alias ssl.comtlseccrootca2022 successfully imported.
2024-07-24T13:21:08.041+02:00	Entry for alias securitycommunicationeccrootca1 successfully imported.
2024-07-24T13:21:08.041+02:00	Entry for alias identrustpublicsectorrootca1 successfully imported.
2024-07-24T13:21:08.041+02:00	Entry for alias digicerthighassuranceevrootca successfully imported.
2024-07-24T13:21:08.041+02:00	Entry for alias accvraiz1 successfully imported.
2024-07-24T13:21:08.041+02:00	Entry for alias godaddyrootcertificateauthority-g2 successfully imported.
2024-07-24T13:21:08.041+02:00	Entry for alias microsofteccrootcertificateauthority2017 successfully imported.
2024-07-24T13:21:08.042+02:00	Entry for alias t-telesecglobalrootclass3 successfully imported.
2024-07-24T13:21:08.042+02:00	Entry for alias t-telesecglobalrootclass2 successfully imported.
2024-07-24T13:21:08.042+02:00	Entry for alias globalsign successfully imported.
2024-07-24T13:21:08.042+02:00	Entry for alias hongkongpostrootca3 successfully imported.
2024-07-24T13:21:08.042+02:00	Entry for alias ssl.comevrootcertificationauthorityrsar2 successfully imported.
2024-07-24T13:21:08.042+02:00	Entry for alias hellenicacademicandresearchinstitutionsrootca2015 successfully imported.
2024-07-24T13:21:08.056+02:00	Import command completed: 147 entries successfully imported, 0 entries failed or cancelled
2024-07-24T13:21:08.750+02:00	Warning: use -cacerts option to access cacerts keystore
2024-07-24T13:21:09.353+02:00	Certificate was added to keystore
2024-07-24T13:21:09.579+02:00	WARNING: ca-cert-aws-ca.pem does not contain exactly one certificate or CRL: skipping

[gdams]

This is likely due to the bump from alpine:3.19 to alpine:3.20. I'll do some digging and see if something needs changing

[gdams]

ping @rassie incase you have any ideas what might be going on here

[rassie]

From what I can see at the first glance, update-ca-certificates from alpine:3.19 also had this behaviour. Will need to dig a bit deeper.

@achimwinter can you give us the exact image versions you are using, i.e. working and breaking? Thanks.

[rassie]

Having dug around a bit, I think it's the changed algorithm around importing certificates. I'll try to figure out what to change and would add the fix to #572 or maybe a separate PR.

[achimwinter]

Sure 21.0.3_9-jre-alpine is definitely working for me. And i think i even tried it with 21-ubi9-minimal, but i would need to make sure of that. I also saw, that all these info statements at the beginning were also the like Entry for alias securitycommunicationeccrootca1 successfully imported. are only found in the broken images. Maybe that helps.

EDIT: I can confirm that the same error applies to e.g. 21.0.4_7-jre-ubi9-minimal which seems in line with the algorithm being the issue

[rassie]

Just for the record: the reason is that the previous algorithm generated system certificates with update-ca-certificates first and then extracted those to the Java trust store which worked with multiple certificates in a single file. Current algorithm doesn't play nice with multiple certificates per file. Have to figure out a workaround now.

[gdams]

Just for the record: the reason is that the previous algorithm generated system certificates with update-ca-certificates first and then extracted those to the Java trust store which worked with multiple certificates in a single file. Current algorithm doesn't play nice with multiple certificates per file. Have to figure out a workaround now.

Thanks for looking into this. Can you make sure that we add a test to prevent this being broken again in the future?

[rassie]

@gdams I've updated #572 to amend this problem. I've extended a test to check for multiple certificates per file.

[gdams]

@achimwinter @rassie has prepared a fix, we'll get his changes merged and roll out a new release.

[gdams]

keeping this open until we've confirmed a fix. The images will be updated once docker-library/official-images#17249 is merged