Resource map links missing in Solr index after Metacat 3.0 migration and reindex

[vchendrix]

Summary

After migrating to Metacat 3.0 and reindexing the entire corpus with the new DataONE indexer, we observed that resource map links are missing in the Solr index for at least 22 public and 3 private datasets. The resourceMap field in the indexed metadata objects is empty even though valid resource maps exist and are properly indexed. This issue was not seen prior to the 3.0 migration and reindex.

Example

• Dataset: https://data.ess-dive.lbl.gov/view/doi:10.15485/1898912
• Resource map PID (first broken version): ess-dive-46903e0c9c3de02-20221117T224253475
• Metadata PID (first broken version): ess-dive-8e6738870d87db3-20221117T224253485
• The first version where the resourceMap stopped appearing is provided above. All subsequent versions in the obsolescence chain are also missing resource map links.

Obsolescence Chain (Representative example)

This is the obsolescence chain for a representative example. NOTE: We have found all the resource maps and noted where the resource map was indexed but not linked to the metadata (see "missing" column).

metadata_id	seriesId	resourceMap_id	missing
ess-dive-9be051bcebbb9e6-20250428T213324618	doi:10.15485/1898912	ess-dive-c1729924d2c59eb-20250428T213324615	X
ess-dive-9be051bcebbb9e6-20250428T213257023	doi:10.15485/1898912	ess-dive-c581eebb5b7b50a-20250428T213257013	X
ess-dive-9be051bcebbb9e6-20250424T205756642	doi:10.15485/1898912	ess-dive-841f3051d9a2708-20250424T205756612	X
ess-dive-9be051bcebbb9e6-20240819T204345623	doi:10.15485/1898912	ess-dive-ce2dc192547db4e-20240819T204345615	X
ess-dive-9be051bcebbb9e6-20240314T204040227	doi:10.15485/1898912	ess-dive-30182ebce4568fe-20240314T204040214	X
ess-dive-9be051bcebbb9e6-20240108T174838968	doi:10.15485/1898912	ess-dive-b6e3586c57f24e8-20240108T174838956	X
ess-dive-9be051bcebbb9e6-20231108T174335343	doi:10.15485/1898912	ess-dive-7095e0ccee0ba15-20231108T174335337	X
ess-dive-9be051bcebbb9e6-20231108T162220676	doi:10.15485/1898912	ess-dive-a4c4b6d97257110-20231108T162220664	X
ess-dive-9be051bcebbb9e6-20231107T003557346	doi:10.15485/1898912	ess-dive-2cb51a443565303-20231107T003557339	X
ess-dive-9be051bcebbb9e6-20231107T000616818	doi:10.15485/1898912	ess-dive-e4959e48bc0d436-20231107T000616813	X
ess-dive-9be051bcebbb9e6-20231106T235858719	doi:10.15485/1898912	ess-dive-c017ba5643cb046-20231106T235858715	X
ess-dive-9be051bcebbb9e6-20231106T235820611	doi:10.15485/1898912	ess-dive-89c84faea686d5f-20231106T235820604	X
ess-dive-9be051bcebbb9e6-20231106T235015592	doi:10.15485/1898912	ess-dive-677c91786303c4b-20231106T235015584	X
ess-dive-9be051bcebbb9e6-20231106T231212959	doi:10.15485/1898912	ess-dive-7fda0c47b8e1f2f-20231106T231212952	X
ess-dive-9be051bcebbb9e6-20231106T231106515	doi:10.15485/1898912	ess-dive-0ee949f8376453a-20231106T231106507	X
ess-dive-9be051bcebbb9e6-20231102T191507382	doi:10.15485/1898912	ess-dive-e1a0ec7bfee78f1-20231102T191507376	X
ess-dive-9be051bcebbb9e6-20231024T165344106	doi:10.15485/1898912	ess-dive-22443e0250facf3-20231024T165344094	X
ess-dive-9be051bcebbb9e6-20231020T171513129	doi:10.15485/1898912	ess-dive-da805827db18fb5-20231020T171513122	X
ess-dive-9be051bcebbb9e6-20231020T151718937	doi:10.15485/1898912	ess-dive-e5fa24875975767-20231020T151718932	X
ess-dive-9be051bcebbb9e6-20231020T151546870	doi:10.15485/1898912	ess-dive-8e968f0b58ae787-20231020T151546863	X
ess-dive-9be051bcebbb9e6-20231020T151029878	doi:10.15485/1898912	ess-dive-288593f8d05d0a8-20231020T151029850	X
ess-dive-9be051bcebbb9e6-20230509T160352692	doi:10.15485/1898912	ess-dive-4d8dbc35b605239-20230509T160352683	X
ess-dive-e77eecf1104cc78-20230504T212124108475	doi:10.15485/1898912	ess-dive-23d73b4b474f11e-20230504T212129266471	X
ess-dive-a5ddb655e7c69bc-20230407T152426839335	doi:10.15485/1898912	ess-dive-140808a28a9d9fc-20230407T152434860390	X
ess-dive-792a06a368570f4-20230406T135257441502	doi:10.15485/1898912	ess-dive-c28f0962d002526-20230406T135302307466	X
ess-dive-7ccb579e570dbd8-20230406T122652349829	doi:10.15485/1898912	ess-dive-8b19a68c5d29e91-20230406T122700866201	X
ess-dive-9be051bcebbb9e6-20221205T175525485	doi:10.15485/1898912	ess-dive-b6c52d4c0d5a817-20221205T175525476	X
ess-dive-8e6738870d87db3-20221117T224253485	doi:10.15485/1898912	ess-dive-46903e0c9c3de02-20221117T224253475	X
ess-dive-20f26dd7286247a-20221111T172542084		ess-dive-d89fb6fad7353a8-20221111T172542074
ess-dive-ae18366438d5362-20221108T220702423		ess-dive-4aa268acb569f9d-20221108T220702416
ess-dive-a2314047b1cfdb2-20221108T214218585		ess-dive-0b825c5aa441e47-20221108T214218568
ess-dive-c0639102a0bd655-20221108T164958962		ess-dive-f7c9cc166222f0f-20221108T164958956
ess-dive-4c48f26f7f8afa0-20221108T000408556		ess-dive-5d6e27e22134ecc-20221108T000408551
ess-dive-8faddcacbf35bb7-20220920T175050062		ess-dive-251ae38a2ebcfd2-20220920T175050053
ess-dive-0824643ddb85a53-20220920T174739806142		ess-dive-dc06abe31a531ef-20220920T174741711311

Log Output (excerpt from dataone-indexer during reindexing)

dataone-indexer 20250610-15:44:47: [INFO]: IndexWorker.consumer.indexObject by multiple thread? true, with the thread id 44 - Received the index task from the index queue with the identifier: ess-dive-46903e0c9c3de02-20221117T224253475 , the index type: create, the priority: 3 [org.dataone.cn.indexer.IndexWorker:indexObject:454]
[... additional log lines omitted for brevity ...]
dataone-indexer 20250610-15:44:50: [INFO]: SeriesIdResolver.getPid - get the system metadata from the mn base url http://metacat-hl:8080/catalog/d1/mn for the object doi:10.15485/1898912 [org.dataone.cn.indexer.parser.utility.SeriesIdResolver:getPid:53]
dataone-indexer 20250610-15:44:50: [WARN]: SeriesIdResolver.getPid - can't get the system metadata from the mn http://metacat-hl:8080/catalog/d1/mn for the object doi:10.15485/1898912 since class org.dataone.client.exception.ClientSideException: /metacat-hl: Name or service not known. We will try to get it from cn. [org.dataone.cn.indexer.parser.utility.SeriesIdResolver:getPid:61]
dataone-indexer 20250610-15:44:50: [INFO]: SeriesIdResolver.getPid - get the system metadata for the object doi:10.15485/1898912 from the cn since the current node is cn or the systemmetadata is not available on a mn with baseurl http://metacat-hl:8080/catalog/d1/mn [org.dataone.cn.indexer.parser.utility.SeriesIdResolver:getPid:67]
dataone-indexer 20250610-15:44:50: [INFO]: response httpCode: 200 [org.dataone.service.util.ExceptionHandler:filterErrors:93]
dataone-indexer 20250610-15:44:50: [INFO]: ||||||||||||||||||| the head version is urn:uuid:da404cb4-1b7e-47ec-9e29-a552f5bb04cf for sid doi:10.15485/1898912 [org.dataone.cn.indexer.resourcemap.ForesiteResourceMap:isHeadVersion:214]
2025-06-10T15:44:50.236318555Z dataone-indexer 20250610-15:44:50: [INFO]: ||||||||||||||||||| the pid ess-dive-8e6738870d87db3-20221117T224253485 is NOT the head version for sid doi:10.15485/1898912 [org.dataone.cn.indexer.resourcemap.ForesiteResourceMap:isHeadVersion:221]
dataone-indexer 20250610-15:44:50: [INFO]: The id org.dataone.service.types.v1.Identifier@c6ea9113 is not the head of the serial id doi:10.15485/1898912 So, skip merge this one!!!!!!!!!!!!!!!!!!!!!!ess-dive-8e6738870d87db3-20221117T224253485 [
[... additional log lines omitted for brevity ...]

Additional Details

• In total, at least 22 public and 3 private datasets are confirmed affected. Not all datasets have been exhaustively checked, but the issue is significant and representative.
• The problem appears to have started immediately after the Metacat 3.0 migration and reindex.
• Reindexing resource maps completes without error but does not restore the missing links.
• Impact: Affected datasets have broken file-to-metadata links in Solr and the user-facing catalog.
• Important: The log output above shows the indexer is attempting to connect to the Member Node at metacat-hl:8080, but this service does not exist in our deployment. The correct service name is metacat:8080. This service name mismatch may be causing or contributing to the issue.

Configuration/Deployment Note

How can we override or configure the indexer (or DataONE client libraries) to use the correct Member Node service name (metacat:8080) instead of the non-existent metacat-hl:8080? Please provide guidance on where this is set (environment variable, config file, or code) and the recommended approach to ensure proper resolution.

Hypothesis

This may be related to changes in how head versions and resource maps are processed in the indexer logic after the Metacat 3.0 upgrade. The indexer determines that the metadata PID is not the head version for its seriesId and therefore skips merging the resource map reference, leaving the resourceMap field empty in Solr.

Request

Please investigate and advise on a fix or workaround. Let us know if additional details, logs, or dataset examples are needed.

[vchendrix]

@artntek I think I found a way to override (idxworker.mn_url)

[vchendrix]

@artntek I think I found a way to override (idxworker.mn_url)

@artntek I am going to see if this fixes the issue. Consider making this merge skip should have been an ERROR log not INFO.

[vchendrix]

I indexed the current RM and it updated fine. However, I tried to reindex all the old RM pids and not of them indexed due to version conflict errors. (errors from one indexer)

    <str name="root-error-class">org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException</str>
2025-06-10T17:11:33.043162005Z   </lst>
  <str name="msg">Async exception during distributed update: Error from server at http://ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local:8983/solr: null
2025-06-10T17:11:33.043167245Z 
2025-06-10T17:11:33.043169505Z 

2025-06-10T17:11:33.043173595Z request: http://ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local:8983/solr
2025-06-10T17:11:33.043175925Z Remote error message: version conflict for ess-dive-685175923eb7bb8-20231106T235042924 expected=1833505466317012992 actual=1834562871258775552</str>
2025-06-10T17:11:33.043178375Z   <int name="code">409</int>
2025-06-10T17:11:33.043180645Z </lst>
2025-06-10T17:11:33.043182745Z </response>
 [org.dataone.cn.indexer.solrhttp.HTTPService:sendUpdate:128]
2025-06-10T17:11:35.917836369Z dataone-indexer 20250610-17:11:35: [ERROR]: <?xml version="1.0" encoding="UTF-8"?>
2025-06-10T17:11:35.917894380Z <response>

2025-06-10T17:11:35.917900920Z <lst name="responseHeader">
  <int name="rf">2147483647</int>
2025-06-10T17:11:35.917908880Z   <int name="status">409</int>
2025-06-10T17:11:35.917912781Z   <int name="QTime">399</int>
</lst>
2025-06-10T17:11:35.917918361Z <lst name="error">
2025-06-10T17:11:35.917920861Z   <lst name="metadata">
2025-06-10T17:11:35.917923381Z     <str name="error-class">org.apache.solr.common.SolrException</str>
2025-06-10T17:11:35.917948372Z     <str name="root-error-class">org.apache.solr.common.SolrException</str>
    <str name="error-class">org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException</str>
2025-06-10T17:11:35.917953362Z     <str name="root-error-class">org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException</str>
2025-06-10T17:11:35.917955852Z   </lst>
2025-06-10T17:11:35.917958352Z   <str name="msg">Async exception during distributed update: Error from server at http://ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local:8983/solr: null
2025-06-10T17:11:35.917960942Z 
2025-06-10T17:11:35.917963012Z 

2025-06-10T17:11:35.917967122Z request: http://ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local:8983/solr
2025-06-10T17:11:35.917969522Z Remote error message: version conflict for ess-dive-07c18d3cc34ffd1-20231106T230314621 expected=1833505452971786240 actual=1834562873307693056</str>
  <int name="code">409</int>
2025-06-10T17:11:35.917974253Z </lst>
2025-06-10T17:11:35.917976402Z </response>
2025-06-10T17:11:35.917978603Z  [org.dataone.cn.indexer.solrhttp.HTTPService:sendUpdate:128]
2025-06-10T17:11:36.145039683Z dataone-indexer 20250610-17:11:36: [ERROR]: <?xml version="1.0" encoding="UTF-8"?>
2025-06-10T17:11:36.145066054Z <response>
2025-06-10T17:11:36.145069974Z 
2025-06-10T17:11:36.145072794Z <lst name="responseHeader">
  <int name="rf">2147483647</int>
2025-06-10T17:11:36.145080684Z   <int name="status">409</int>
2025-06-10T17:11:36.145084384Z   <int name="QTime">5</int>
2025-06-10T17:11:36.145086724Z </lst>
2025-06-10T17:11:36.145088914Z <lst name="error">
2025-06-10T17:11:36.145091145Z   <lst name="metadata">
    <str name="error-class">org.apache.solr.common.SolrException</str>
2025-06-10T17:11:36.145095915Z     <str name="root-error-class">org.apache.solr.common.SolrException</str>
    <str name="error-class">org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException</str>
2025-06-10T17:11:36.145101875Z     <str name="root-error-class">org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException</str>
2025-06-10T17:11:36.145104605Z   </lst>
2025-06-10T17:11:36.145106825Z   <str name="msg">Async exception during distributed update: Error from server at http://ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local:8983/solr: null

2025-06-10T17:11:36.145130456Z 
2025-06-10T17:11:36.145132546Z 
2025-06-10T17:11:36.145134666Z request: http://ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local:8983/solr
Remote error message: version conflict for ess-dive-685175923eb7bb8-20231106T235042924 expected=1834562871258775552 actual=1834562874935083008</str>
2025-06-10T17:11:36.145139566Z   <int name="code">409</int>
</lst>
2025-06-10T17:11:36.145144006Z </response>
2025-06-10T17:11:36.145146196Z  [org.dataone.cn.indexer.solrhttp.HTTPService:sendUpdate:128]
dataone-indexer 20250610-17:11:37: [ERROR]: <?xml version="1.0" encoding="UTF-8"?>
2025-06-10T17:11:37.595498494Z <response>
2025-06-10T17:11:37.595501924Z 
2025-06-10T17:11:37.595505614Z <lst name="responseHeader">
2025-06-10T17:11:37.595510394Z   <int name="rf">2147483647</int>
  <int name="status">409</int>
2025-06-10T17:11:37.595517215Z   <int name="QTime">168</int>
</lst>
2025-06-10T17:11:37.595522875Z <lst name="error">
2025-06-10T17:11:37.595525595Z   <lst name="metadata">
    <str name="error-class">org.apache.solr.common.SolrException</str>
2025-06-10T17:11:37.595531595Z     <str name="root-error-class">org.apache.solr.common.SolrException</str>
2025-06-10T17:11:37.595534785Z     <str name="error-class">org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException</str>
    <str name="root-error-class">org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException</str>
2025-06-10T17:11:37.595541555Z   </lst>
  <str name="msg">Async exception during distributed update: Error from server at http://ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local:8983/solr: null
2025-06-10T17:11:37.595547925Z 
2025-06-10T17:11:37.595550585Z 
2025-06-10T17:11:37.595553365Z 
2025-06-10T17:11:37.595555945Z request: http://ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local:8983/solr
Remote error message: version conflict for ess-dive-f0807b9891745e2-20231020T151716445 expected=1834562873545719808 actual=1834562876522627072</str>
2025-06-10T17:11:37.595576846Z   <int name="code">409</int>
2025-06-10T17:11:37.595579466Z </lst>
</response>
2025-06-10T17:11:37.595584026Z  [org.dataone.cn.indexer.solrhttp.HTTPService:sendUpdate:128]
2025-06-10T17:11:40.674069571Z dataone-indexer 20250610-17:11:40: [ERROR]: <?xml version="1.0" encoding="UTF-8"?>
<response>
2025-06-10T17:11:40.674120323Z 
<lst name="responseHeader">
2025-06-10T17:11:40.674128073Z   <int name="rf">2147483647</int>
2025-06-10T17:11:40.674131143Z   <int name="status">409</int>
  <int name="QTime">460</int>
2025-06-10T17:11:40.674138804Z </lst>
2025-06-10T17:11:40.674141364Z <lst name="error">
  <lst name="metadata">
2025-06-10T17:11:40.674146104Z     <str name="error-class">org.apache.solr.common.SolrException</str>
    <str name="root-error-class">org.apache.solr.common.SolrException</str>
    <str name="error-class">org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException</str>
    <str name="root-error-class">org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException</str>
  </lst>
2025-06-10T17:11:40.674158934Z   <str name="msg">Async exception during distributed update: Error from server at http://ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local:8983/solr: null

2025-06-10T17:11:40.674163574Z 
2025-06-10T17:11:40.674165614Z 
request: http://ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local:8983/solr
Remote error message: version conflict for ess-dive-f0807b9891745e2-20231020T151716445 expected=1834562878550573056 actual=1834562879383142400</str>
2025-06-10T17:11:40.674172874Z   <int name="code">409</int>
2025-06-10T17:11:40.674175114Z </lst>
2025-06-10T17:11:40.674177195Z </response>
2025-06-10T17:11:40.674179345Z  [org.dataone.cn.indexer.solrhttp.HTTPService:sendUpdate:128]
2025-06-10T17:11:42.897616917Z dataone-indexer 20250610-17:11:42: [ERROR]: <?xml version="1.0" encoding="UTF-8"?>
<response>
2025-06-10T17:11:42.897641317Z 
2025-06-10T17:11:42.897963777Z <lst name="responseHeader">
  <int name="rf">2147483647</int>
  <int name="status">409</int>
  <int name="QTime">735</int>
2025-06-10T17:11:42.897976337Z </lst>
2025-06-10T17:11:42.897978757Z <lst name="error">
2025-06-10T17:11:42.897981198Z   <lst name="metadata">
    <str name="error-class">org.apache.solr.common.SolrException</str>
    <str name="root-error-class">org.apache.solr.common.SolrException</str>
2025-06-10T17:11:42.897988828Z     <str name="error-class">org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException</str>
    <str name="root-error-class">org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException</str>
2025-06-10T17:11:42.897994068Z   </lst>
2025-06-10T17:11:42.897996518Z   <str name="msg">Async exception during distributed update: Error from server at http://ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local:8983/solr: null
2025-06-10T17:11:42.897999348Z 
2025-06-10T17:11:42.898001428Z 

2025-06-10T17:11:42.898005608Z request: http://ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local:8983/solr
Remote error message: version conflict for ess-dive-f0807b9891745e2-20231020T151716445 expected=1834562880148602880 actual=1834562881450934272</str>
2025-06-10T17:11:42.898010538Z   <int name="code">409</int>
</lst>
2025-06-10T17:11:42.898014999Z </response>
2025-06-10T17:11:42.898017208Z  [org.dataone.cn.indexer.solrhttp.HTTPService:sendUpdate:128]
2025-06-10T17:11:43.779769174Z dataone-indexer 20250610-17:11:43: [ERROR]: <?xml version="1.0" encoding="UTF-8"?>
<response>

2025-06-10T17:11:43.779807225Z <lst name="responseHeader">
  <int name="rf">2147483647</int>
2025-06-10T17:11:43.779812585Z   <int name="status">409</int>
  <int name="QTime">7</int>
2025-06-10T17:11:43.779817615Z </lst>
<lst name="error">
  <lst name="metadata">
2025-06-10T17:11:43.779839866Z     <str name="error-class">org.apache.solr.common.SolrException</str>
    <str name="root-error-class">org.apache.solr.common.SolrException</str>
2025-06-10T17:11:43.779845186Z     <str name="error-class">org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException</str>
2025-06-10T17:11:43.779848066Z     <str name="root-error-class">org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException</str>
  </lst>
2025-06-10T17:11:43.779853126Z   <str name="msg">Async exception during distributed update: Error from server at http://ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local:8983/solr: null
2025-06-10T17:11:43.779856036Z 
2025-06-10T17:11:43.779858386Z 
2025-06-10T17:11:43.779860446Z 
request: http://ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local:8983/solr
2025-06-10T17:11:43.779865646Z Remote error message: version conflict for ess-dive-8b3d30649d708c7-20221111T171843927 expected=1834562880424378368 actual=1834562882931523584</str>
  <int name="code">409</int>
2025-06-10T17:11:43.779871487Z </lst>
2025-06-10T17:11:43.779874177Z </response>
2025-06-10T17:11:43.779877017Z  [org.dataone.cn.indexer.solrhttp.HTTPService:sendUpdate:128]
2025-06-10T17:11:45.111284920Z dataone-indexer 20250610-17:11:45: [ERROR]: <?xml version="1.0" encoding="UTF-8"?>
<response>
2025-06-10T17:11:45.111315370Z 
2025-06-10T17:11:45.111318410Z <lst name="responseHeader">
2025-06-10T17:11:45.111321950Z   <int name="rf">2147483647</int>
2025-06-10T17:11:45.111326051Z   <int name="status">409</int>
2025-06-10T17:11:45.111329871Z   <int name="QTime">673</int>
</lst>
2025-06-10T17:11:45.111335521Z <lst name="error">
  <lst name="metadata">
    <str name="error-class">org.apache.solr.common.SolrException</str>
2025-06-10T17:11:45.111343511Z     <str name="root-error-class">org.apache.solr.common.SolrException</str>
    <str name="error-class">org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException</str>
    <str name="root-error-class">org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException</str>
  </lst>
2025-06-10T17:11:45.111375222Z   <str name="msg">Async exception during distributed update: Error from server at http://ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local:8983/solr: null

2025-06-10T17:11:45.111380282Z 
2025-06-10T17:11:45.111382322Z 
2025-06-10T17:11:45.111385443Z request: http://ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local:8983/solr
Remote error message: version conflict for ess-dive-f0807b9891745e2-20231020T151716445 expected=1834562882620096512 actual=1834562883734732800</str>
  <int name="code">409</int>
2025-06-10T17:11:45.111393003Z </lst>
2025-06-10T17:11:45.111395103Z </response>
 [org.dataone.cn.indexer.solrhttp.HTTPService:sendUpdate:128]
2025-06-10T17:11:48.180122209Z dataone-indexer 20250610-17:11:48: [ERROR]: <?xml version="1.0" encoding="UTF-8"?>
2025-06-10T17:11:48.180144670Z <response>

<lst name="responseHeader">
2025-06-10T17:11:48.180153260Z   <int name="rf">2147483647</int>
  <int name="status">409</int>
2025-06-10T17:11:48.180159370Z   <int name="QTime">393</int>
</lst>
<lst name="error">
2025-06-10T17:11:48.180168111Z   <lst name="metadata">
2025-06-10T17:11:48.180170941Z     <str name="error-class">org.apache.solr.common.SolrException</str>
2025-06-10T17:11:48.180174221Z     <str name="root-error-class">org.apache.solr.common.SolrException</str>
2025-06-10T17:11:48.180177221Z     <str name="error-class">org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException</str>
2025-06-10T17:11:48.180180921Z     <str name="root-error-class">org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException</str>
  </lst>
2025-06-10T17:11:48.180186971Z   <str name="msg">Async exception during distributed update: Error from server at http://ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local:8983/solr: null


2025-06-10T17:11:48.180196202Z 
2025-06-10T17:11:48.180198802Z request: http://ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local:8983/solr
2025-06-10T17:11:48.180201682Z Remote error message: version conflict for ess-dive-8b3d30649d708c7-20221111T171843927 expected=1834562882931523584 actual=1834562884932206592</str>
2025-06-10T17:11:48.180214192Z   <int name="code">409</int>
2025-06-10T17:11:48.180216602Z </lst>
2025-06-10T17:11:48.180218742Z </response>
2025-06-10T17:11:48.180220932Z  [org.dataone.cn.indexer.solrhttp.HTTPService:sendUpdate:128]
dataone-indexer 20250610-17:11:48: [ERROR]: <?xml version="1.0" encoding="UTF-8"?>
2025-06-10T17:11:48.405802769Z <response>
2025-06-10T17:11:48.405809499Z 
2025-06-10T17:11:48.405813339Z <lst name="responseHeader">
2025-06-10T17:11:48.405816759Z   <int name="rf">2147483647</int>
  <int name="status">409</int>
2025-06-10T17:11:48.405824979Z   <int name="QTime">212</int>
2025-06-10T17:11:48.405827659Z </lst>
<lst name="error">
2025-06-10T17:11:48.405832499Z   <lst name="metadata">
2025-06-10T17:11:48.405834810Z     <str name="error-class">org.apache.solr.common.SolrException</str>
    <str name="root-error-class">org.apache.solr.common.SolrException</str>
2025-06-10T17:11:48.405840060Z     <str name="error-class">org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException</str>
    <str name="root-error-class">org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException</str>
2025-06-10T17:11:48.405845830Z   </lst>
  <str name="msg">Async exception during distributed update: Error from server at http://ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local:8983/solr: null

2025-06-10T17:11:48.405853400Z 
2025-06-10T17:11:48.405855830Z 
request: http://ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local:8983/solr
2025-06-10T17:11:48.405860430Z Remote error message: version conflict for ess-dive-f0807b9891745e2-20231020T151716445 expected=1834562883734732800 actual=1834562885000364032</str>
2025-06-10T17:11:48.405863100Z   <int name="code">409</int>
</lst>
2025-06-10T17:11:48.405868180Z </response>
 [org.dataone.cn.indexer.solrhttp.HTTPService:sendUpdate:128]
2025-06-10T17:11:48.405872581Z dataone-indexer 20250610-17:11:48: [ERROR]: <?xml version="1.0" encoding="UTF-8"?>
<response>
2025-06-10T17:11:48.405897651Z 
2025-06-10T17:11:48.405899692Z <lst name="responseHeader">
2025-06-10T17:11:48.405901912Z   <int name="rf">2147483647</int>
2025-06-10T17:11:48.405904132Z   <int name="status">409</int>
2025-06-10T17:11:48.405906352Z   <int name="QTime">156</int>
2025-06-10T17:11:48.405908532Z </lst>
<lst name="error">
2025-06-10T17:11:48.405912782Z   <lst name="metadata">
2025-06-10T17:11:48.405915472Z     <str name="error-class">org.apache.solr.common.SolrException</str>
    <str name="root-error-class">org.apache.solr.common.SolrException</str>
2025-06-10T17:11:48.405924642Z     <str name="error-class">org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException</str>
2025-06-10T17:11:48.405928212Z     <str name="root-error-class">org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException</str>
  </lst>
2025-06-10T17:11:48.405938353Z   <str name="msg">Async exception during distributed update: Error from server at http://ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local:8983/solr: null
2025-06-10T17:11:48.405941513Z 

2025-06-10T17:11:48.405946513Z 
request: http://ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local:8983/solr
2025-06-10T17:11:48.405951933Z Remote error message: version conflict for ess-dive-8b3d30649d708c7-20221111T171843927 expected=1834562882931523584 actual=1834562888065351680</str>
2025-06-10T17:11:48.405954923Z   <int name="code">409</int>
</lst>
</response>
2025-06-10T17:11:48.405962923Z  [org.dataone.cn.indexer.solrhttp.HTTPService:sendUpdate:128]
2025-06-10T17:11:51.768799508Z dataone-indexer 20250610-17:11:51: [ERROR]: <?xml version="1.0" encoding="UTF-8"?>
2025-06-10T17:11:51.768826448Z <response>
2025-06-10T17:11:51.768829598Z 
2025-06-10T17:11:51.768832428Z <lst name="responseHeader">
2025-06-10T17:11:51.768835969Z   <int name="rf">2147483647</int>
2025-06-10T17:11:51.768838418Z   <int name="status">409</int>
2025-06-10T17:11:51.768841159Z   <int name="QTime">109</int>
</lst>
2025-06-10T17:11:51.768846169Z <lst name="error">
2025-06-10T17:11:51.768857129Z   <lst name="metadata">
    <str name="error-class">org.apache.solr.common.SolrException</str>
2025-06-10T17:11:51.768862189Z     <str name="root-error-class">org.apache.solr.common.SolrException</str>
2025-06-10T17:11:51.768864769Z     <str name="error-class">org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException</str>
    <str name="root-error-class">org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException</str>
2025-06-10T17:11:51.768870190Z   </lst>
2025-06-10T17:11:51.768872370Z   <str name="msg">Async exception during distributed update: Error from server at http://ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local:8983/solr: null
2025-06-10T17:11:51.768875200Z 
2025-06-10T17:11:51.768877250Z 
2025-06-10T17:11:51.768879490Z 
2025-06-10T17:11:51.768881780Z request: http://ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local:8983/solr
2025-06-10T17:11:51.768884380Z Remote error message: version conflict for ess-dive-8b3d30649d708c7-20221111T171843927 expected=1834562888065351680 actual=1834562889479880704</str>
2025-06-10T17:11:51.768886810Z   <int name="code">409</int>
2025-06-10T17:11:51.768889290Z </lst>
</response>
2025-06-10T17:11:51.768893770Z  [org.dataone.cn.indexer.solrhttp.HTTPService:sendUpdate:128]
dataone-indexer 20250610-17:11:52: [ERROR]: <?xml version="1.0" encoding="UTF-8"?>
2025-06-10T17:11:52.143072781Z <response>
2025-06-10T17:11:52.143076441Z 
2025-06-10T17:11:52.143080101Z <lst name="responseHeader">
  <int name="rf">2147483647</int>
2025-06-10T17:11:52.143087091Z   <int name="status">409</int>
  <int name="QTime">369</int>
2025-06-10T17:11:52.143093331Z </lst>
2025-06-10T17:11:52.143096102Z <lst name="error">
  <lst name="metadata">
2025-06-10T17:11:52.143101491Z     <str name="error-class">org.apache.solr.common.SolrException</str>
    <str name="root-error-class">org.apache.solr.common.SolrException</str>
2025-06-10T17:11:52.143107422Z     <str name="error-class">org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException</str>
2025-06-10T17:11:52.143110562Z     <str name="root-error-class">org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException</str>
  </lst>
2025-06-10T17:11:52.143129632Z   <str name="msg">Async exception during distributed update: Error from server at http://ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local:8983/solr: null
2025-06-10T17:11:52.143132672Z 
2025-06-10T17:11:52.143135263Z 

2025-06-10T17:11:52.143140023Z request: http://ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local:8983/solr
2025-06-10T17:11:52.143142463Z Remote error message: version conflict for ess-dive-f0807b9891745e2-20231020T151716445 expected=1834562885000364032 actual=1834562889452617728</str>
  <int name="code">409</int>
2025-06-10T17:11:52.143147483Z </lst>
2025-06-10T17:11:52.143149633Z </response>
2025-06-10T17:11:52.143151833Z  [org.dataone.cn.indexer.solrhttp.HTTPService:sendUpdate:128]
2025-06-10T17:11:52.385575169Z dataone-indexer 20250610-17:11:52: [ERROR]: <?xml version="1.0" encoding="UTF-8"?>
<response>
2025-06-10T17:11:52.385601889Z 
2025-06-10T17:11:52.385604719Z <lst name="responseHeader">
  <int name="rf">2147483647</int>
2025-06-10T17:11:52.385610819Z   <int name="status">409</int>
2025-06-10T17:11:52.385614410Z   <int name="QTime">184</int>
</lst>
2025-06-10T17:11:52.385619990Z <lst name="error">
2025-06-10T17:11:52.385622430Z   <lst name="metadata">
2025-06-10T17:11:52.385624710Z     <str name="error-class">org.apache.solr.common.SolrException</str>
2025-06-10T17:11:52.385627440Z     <str name="root-error-class">org.apache.solr.common.SolrException</str>
    <str name="error-class">org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException</str>
2025-06-10T17:11:52.385632420Z     <str name="root-error-class">org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException</str>
2025-06-10T17:11:52.385634900Z   </lst>
  <str name="msg">Async exception during distributed update: Error from server at http://ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local:8983/solr: null
2025-06-10T17:11:52.385640630Z 
2025-06-10T17:11:52.385642731Z 

2025-06-10T17:11:52.385646920Z request: http://ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local:8983/solr
Remote error message: version conflict for ess-dive-8b3d30649d708c7-20221111T171843927 expected=1834562888065351680 actual=1834562889479880704</str>
2025-06-10T17:11:52.385660711Z   <int name="code">409</int>
</lst>
2025-06-10T17:11:52.385665191Z </response>
2025-06-10T17:11:52.385667471Z  [org.dataone.cn.indexer.solrhttp.HTTPService:sendUpdate:128]
dataone-indexer 20250610-17:11:55: [ERROR]: <?xml version="1.0" encoding="UTF-8"?>
2025-06-10T17:11:55.102176808Z <response>
2025-06-10T17:11:55.102179929Z 
<lst name="responseHeader">
2025-06-10T17:11:55.102186398Z   <int name="rf">1</int>
2025-06-10T17:11:55.102189269Z   <int name="status">409</int>
  <int name="QTime">2</int>
2025-06-10T17:11:55.102196809Z </lst>
2025-06-10T17:11:55.102199389Z <lst name="error">
  <lst name="metadata">
2025-06-10T17:11:55.102204519Z     <str name="error-class">org.apache.solr.common.SolrException</str>
2025-06-10T17:11:55.102207209Z     <str name="root-error-class">org.apache.solr.common.SolrException</str>
  </lst>
2025-06-10T17:11:55.102211719Z   <str name="msg">version conflict for ess-dive-8b3d30649d708c7-20221111T171843927 expected=1834562889479880704 actual=1834562893901725696</str>
  <int name="code">409</int>
2025-06-10T17:11:55.102217210Z </lst>
</response>
2025-06-10T17:11:55.102222220Z  [org.dataone.cn.indexer.solrhttp.HTTPService:sendUpdate:128]
2025-06-10T17:11:55.928785959Z dataone-indexer 20250610-17:11:55: [ERROR]: <?xml version="1.0" encoding="UTF-8"?>
<response>
2025-06-10T17:11:55.928816120Z 
<lst name="responseHeader">
2025-06-10T17:11:55.928820740Z   <int name="rf">1</int>
  <int name="status">409</int>
2025-06-10T17:11:55.928825620Z   <int name="QTime">2</int>
2025-06-10T17:11:55.928827961Z </lst>
<lst name="error">
2025-06-10T17:11:55.928832401Z   <lst name="metadata">
    <str name="error-class">org.apache.solr.common.SolrException</str>
2025-06-10T17:11:55.928855492Z     <str name="root-error-class">org.apache.solr.common.SolrException</str>
  </lst>
2025-06-10T17:11:55.928860272Z   <str name="msg">version conflict for ess-dive-8b3d30649d708c7-20221111T171843927 expected=1834562889479880704 actual=1834562893901725696</str>
  <int name="code">409</int>
2025-06-10T17:11:55.928865152Z </lst>
2025-06-10T17:11:55.928867252Z </response>
 [org.dataone.cn.indexer.solrhttp.HTTPService:sendUpdate:128]
2025-06-10T17:11:57.313514753Z dataone-indexer 20250610-17:11:57: [ERROR]: <?xml version="1.0" encoding="UTF-8"?>
2025-06-10T17:11:57.313566024Z <response>

2025-06-10T17:11:57.313572544Z <lst name="responseHeader">
  <int name="rf">1</int>
2025-06-10T17:11:57.313579844Z   <int name="status">409</int>
2025-06-10T17:11:57.313583115Z   <int name="QTime">2</int>
</lst>
2025-06-10T17:11:57.313589445Z <lst name="error">
  <lst name="metadata">
2025-06-10T17:11:57.313595325Z     <str name="error-class">org.apache.solr.common.SolrException</str>
2025-06-10T17:11:57.313599195Z     <str name="root-error-class">org.apache.solr.common.SolrException</str>
  </lst>
  <str name="msg">version conflict for ess-dive-8b3d30649d708c7-20221111T171843927 expected=1834562893901725696 actual=1834562897366220800</str>
2025-06-10T17:11:57.313610645Z   <int name="code">409</int>
2025-06-10T17:11:57.313613355Z </lst>
</response>
2025-06-10T17:11:57.313618945Z  [org.dataone.cn.indexer.solrhttp.HTTPService:sendUpdate:128]
2025-06-10T17:11:58.647303746Z dataone-indexer 20250610-17:11:58: [ERROR]: <?xml version="1.0" encoding="UTF-8"?>
<response>
2025-06-10T17:11:58.647326257Z 
<lst name="responseHeader">
2025-06-10T17:11:58.647332137Z   <int name="rf">1</int>
2025-06-10T17:11:58.647336477Z   <int name="status">409</int>
2025-06-10T17:11:58.647340407Z   <int name="QTime">2</int>
2025-06-10T17:11:58.647343357Z </lst>
2025-06-10T17:11:58.647350618Z <lst name="error">
  <lst name="metadata">
2025-06-10T17:11:58.647355408Z     <str name="error-class">org.apache.solr.common.SolrException</str>
2025-06-10T17:11:58.647358098Z     <str name="root-error-class">org.apache.solr.common.SolrException</str>
2025-06-10T17:11:58.647360478Z   </lst>
2025-06-10T17:11:58.647363328Z   <str name="msg">version conflict for ess-dive-8b3d30649d708c7-20221111T171843927 expected=1834562893901725696 actual=1834562897366220800</str>
  <int name="code">409</int>
2025-06-10T17:11:58.647368858Z </lst>
2025-06-10T17:11:58.647370948Z </response>
 [org.dataone.cn.indexer.solrhttp.HTTPService:sendUpdate:128]
2025-06-10T17:11:59.895603186Z dataone-indexer 20250610-17:11:59: [ERROR]: <?xml version="1.0" encoding="UTF-8"?>
<response>

<lst name="responseHeader">
  <int name="rf">1</int>
2025-06-10T17:11:59.895659888Z   <int name="status">409</int>
2025-06-10T17:11:59.895662348Z   <int name="QTime">2</int>
</lst>
2025-06-10T17:11:59.895667238Z <lst name="error">
2025-06-10T17:11:59.895669868Z   <lst name="metadata">
    <str name="error-class">org.apache.solr.common.SolrException</str>
    <str name="root-error-class">org.apache.solr.common.SolrException</str>
  </lst>
2025-06-10T17:11:59.895679878Z   <str name="msg">version conflict for ess-dive-8b3d30649d708c7-20221111T171843927 expected=1834562897366220800 actual=1834562899610173440</str>
2025-06-10T17:11:59.895682968Z   <int name="code">409</int>
2025-06-10T17:11:59.895685338Z </lst>
2025-06-10T17:11:59.895687469Z </response>
 [org.dataone.cn.indexer.solrhttp.HTTPService:sendUpdate:128]
2025-06-10T17:12:01.188006783Z dataone-indexer 20250610-17:12:01: [ERROR]: <?xml version="1.0" encoding="UTF-8"?>
2025-06-10T17:12:01.188027314Z <response>
2025-06-10T17:12:01.188030474Z 
2025-06-10T17:12:01.188032924Z <lst name="responseHeader">
2025-06-10T17:12:01.188035674Z   <int name="rf">1</int>
2025-06-10T17:12:01.188039204Z   <int name="status">409</int>
2025-06-10T17:12:01.188047364Z   <int name="QTime">0</int>
2025-06-10T17:12:01.188051954Z </lst>
2025-06-10T17:12:01.188054915Z <lst name="error">
  <lst name="metadata">
2025-06-10T17:12:01.188059604Z     <str name="error-class">org.apache.solr.common.SolrException</str>
2025-06-10T17:12:01.188062285Z     <str name="root-error-class">org.apache.solr.common.SolrException</str>
  </lst>
2025-06-10T17:12:01.188067085Z   <str name="msg">version conflict for ess-dive-8b3d30649d708c7-20221111T171843927 expected=1834562899610173440 actual=1834562901217640448</str>
  <int name="code">409</int>
2025-06-10T17:12:01.188072225Z </lst>
2025-06-10T17:12:01.188074365Z </response>
 [org.dataone.cn.indexer.solrhttp.HTTPService:sendUpdate:128]
2025-06-10T17:12:02.666457075Z dataone-indexer 20250610-17:12:02: [ERROR]: <?xml version="1.0" encoding="UTF-8"?>
2025-06-10T17:12:02.666491356Z <response>

2025-06-10T17:12:02.666500596Z <lst name="responseHeader">
2025-06-10T17:12:02.666503796Z   <int name="rf">1</int>
2025-06-10T17:12:02.666507526Z   <int name="status">409</int>
  <int name="QTime">1</int>
2025-06-10T17:12:02.666513886Z </lst>
<lst name="error">
2025-06-10T17:12:02.666519006Z   <lst name="metadata">
2025-06-10T17:12:02.666521596Z     <str name="error-class">org.apache.solr.common.SolrException</str>
    <str name="root-error-class">org.apache.solr.common.SolrException</str>
2025-06-10T17:12:02.666527417Z   </lst>
2025-06-10T17:12:02.666530086Z   <str name="msg">version conflict for ess-dive-f0807b9891745e2-20231020T151716445 expected=1834562900574863360 actual=1834562902906896384</str>
2025-06-10T17:12:02.666533267Z   <int name="code">409</int>
2025-06-10T17:12:02.666536107Z </lst>
</response>
2025-06-10T17:12:02.666541537Z  [org.dataone.cn.indexer.solrhttp.HTTPService:sendUpdate:128]
2025-06-10T17:12:05.372020138Z dataone-indexer 20250610-17:12:05: [ERROR]: <?xml version="1.0" encoding="UTF-8"?>
<response>
2025-06-10T17:12:05.372050839Z 
2025-06-10T17:12:05.372053619Z <lst name="responseHeader">
2025-06-10T17:12:05.372074139Z   <int name="rf">1</int>
2025-06-10T17:12:05.372078150Z   <int name="status">409</int>
2025-06-10T17:12:05.372080900Z   <int name="QTime">2</int>
2025-06-10T17:12:05.372083260Z </lst>
2025-06-10T17:12:05.372085660Z <lst name="error">
  <lst name="metadata">
2025-06-10T17:12:05.372090510Z     <str name="error-class">org.apache.solr.common.SolrException</str>
2025-06-10T17:12:05.372093190Z     <str name="root-error-class">org.apache.solr.common.SolrException</str>
  </lst>
2025-06-10T17:12:05.372098000Z   <str name="msg">version conflict for ess-dive-8b3d30649d708c7-20221111T171843927 expected=1834562902748561408 actual=1834562904329814016</str>
2025-06-10T17:12:05.372100590Z   <int name="code">409</int>
</lst>
2025-06-10T17:12:05.372105240Z </response>
 [org.dataone.cn.indexer.solrhttp.HTTPService:sendUpdate:128]
2025-06-10T17:12:06.861778255Z dataone-indexer 20250610-17:12:06: [ERROR]: <?xml version="1.0" encoding="UTF-8"?>
2025-06-10T17:12:06.861803766Z <response>

2025-06-10T17:12:06.861809556Z <lst name="responseHeader">
2025-06-10T17:12:06.861812006Z   <int name="rf">1</int>
2025-06-10T17:12:06.861815606Z   <int name="status">409</int>
  <int name="QTime">0</int>
</lst>
2025-06-10T17:12:06.861824036Z <lst name="error">
2025-06-10T17:12:06.861827506Z   <lst name="metadata">
2025-06-10T17:12:06.861831587Z     <str name="error-class">org.apache.solr.common.SolrException</str>
2025-06-10T17:12:06.861834207Z     <str name="root-error-class">org.apache.solr.common.SolrException</str>
  </lst>
2025-06-10T17:12:06.861838827Z   <str name="msg">version conflict for ess-dive-f0807b9891745e2-20231020T151716445 expected=1834562905597542400 actual=1834562907627585536</str>
2025-06-10T17:12:06.861841417Z   <int name="code">409</int>
2025-06-10T17:12:06.861843737Z </lst>
2025-06-10T17:12:06.861845907Z </response>
2025-06-10T17:12:06.861848157Z  [org.dataone.cn.indexer.solrhttp.HTTPService:sendUpdate:128]
dataone-indexer 20250610-17:12:09: [ERROR]: <?xml version="1.0" encoding="UTF-8"?>
2025-06-10T17:12:09.177206075Z <response>
2025-06-10T17:12:09.177209425Z 
<lst name="responseHeader">
2025-06-10T17:12:09.177220385Z   <int name="rf">1</int>
2025-06-10T17:12:09.177222855Z   <int name="status">409</int>
  <int name="QTime">1</int>
2025-06-10T17:12:09.177227745Z </lst>
2025-06-10T17:12:09.177229896Z <lst name="error">
  <lst name="metadata">
    <str name="error-class">org.apache.solr.common.SolrException</str>
2025-06-10T17:12:09.177237336Z     <str name="root-error-class">org.apache.solr.common.SolrException</str>
2025-06-10T17:12:09.177239796Z   </lst>
2025-06-10T17:12:09.177242236Z   <str name="msg">version conflict for ess-dive-8b3d30649d708c7-20221111T171843927 expected=1834562904329814016 actual=1834562908846030848</str>
2025-06-10T17:12:09.177244796Z   <int name="code">409</int>
2025-06-10T17:12:09.177247116Z </lst>
</response>
2025-06-10T17:12:09.177251456Z  [org.dataone.cn.indexer.solrhttp.HTTPService:sendUpdate:128]
2025-06-10T17:12:10.638489468Z dataone-indexer 20250610-17:12:10: [ERROR]: <?xml version="1.0" encoding="UTF-8"?>
<response>
2025-06-10T17:12:10.638540019Z 
2025-06-10T17:12:10.638544379Z <lst name="responseHeader">
2025-06-10T17:12:10.638547999Z   <int name="rf">1</int>
  <int name="status">409</int>
2025-06-10T17:12:10.638554890Z   <int name="QTime">0</int>
2025-06-10T17:12:10.638557350Z </lst>
2025-06-10T17:12:10.638559950Z <lst name="error">
2025-06-10T17:12:10.638562310Z   <lst name="metadata">
    <str name="error-class">org.apache.solr.common.SolrException</str>
    <str name="root-error-class">org.apache.solr.common.SolrException</str>
2025-06-10T17:12:10.638570090Z   </lst>
  <str name="msg">version conflict for ess-dive-f0807b9891745e2-20231020T151716445 expected=1834562909196255232 actual=1834562911032311808</str>
2025-06-10T17:12:10.638575490Z   <int name="code">409</int>
2025-06-10T17:12:10.638577900Z </lst>
2025-06-10T17:12:10.638580430Z </response>
 [org.dataone.cn.indexer.solrhttp.HTTPService:sendUpdate:128]
2025-06-10T17:12:12.107856010Z dataone-indexer 20250610-17:12:12: [ERROR]: <?xml version="1.0" encoding="UTF-8"?>
<response>
2025-06-10T17:12:12.107896001Z 
2025-06-10T17:12:12.107898711Z <lst name="responseHeader">
2025-06-10T17:12:12.107901851Z   <int name="rf">1</int>
2025-06-10T17:12:12.107905692Z   <int name="status">409</int>
  <int name="QTime">0</int>
2025-06-10T17:12:12.107911522Z </lst>
2025-06-10T17:12:12.107913972Z <lst name="error">
  <lst name="metadata">
2025-06-10T17:12:12.107918722Z     <str name="error-class">org.apache.solr.common.SolrException</str>
2025-06-10T17:12:12.107921312Z     <str name="root-error-class">org.apache.solr.common.SolrException</str>
  </lst>
  <str name="msg">version conflict for ess-dive-f0807b9891745e2-20231020T151716445 expected=1834562911032311808 actual=1834562913095909376</str>
2025-06-10T17:12:12.107929192Z   <int name="code">409</int>
</lst>
2025-06-10T17:12:12.107934003Z </response>
 [org.dataone.cn.indexer.solrhttp.HTTPService:sendUpdate:128]
2025-06-10T17:12:14.463157173Z dataone-indexer 20250610-17:12:14: [ERROR]: <?xml version="1.0" encoding="UTF-8"?>
<response>

2025-06-10T17:12:14.463191694Z <lst name="responseHeader">
2025-06-10T17:12:14.463195374Z   <int name="rf">1</int>
  <int name="status">409</int>
2025-06-10T17:12:14.463202114Z   <int name="QTime">2</int>
</lst>
2025-06-10T17:12:14.463207464Z <lst name="error">
  <lst name="metadata">
2025-06-10T17:12:14.463212954Z     <str name="error-class">org.apache.solr.common.SolrException</str>
2025-06-10T17:12:14.463216024Z     <str name="root-error-class">org.apache.solr.common.SolrException</str>
2025-06-10T17:12:14.463218684Z   </lst>
2025-06-10T17:12:14.463221364Z   <str name="msg">version conflict for ess-dive-8b3d30649d708c7-20221111T171843927 expected=1834562911834472448 actual=1834562914449620992</str>
  <int name="code">409</int>
2025-06-10T17:12:14.463236915Z </lst>
</response>
2025-06-10T17:12:14.463241465Z  [org.dataone.cn.indexer.solrhttp.HTTPService:sendUpdate:128]
dataone-indexer 20250610-17:12:17: [ERROR]: <?xml version="1.0" encoding="UTF-8"?>
<response>
2025-06-10T17:12:17.995794762Z 
2025-06-10T17:12:17.995797622Z <lst name="responseHeader">
  <int name="rf">1</int>
2025-06-10T17:12:17.995805572Z   <int name="status">409</int>
  <int name="QTime">1</int>
2025-06-10T17:12:17.995812062Z </lst>
2025-06-10T17:12:17.995815072Z <lst name="error">
2025-06-10T17:12:17.995817952Z   <lst name="metadata">
    <str name="error-class">org.apache.solr.common.SolrException</str>
2025-06-10T17:12:17.995823822Z     <str name="root-error-class">org.apache.solr.common.SolrException</str>
2025-06-10T17:12:17.995826823Z   </lst>
  <str name="msg">version conflict for ess-dive-8b3d30649d708c7-20221111T171843927 expected=1834562914449620992 actual=1834562916777459712</str>
2025-06-10T17:12:17.995832543Z   <int name="code">409</int>
2025-06-10T17:12:17.995835553Z </lst>
</response>
2025-06-10T17:12:17.995840963Z  [org.dataone.cn.indexer.solrhttp.HTTPService:sendUpdate:128]

[vchendrix]

@artntek @taojing2002 @mbjones I forced my replica nodes to sync with the leader nodes (there are three total) and I see the following in the logs. This indicates that the nodes are not out of sync. So why do I still get version conflict errors when I try to reindex?

Replica

2025-06-10 21:30:56.705 INFO  (qtp26757919-20-ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local-126983) [c:dataone-index s:shard1 r:core_node5 x:dataone-index_shard1_replica_n2 t:ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local-126983] o.a.s.u.PeerSync PeerSync: core=dataone-index_shard1_replica_n2 url=http://ess-dive-solr-20250528-1.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local:8983/solr  START replicas=[http://ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local:8983/solr/dataone-index_shard1_replica_n1/] nUpdates=100
2025-06-10 21:30:56.712 INFO  (qtp26757919-20-ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local-126983) [c:dataone-index s:shard1 r:core_node5 x:dataone-index_shard1_replica_n2 t:ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local-126983] o.a.s.u.PeerSync We are already in sync. No need to do a PeerSync 
2025-06-10 21:30:56.712 INFO  (qtp26757919-20-ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local-126983) [c:dataone-index s:shard1 r:core_node5 x:dataone-index_shard1_replica_n2 t:ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local-126983] o.a.s.c.S.Request webapp=/solr path=/get params={distrib=false&qt=/get&getVersions=100&sync=http://ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local:8983/solr/dataone-index_shard1_replica_n1/&wt=javabin&version=2} status=0 QTime=8

Replica

2025-06-10 21:30:56.705 INFO  (qtp26757919-295-ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local-126983) [c:dataone-index s:shard1 r:core_node6 x:dataone-index_shard1_replica_n4 t:ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local-126983] o.a.s.u.PeerSync PeerSync: core=dataone-index_shard1_replica_n4 url=http://ess-dive-solr-20250528-0.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local:8983/solr  START replicas=[http://ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local:8983/solr/dataone-index_shard1_replica_n1/] nUpdates=100
2025-06-10 21:30:56.712 INFO  (qtp26757919-295-ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local-126983) [c:dataone-index s:shard1 r:core_node6 x:dataone-index_shard1_replica_n4 t:ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local-126983] o.a.s.u.PeerSync We are already in sync. No need to do a PeerSync 

Leader

2025-06-10 21:30:56.697 INFO  (qtp26757919-18-ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local-126983) [c: s: r: x:dataone-index_shard1_replica_n1 t:ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local-126983] o.a.s.h.a.RequestSyncShardOp I have been requested to sync up my shard
2025-06-10 21:30:56.713 INFO  (qtp26757919-18-ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local-126983) [c:dataone-index s:shard1 r:core_node3 x:dataone-index_shard1_replica_n1 t:ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local-126983] o.a.s.c.SyncStrategy http://ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local:8983/solr/dataone-index_shard1_replica_n1/: sync completed with http://ess-dive-solr-20250528-1.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local:8983/solr/dataone-index_shard1_replica_n2
2025-06-10T21:30:56.713624979Z 2025-06-10 21:30:56.713 INFO  (qtp26757919-18-ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local-126983) [c:dataone-index s:shard1 r:core_node3 x:dataone-index_shard1_replica_n1 t:ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local-126983] o.a.s.c.SyncStrategy http://ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local:8983/solr/dataone-index_shard1_replica_n1/: sync completed with http://ess-dive-solr-20250528-0.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local:8983/solr/dataone-index_shard1_replica_n4
2025-06-10T21:30:56.713640820Z 2025-06-10 21:30:56.713 INFO  (qtp26757919-18-ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local-126983) [c: s: r: x: t:ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local-126983] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={core=dataone-index_shard1_replica_n1&action=REQUESTSYNCSHARD&shard=shard1&collection=dataone-index&wt=javabin&version=2} status=0 QTime=16
2025-06-10T21:30:56.714362747Z 2025-06-10 21:30:56.714 INFO  (qtp26757919-21-ess-dive-solr-20250528-2.ess-dive-solr-20250528-headless.essdive-production.svc.cluster.local-12698

I tried to index the version previous to the current version and got (note that it skipped the merge due to the sid being a different version)

dataone-indexer 20250610-22:01:58: [INFO]: IndexWorker.consumer.indexObject by multiple thread? true, with the thread id 44 - Received the index task from the index queue with the identifier: ess-dive-c581eebb5b7b50a-20250428T213257013 , the index type: create, the priority: 3 [org.dataone.cn.indexer.IndexWorker:indexObject:454]
dataone-indexer 20250610-22:01:58: [INFO]: Retrieved metadata for pid: ess-dive-c581eebb5b7b50a-20250428T213257013 with formatId: https://ns.dataone.org/service/types/v2.0#SystemMetadata [org.dataone.hashstore.filehashstore.FileHashStore:getHashStoreMetadataInputStream:2208]
dataone-indexer 20250610-22:01:58: [INFO]: Finish getting the system metadata via the file system for the pid ess-dive-c581eebb5b7b50a-20250428T213257013 and it took 31milliseconds [org.dataone.cn.indexer.object.ObjectManager:getSystemMetadataStream:99]
dataone-indexer 20250610-22:01:58: [INFO]: entering refreshCache()... [org.dataone.client.v2.formats.ObjectFormatCache:refreshCache:181]
2025-06-10T22:01:58.101123143Z dataone-indexer 20250610-22:01:58: [INFO]: entering refreshCache()... [org.dataone.client.v2.formats.ObjectFormatCache:refreshCache:181]
dataone-indexer 20250610-22:01:58: [INFO]: selectSession: using the default certificate location [org.dataone.client.auth.CertificateManager:getCertificateFile:1144]
2025-06-10T22:01:58.102871346Z dataone-indexer 20250610-22:01:58: [INFO]: selectSession: using the default certificate location [org.dataone.client.auth.CertificateManager:getCertificateFile:1144]
dataone-indexer 20250610-22:01:58: [INFO]: Did not find a certificate for the subject specified: null [org.dataone.client.auth.CertificateManager:selectSession:1134]
2025-06-10T22:01:58.106377072Z dataone-indexer 20250610-22:01:58: [INFO]: Did not find a certificate for the subject specified: null [org.dataone.client.auth.CertificateManager:selectSession:1134]
2025-06-10T22:01:58.106739043Z dataone-indexer 20250610-22:01:58: [INFO]: ...trying SSLContext protocol: TLSv1.2 [org.dataone.client.auth.CertificateManager:buildSSLContext:858]
2025-06-10T22:01:58.106767234Z dataone-indexer 20250610-22:01:58: [INFO]: ...trying SSLContext protocol: TLSv1.2 [org.dataone.client.auth.CertificateManager:buildSSLContext:858]
2025-06-10T22:01:58.107176336Z dataone-indexer 20250610-22:01:58: [INFO]: ...setting SSLContext with protocol: TLSv1.2 [org.dataone.client.auth.CertificateManager:buildSSLContext:860]
2025-06-10T22:01:58.107225038Z dataone-indexer 20250610-22:01:58: [INFO]: ...setting SSLContext with protocol: TLSv1.2 [org.dataone.client.auth.CertificateManager:buildSSLContext:860]
dataone-indexer 20250610-22:01:58: [INFO]: creating custom TrustManager [org.dataone.client.auth.CertificateManager:getTrustManager:1008]
2025-06-10T22:01:58.107619400Z dataone-indexer 20250610-22:01:58: [INFO]: creating custom TrustManager [org.dataone.client.auth.CertificateManager:getTrustManager:1008]
2025-06-10T22:01:58.108118705Z dataone-indexer 20250610-22:01:58: [INFO]: getSSLConnectionSocketFactory: using allow-all hostname verifier [org.dataone.client.auth.CertificateManager:getSSLConnectionSocketFactory:958]
dataone-indexer 20250610-22:01:58: [INFO]: getSSLConnectionSocketFactory: using allow-all hostname verifier [org.dataone.client.auth.CertificateManager:getSSLConnectionSocketFactory:958]
dataone-indexer 20250610-22:01:58: [WARN]: registering ConnectionManager... [org.dataone.client.utils.HttpConnectionMonitorService:addMonitor:65]
2025-06-10T22:01:58.108787295Z dataone-indexer 20250610-22:01:58: [WARN]: registering ConnectionManager... [org.dataone.client.utils.HttpConnectionMonitorService:addMonitor:65]
dataone-indexer 20250610-22:01:59: [INFO]: RestClient.doRequestNoBody, thread(44) call Info: GET https://cn-stage.test.dataone.org/cn/v2/node [org.dataone.client.rest.RestClient:doRequestNoBody:268]
dataone-indexer 20250610-22:01:59: [INFO]: RestClient.doRequestNoBody, thread(44) call Info: GET https://cn-stage.test.dataone.org/cn/v2/node [org.dataone.client.rest.RestClient:doRequestNoBody:268]
dataone-indexer 20250610-22:01:59: [INFO]: response httpCode: 200 [org.dataone.service.util.ExceptionHandler:filterErrors:93]
dataone-indexer 20250610-22:01:59: [INFO]: refreshing objectFormatCache from cn: null [org.dataone.client.v2.formats.ObjectFormatCache:refreshCache:196]
2025-06-10T22:01:59.356173395Z dataone-indexer 20250610-22:01:59: [INFO]: refreshing objectFormatCache from cn: null [org.dataone.client.v2.formats.ObjectFormatCache:refreshCache:196]
dataone-indexer 20250610-22:01:59: [INFO]: RestClient.doRequestNoBody, thread(44) call Info: GET https://cn-stage.test.dataone.org/cn/v2/formats [org.dataone.client.rest.RestClient:doRequestNoBody:268]
2025-06-10T22:01:59.688333086Z dataone-indexer 20250610-22:01:59: [INFO]: RestClient.doRequestNoBody, thread(44) call Info: GET https://cn-stage.test.dataone.org/cn/v2/formats [org.dataone.client.rest.RestClient:doRequestNoBody:268]
dataone-indexer 20250610-22:01:59: [INFO]: response httpCode: 200 [org.dataone.service.util.ExceptionHandler:filterErrors:93]
dataone-indexer 20250610-22:01:59: [INFO]: successfully refreshed cache from cn.listFormats() [org.dataone.client.v2.formats.ObjectFormatCache:refreshCache:231]
2025-06-10T22:01:59.705997350Z dataone-indexer 20250610-22:01:59: [INFO]: successfully refreshed cache from cn.listFormats() [org.dataone.client.v2.formats.ObjectFormatCache:refreshCache:231]
dataone-indexer 20250610-22:01:59: [INFO]: Retrieved object for pid: ess-dive-c581eebb5b7b50a-20250428T213257013 [org.dataone.hashstore.filehashstore.FileHashStore:retrieveObject:650]
dataone-indexer 20250610-22:01:59: [INFO]: Retrieved metadata for pid: ess-dive-9be051bcebbb9e6-20250428T213257023 with formatId: https://ns.dataone.org/service/types/v2.0#SystemMetadata [org.dataone.hashstore.filehashstore.FileHashStore:getHashStoreMetadataInputStream:2208]
2025-06-10T22:01:59.796343632Z dataone-indexer 20250610-22:01:59: [INFO]: Finish getting the system metadata via the file system for the pid ess-dive-9be051bcebbb9e6-20250428T213257023 and it took 19milliseconds [org.dataone.cn.indexer.object.ObjectManager:getSystemMetadataStream:99]
dataone-indexer 20250610-22:01:59: [INFO]: Retrieved metadata for pid: ess-dive-9be051bcebbb9e6-20250428T213257023 with formatId: https://ns.dataone.org/service/types/v2.0#SystemMetadata [org.dataone.hashstore.filehashstore.FileHashStore:getHashStoreMetadataInputStream:2208]
dataone-indexer 20250610-22:01:59: [INFO]: Finish getting the system metadata via the file system for the pid ess-dive-9be051bcebbb9e6-20250428T213257023 and it took 3milliseconds [org.dataone.cn.indexer.object.ObjectManager:getSystemMetadataStream:99]
dataone-indexer 20250610-22:01:59: [INFO]: Retrieved metadata for pid: ess-dive-ad64024019400c7-20250424T205626697 with formatId: https://ns.dataone.org/service/types/v2.0#SystemMetadata [org.dataone.hashstore.filehashstore.FileHashStore:getHashStoreMetadataInputStream:2208]
2025-06-10T22:01:59.860441559Z dataone-indexer 20250610-22:01:59: [INFO]: Finish getting the system metadata via the file system for the pid ess-dive-ad64024019400c7-20250424T205626697 and it took 22milliseconds [org.dataone.cn.indexer.object.ObjectManager:getSystemMetadataStream:99]
dataone-indexer 20250610-22:01:59: [INFO]: Retrieved metadata for pid: ess-dive-9be051bcebbb9e6-20250428T213257023 with formatId: https://ns.dataone.org/service/types/v2.0#SystemMetadata [org.dataone.hashstore.filehashstore.FileHashStore:getHashStoreMetadataInputStream:2208]
dataone-indexer 20250610-22:01:59: [INFO]: Finish getting the system metadata via the file system for the pid ess-dive-9be051bcebbb9e6-20250428T213257023 and it took 2milliseconds [org.dataone.cn.indexer.object.ObjectManager:getSystemMetadataStream:99]
dataone-indexer 20250610-22:01:59: [INFO]: Retrieved metadata for pid: ess-dive-9be051bcebbb9e6-20250428T213257023 with formatId: https://ns.dataone.org/service/types/v2.0#SystemMetadata [org.dataone.hashstore.filehashstore.FileHashStore:getHashStoreMetadataInputStream:2208]
dataone-indexer 20250610-22:01:59: [INFO]: Finish getting the system metadata via the file system for the pid ess-dive-9be051bcebbb9e6-20250428T213257023 and it took 2milliseconds [org.dataone.cn.indexer.object.ObjectManager:getSystemMetadataStream:99]
dataone-indexer 20250610-22:01:59: [INFO]: Retrieved metadata for pid: ess-dive-9be051bcebbb9e6-20250428T213257023 with formatId: https://ns.dataone.org/service/types/v2.0#SystemMetadata [org.dataone.hashstore.filehashstore.FileHashStore:getHashStoreMetadataInputStream:2208]
dataone-indexer 20250610-22:01:59: [INFO]: Finish getting the system metadata via the file system for the pid ess-dive-9be051bcebbb9e6-20250428T213257023 and it took 2milliseconds [org.dataone.cn.indexer.object.ObjectManager:getSystemMetadataStream:99]
dataone-indexer 20250610-22:01:59: [INFO]: Retrieved metadata for pid: ess-dive-ad64024019400c7-20250424T205626697 with formatId: https://ns.dataone.org/service/types/v2.0#SystemMetadata [org.dataone.hashstore.filehashstore.FileHashStore:getHashStoreMetadataInputStream:2208]
dataone-indexer 20250610-22:01:59: [INFO]: Finish getting the system metadata via the file system for the pid ess-dive-ad64024019400c7-20250424T205626697 and it took 2milliseconds [org.dataone.cn.indexer.object.ObjectManager:getSystemMetadataStream:99]
dataone-indexer 20250610-22:01:59: [INFO]: Retrieved metadata for pid: ess-dive-ad64024019400c7-20250424T205626697 with formatId: https://ns.dataone.org/service/types/v2.0#SystemMetadata [org.dataone.hashstore.filehashstore.FileHashStore:getHashStoreMetadataInputStream:2208]
dataone-indexer 20250610-22:01:59: [INFO]: Finish getting the system metadata via the file system for the pid ess-dive-ad64024019400c7-20250424T205626697 and it took 2milliseconds [org.dataone.cn.indexer.object.ObjectManager:getSystemMetadataStream:99]
dataone-indexer 20250610-22:01:59: [INFO]: Retrieved metadata for pid: ess-dive-9be051bcebbb9e6-20250428T213257023 with formatId: https://ns.dataone.org/service/types/v2.0#SystemMetadata [org.dataone.hashstore.filehashstore.FileHashStore:getHashStoreMetadataInputStream:2208]
dataone-indexer 20250610-22:01:59: [INFO]: Finish getting the system metadata via the file system for the pid ess-dive-9be051bcebbb9e6-20250428T213257023 and it took 3milliseconds [org.dataone.cn.indexer.object.ObjectManager:getSystemMetadataStream:99]
dataone-indexer 20250610-22:01:59: [INFO]: SeriesIdResolver.getPid - the current node type is null [org.dataone.cn.indexer.parser.utility.SeriesIdResolver:getPid:49]
dataone-indexer 20250610-22:01:59: [INFO]: SeriesIdResolver.getPid - get the system metadata from the mn base urlhttp://metacat:8080/catalog/d1/mn for the object doi:10.15485/1898912 [org.dataone.cn.indexer.parser.utility.SeriesIdResolver:getPid:53]
2025-06-10T22:01:59.961215026Z dataone-indexer 20250610-22:01:59: [INFO]: selectSession: using the default certificate location [org.dataone.client.auth.CertificateManager:getCertificateFile:1144]
dataone-indexer 20250610-22:01:59: [INFO]: selectSession: using the default certificate location [org.dataone.client.auth.CertificateManager:getCertificateFile:1144]
2025-06-10T22:01:59.963583988Z dataone-indexer 20250610-22:01:59: [INFO]: Did not find a certificate for the subject specified: null [org.dataone.client.auth.CertificateManager:selectSession:1134]
2025-06-10T22:01:59.963683371Z dataone-indexer 20250610-22:01:59: [INFO]: Did not find a certificate for the subject specified: null [org.dataone.client.auth.CertificateManager:selectSession:1134]
2025-06-10T22:01:59.963937698Z dataone-indexer 20250610-22:01:59: [INFO]: ...trying SSLContext protocol: TLSv1.2 [org.dataone.client.auth.CertificateManager:buildSSLContext:858]
2025-06-10T22:01:59.963951768Z dataone-indexer 20250610-22:01:59: [INFO]: ...trying SSLContext protocol: TLSv1.2 [org.dataone.client.auth.CertificateManager:buildSSLContext:858]
dataone-indexer 20250610-22:01:59: [INFO]: ...setting SSLContext with protocol: TLSv1.2 [org.dataone.client.auth.CertificateManager:buildSSLContext:860]
2025-06-10T22:01:59.964352981Z dataone-indexer 20250610-22:01:59: [INFO]: ...setting SSLContext with protocol: TLSv1.2 [org.dataone.client.auth.CertificateManager:buildSSLContext:860]
dataone-indexer 20250610-22:01:59: [INFO]: creating custom TrustManager [org.dataone.client.auth.CertificateManager:getTrustManager:1008]
2025-06-10T22:01:59.964598578Z dataone-indexer 20250610-22:01:59: [INFO]: creating custom TrustManager [org.dataone.client.auth.CertificateManager:getTrustManager:1008]
dataone-indexer 20250610-22:01:59: [INFO]: getSSLConnectionSocketFactory: using allow-all hostname verifier [org.dataone.client.auth.CertificateManager:getSSLConnectionSocketFactory:958]
2025-06-10T22:01:59.964898727Z dataone-indexer 20250610-22:01:59: [INFO]: getSSLConnectionSocketFactory: using allow-all hostname verifier [org.dataone.client.auth.CertificateManager:getSSLConnectionSocketFactory:958]
2025-06-10T22:01:59.965627949Z dataone-indexer 20250610-22:01:59: [WARN]: registering ConnectionManager... [org.dataone.client.utils.HttpConnectionMonitorService:addMonitor:65]
dataone-indexer 20250610-22:01:59: [WARN]: registering ConnectionManager... [org.dataone.client.utils.HttpConnectionMonitorService:addMonitor:65]
dataone-indexer 20250610-22:02:00: [INFO]: RestClient.doRequestNoBody, thread(44) call Info: GET http://metacat:8080/catalog/d1/mn/v2/meta/doi:10.15485%2F1898912 [org.dataone.client.rest.RestClient:doRequestNoBody:268]
2025-06-10T22:02:00.328584872Z dataone-indexer 20250610-22:02:00: [INFO]: RestClient.doRequestNoBody, thread(44) call Info: GET http://metacat:8080/catalog/d1/mn/v2/meta/doi:10.15485%2F1898912 [org.dataone.client.rest.RestClient:doRequestNoBody:268]
2025-06-10T22:02:00.328599472Z dataone-indexer 20250610-22:02:00: [INFO]: response httpCode: 200 [org.dataone.service.util.ExceptionHandler:filterErrors:93]
dataone-indexer 20250610-22:02:00: [INFO]: ||||||||||||||||||| the head version is ess-dive-9be051bcebbb9e6-20250428T213324618 for sid doi:10.15485/1898912 [org.dataone.cn.indexer.resourcemap.ForesiteResourceMap:isHeadVersion:214]
2025-06-10T22:02:00.332082207Z dataone-indexer 20250610-22:02:00: [INFO]: ||||||||||||||||||| the pid ess-dive-9be051bcebbb9e6-20250428T213257023 is NOT the head version for sid doi:10.15485/1898912 [org.dataone.cn.indexer.resourcemap.ForesiteResourceMap:isHeadVersion:221]
dataone-indexer 20250610-22:02:00: [INFO]: The id org.dataone.service.types.v1.Identifier@98e19f0a is not the head of the serial id doi:10.15485/1898912 So, skip merge this one!!!!!!!!!!!!!!!!!!!!!!ess-dive-9be051bcebbb9e6-20250428T213257023 [org.dataone.cn.indexer.resourcemap.ForesiteResourceMap:_mergeMappedReference:248]
dataone-indexer 20250610-22:02:00: [INFO]: Retrieved metadata for pid: ess-dive-ad64024019400c7-20250424T205626697 with formatId: https://ns.dataone.org/service/types/v2.0#SystemMetadata [org.dataone.hashstore.filehashstore.FileHashStore:getHashStoreMetadataInputStream:2208]
2025-06-10T22:02:00.335448129Z dataone-indexer 20250610-22:02:00: [INFO]: Finish getting the system metadata via the file system for the pid ess-dive-ad64024019400c7-20250424T205626697 and it took 3milliseconds [org.dataone.cn.indexer.object.ObjectManager:getSystemMetadataStream:99]
dataone-indexer 20250610-22:02:00: [INFO]: SolrIndex.process - the time for calling processDocument for the subprocessor org.dataone.cn.indexer.parser.ResourceMapSubprocessor for the pid ess-dive-c581eebb5b7b50a-20250428T213257013 is 587milliseconds. [org.dataone.cn.indexer.SolrIndex:process:199]
dataone-indexer 20250610-22:02:00: [INFO]: Retrieved object for pid: ess-dive-c581eebb5b7b50a-20250428T213257013 [org.dataone.hashstore.filehashstore.FileHashStore:retrieveObject:650]
dataone-indexer 20250610-22:02:00: [INFO]: TripleStoreService.getDataset - the store directory  ================= is /etc/dataone/tdb-cache/tdb17388372548876149407 [org.dataone.cn.indexer.annotation.TripleStoreService:getDataset:43]
dataone-indexer 20250610-22:02:00: [INFO]: SolrIndex.process - the time for calling processDocument for the subprocessor org.dataone.cn.indexer.annotation.RdfXmlSubprocessor for the pid ess-dive-c581eebb5b7b50a-20250428T213257013 is 378milliseconds. [org.dataone.cn.indexer.SolrIndex:process:199]
dataone-indexer 20250610-22:02:00: [INFO]: SolrIndex.insert - the subprocessor processing time of ess-dive-c581eebb5b7b50a-20250428T213257013 is 2727 milliseconds. [org.dataone.cn.indexer.SolrIndex:insert:334]
dataone-indexer 20250610-22:02:00: [INFO]: Creating HTTP Output Stream for 1 docs:  [org.dataone.cn.indexer.solrhttp.OutputStreamHttpEntity:writeTo:94]
dataone-indexer 20250610-22:02:00: [INFO]: OutputStreamHttpEntity.writeTo - after put the the solr doc into the http output stream. [org.dataone.cn.indexer.solrhttp.OutputStreamHttpEntity:writeTo:98]
dataone-indexer 20250610-22:02:01: [INFO]: HTTPService.sendUpdate - after get the http response entity. [org.dataone.cn.indexer.solrhttp.HTTPService:sendUpdate:122]
dataone-indexer 20250610-22:02:01: [INFO]: Creating HTTP Output Stream for 1 docs:  [org.dataone.cn.indexer.solrhttp.OutputStreamHttpEntity:writeTo:94]
dataone-indexer 20250610-22:02:01: [INFO]: OutputStreamHttpEntity.writeTo - after put the the solr doc into the http output stream. [org.dataone.cn.indexer.solrhttp.OutputStreamHttpEntity:writeTo:98]
dataone-indexer 20250610-22:02:01: [INFO]: HTTPService.sendUpdate - after get the http response entity. [org.dataone.cn.indexer.solrhttp.HTTPService:sendUpdate:122]
dataone-indexer 20250610-22:02:01: [INFO]: Creating HTTP Output Stream for 1 docs:  [org.dataone.cn.indexer.solrhttp.OutputStreamHttpEntity:writeTo:94]
dataone-indexer 20250610-22:02:01: [INFO]: OutputStreamHttpEntity.writeTo - after put the the solr doc into the http output stream. [org.dataone.cn.indexer.solrhttp.OutputStreamHttpEntity:writeTo:98]
dataone-indexer 20250610-22:02:01: [INFO]: HTTPService.sendUpdate - after get the http response entity. [org.dataone.cn.indexer.solrhttp.HTTPService:sendUpdate:122]
dataone-indexer 20250610-22:02:01: [INFO]: SolrIndex.insert - finished to insert the solrDoc to the solr server for  object ess-dive-c581eebb5b7b50a-20250428T213257013 and it took 745 milliseconds. [org.dataone.cn.indexer.SolrIndex:insert:351]
2025-06-10T22:02:01.482640510Z dataone-indexer 20250610-22:02:01: [INFO]: SolrIndex.update - successfully inserted the solr index of the object ess-dive-c581eebb5b7b50a-20250428T213257013 [org.dataone.cn.indexer.SolrIndex:update:440]
dataone-indexer 20250610-22:02:01: [INFO]: IndexWorker.indexOjbect with the thread id 44 - Completed the index task from the index queue with the identifier: ess-dive-c581eebb5b7b50a-20250428T213257013 , the index type: create, the priority: 3 and the time taking is 3481 milliseconds [org.dataone.cn.indexer.IndexWorker:indexObject:475]

I submitted all unindexed resource maps for this dataset and got the attached indexer.log.

Pid:ess-dive-c581eebb5b7b50a-20250428T213257013 Status Code: 200, Response:  <?xml version="1.0" encoding="UTF-8"?>     <scheduled>true</scheduled>
Pid:ess-dive-841f3051d9a2708-20250424T205756612 Status Code: 200, Response:  <?xml version="1.0" encoding="UTF-8"?>     <scheduled>true</scheduled>
Pid:ess-dive-ce2dc192547db4e-20240819T204345615 Status Code: 200, Response:  <?xml version="1.0" encoding="UTF-8"?>     <scheduled>true</scheduled>
Pid:ess-dive-30182ebce4568fe-20240314T204040214 Status Code: 200, Response:  <?xml version="1.0" encoding="UTF-8"?>     <scheduled>true</scheduled>
Pid:ess-dive-b6e3586c57f24e8-20240108T174838956 Status Code: 200, Response:  <?xml version="1.0" encoding="UTF-8"?>     <scheduled>true</scheduled>
Pid:ess-dive-7095e0ccee0ba15-20231108T174335337 Status Code: 200, Response:  <?xml version="1.0" encoding="UTF-8"?>     <scheduled>true</scheduled>
Pid:ess-dive-a4c4b6d97257110-20231108T162220664 Status Code: 200, Response:  <?xml version="1.0" encoding="UTF-8"?>     <scheduled>true</scheduled>
Pid:ess-dive-2cb51a443565303-20231107T003557339 Status Code: 200, Response:  <?xml version="1.0" encoding="UTF-8"?>     <scheduled>true</scheduled>
Pid:ess-dive-e4959e48bc0d436-20231107T000616813 Status Code: 200, Response:  <?xml version="1.0" encoding="UTF-8"?>     <scheduled>true</scheduled>
Pid:ess-dive-c017ba5643cb046-20231106T235858715 Status Code: 200, Response:  <?xml version="1.0" encoding="UTF-8"?>     <scheduled>true</scheduled>
Pid:ess-dive-89c84faea686d5f-20231106T235820604 Status Code: 200, Response:  <?xml version="1.0" encoding="UTF-8"?>     <scheduled>true</scheduled>
Pid:ess-dive-677c91786303c4b-20231106T235015584 Status Code: 200, Response:  <?xml version="1.0" encoding="UTF-8"?>     <scheduled>true</scheduled>
Pid:ess-dive-7fda0c47b8e1f2f-20231106T231212952 Status Code: 200, Response:  <?xml version="1.0" encoding="UTF-8"?>     <scheduled>true</scheduled>
Pid:ess-dive-0ee949f8376453a-20231106T231106507 Status Code: 200, Response:  <?xml version="1.0" encoding="UTF-8"?>     <scheduled>true</scheduled>
Pid:ess-dive-e1a0ec7bfee78f1-20231102T191507376 Status Code: 200, Response:  <?xml version="1.0" encoding="UTF-8"?>     <scheduled>true</scheduled>
Pid:ess-dive-22443e0250facf3-20231024T165344094 Status Code: 200, Response:  <?xml version="1.0" encoding="UTF-8"?>     <scheduled>true</scheduled>
Pid:ess-dive-da805827db18fb5-20231020T171513122 Status Code: 200, Response:  <?xml version="1.0" encoding="UTF-8"?>     <scheduled>true</scheduled>
Pid:ess-dive-e5fa24875975767-20231020T151718932 Status Code: 200, Response:  <?xml version="1.0" encoding="UTF-8"?>     <scheduled>true</scheduled>
Pid:ess-dive-8e968f0b58ae787-20231020T151546863 Status Code: 200, Response:  <?xml version="1.0" encoding="UTF-8"?>     <scheduled>true</scheduled>
Pid:ess-dive-288593f8d05d0a8-20231020T151029850 Status Code: 200, Response:  <?xml version="1.0" encoding="UTF-8"?>     <scheduled>true</scheduled>
Pid:ess-dive-4d8dbc35b605239-20230509T160352683 Status Code: 200, Response:  <?xml version="1.0" encoding="UTF-8"?>     <scheduled>true</scheduled>
Pid:ess-dive-23d73b4b474f11e-20230504T212129266471 Status Code: 200, Response:  <?xml version="1.0" encoding="UTF-8"?>     <scheduled>true</scheduled>
Pid:ess-dive-140808a28a9d9fc-20230407T152434860390 Status Code: 200, Response:  <?xml version="1.0" encoding="UTF-8"?>     <scheduled>true</scheduled>
Pid:ess-dive-c28f0962d002526-20230406T135302307466 Status Code: 200, Response:  <?xml version="1.0" encoding="UTF-8"?>     <scheduled>true</scheduled>
Pid:ess-dive-8b19a68c5d29e91-20230406T122700866201 Status Code: 200, Response:  <?xml version="1.0" encoding="UTF-8"?>     <scheduled>true</scheduled>
Pid:ess-dive-b6c52d4c0d5a817-20221205T175525476 Status Code: 200, Response:  <?xml version="1.0" encoding="UTF-8"?>     <scheduled>true</scheduled>
Pid:ess-dive-46903e0c9c3de02-20221117T224253475 Status Code: 200, Response:  <?xml version="1.0" encoding="UTF-8"?>     <scheduled>true</scheduled>

[taojing2002]

As I mentioned in Slack, dataone-indexer logs the error even though it would try again. The real error message for the failure of reindex is: ``` log.error("SolrIndex.update - Indexer grabbed an older version of " + "a solr doc when it processed object " + pid.getValue() + ". However, Metacat already tried the max times - " + VERSION_CONFLICT_MAX_ATTEMPTS + " and still can't fix the issue."); ``` By the way, were those pids eventually indexed or not?

[taojing2002]

I talked with Val and she said those pids were indexed successfully. Those version conflict errors are red herrings. Please see this ticket

[vchendrix]

I talked with Val and she said those pids were indexed successfully. Those version conflict errors are red herrings. Please see this ticket

That is not exactly what I said. I said that I ran a check on solr to update the nodes but none of the nodes were out of sync. I would not know if the data objects were indexed successfully or not because I saw these messages on reindex all. So they could have been or not. I don't know for sure.

[vchendrix]

@taojing2002 @artntek This is still an issue and should be reopened. The issue is not a version conflict it is the fact that datasets with seriesId are not indexing older version of the datasets. Please read the description

[vchendrix]

@taojing2002 @artntek Note that this is the crux of the issue

[... additional log lines omitted for brevity ...]
dataone-indexer 20250610-15:44:50: [INFO]: SeriesIdResolver.getPid - get the system metadata from the mn base url http://metacat-hl:8080/catalog/d1/mn for the object doi:10.15485/1898912 [org.dataone.cn.indexer.parser.utility.SeriesIdResolver:getPid:53]
dataone-indexer 20250610-15:44:50: [WARN]: SeriesIdResolver.getPid - can't get the system metadata from the mn http://metacat-hl:8080/catalog/d1/mn for the object doi:10.15485/1898912 since class org.dataone.client.exception.ClientSideException: /metacat-hl: Name or service not known. We will try to get it from cn. [org.dataone.cn.indexer.parser.utility.SeriesIdResolver:getPid:61]
dataone-indexer 20250610-15:44:50: [INFO]: SeriesIdResolver.getPid - get the system metadata for the object doi:10.15485/1898912 from the cn since the current node is cn or the systemmetadata is not available on a mn with baseurl http://metacat-hl:8080/catalog/d1/mn [org.dataone.cn.indexer.parser.utility.SeriesIdResolver:getPid:67]
dataone-indexer 20250610-15:44:50: [INFO]: response httpCode: 200 [org.dataone.service.util.ExceptionHandler:filterErrors:93]
dataone-indexer 20250610-15:44:50: [INFO]: ||||||||||||||||||| the head version is urn:uuid:da404cb4-1b7e-47ec-9e29-a552f5bb04cf for sid doi:10.15485/1898912 [org.dataone.cn.indexer.resourcemap.ForesiteResourceMap:isHeadVersion:214]
2025-06-10T15:44:50.236318555Z dataone-indexer 20250610-15:44:50: [INFO]: ||||||||||||||||||| the pid ess-dive-8e6738870d87db3-20221117T224253485 is NOT the head version for sid doi:10.15485/1898912 [org.dataone.cn.indexer.resourcemap.ForesiteResourceMap:isHeadVersion:221]
dataone-indexer 20250610-15:44:50: [INFO]: The id org.dataone.service.types.v1.Identifier@c6ea9113 is not the head of the serial id doi:10.15485/1898912 So, skip merge this one!!!!!!!!!!!!!!!!!!!!!!ess-dive-8e6738870d87db3-20221117T224253485 [
[... additional log lines omitted for brevity ...]