Improve overcrawl tracking for BioCASe datasets

[MattBlissett]

If you put this in an file count_request.xml:

<?xml version='1.0' encoding='UTF-8'?>
<request xmlns="http://www.biocase.org/schemas/protocol/1.3">
        <header><type>search</type></header>
        <search>
                <requestFormat>http://www.tdwg.org/schemas/abcd/2.06</requestFormat>
                <responseFormat start='0' limit='10'>http://www.tdwg.org/schemas/abcd/2.06</responseFormat>
                <filter>
                        <like path='/DataSets/DataSet/Metadata/Description/Representation/Title'>University of Vienna, Institute for Botany - Herbarium WU</like>
                </filter>
                <count>true</count>
        </search>
</request>

and then query the appropriate BioCASe endpoint:

curl --data-urlencode query@count_request.xml 'http://131.130.131.9/biocase/pywrapper.cgi?dsa=gbif_wu'

The response gives us a record count:

  <biocase:content recordCount="0" recordDropped="0" recordStart="0" totalSearchHits="0">
    <biocase:count>107838</biocase:count>
  </biocase:content>

It's way off from what we have in GBIF, so perhaps this dataset is very overingested: https://management-tools.gbif.org/crawl-history?uuid=0afba960-be3b-4202-a7de-736ae05aec9e

{
  "datasetKey" : "0afba960-be3b-4202-a7de-736ae05aec9e",
  "lastCrawlId" : 34,
  "lastCrawlCount" : 80655,
  "recordCount" : 155141,
  "lastCrawlFragmentEmittedCount" : 104005,
  "finishReason" : "NORMAL",
  "processStateOccurrence" : "RUNNING",
  "crawlInfo" : [ {
    "crawlId" : 3,
    "count" : 1046
  }, {
    "crawlId" : 4,
    "count" : 4
  }, {
    "crawlId" : 5,
    "count" : 2
  }, {
    "crawlId" : 8,
    "count" : 1
  }, {
    "crawlId" : 9,
    "count" : 2
  }, {
    "crawlId" : 10,
    "count" : 2
  }, {
    "crawlId" : 11,
    "count" : 29
  }, {
    "crawlId" : 12,
    "count" : 29381
  }, {
    "crawlId" : 13,
    "count" : 10192
  }, {
    "crawlId" : 14,
    "count" : 33069
  }, {
    "crawlId" : 15,
    "count" : 1
  }, {
    "crawlId" : 16,
    "count" : 58
  }, {
    "crawlId" : 17,
    "count" : 2
  }, {
    "crawlId" : 18,
    "count" : 2
  }, {
    "crawlId" : 20,
    "count" : 268
  }, {
    "crawlId" : 21,
    "count" : 1
  }, {
    "crawlId" : 22,
    "count" : 111
  }, {
    "crawlId" : 23,
    "count" : 27
  }, {
    "crawlId" : 24,
    "count" : 134
  }, {
    "crawlId" : 25,
    "count" : 123
  }, {
    "crawlId" : 26,
    "count" : 3
  }, {
    "crawlId" : 30,
    "count" : 17
  }, {
    "crawlId" : 31,
    "count" : 1
  }, {
    "crawlId" : 33,
    "count" : 10
  }, {
    "crawlId" : 34,
    "count" : 80655
  } ],
  "percentagePreviousCrawls" : 48.011808612810285
}

A different dataset always returns less than the search.

This could belong as a core part of the crawler, we have a mostly-unused "declaredCount" field in the registry with a related purpose, but I think it hasn't been implemented as it's very unreliable. Maybe it would be better as a tool on the management console.

@gbif/content can think about it.

[ahahn-gbif]

The count relies on the dataset title to be searchable, which is frequently not the case. I would agree, though, that at least making use of this count where it is available would be helpful to highlight cases that need more investigation.

With the question-response protocols, it is less straightforward to draw a conclusion from the harvested numbers, as the communication may break off in the middle of transfer, either because of data content issues or unstable services - we may never have managed to index the full dataset, but on the other hand sit with a lot of stale records. With the given example, it is certainly over-indexed, but would require checking that a full indexing run has completed before then deleting any leftover records.

[MattBlissett]

All queries for BioCASe rely on the dataset title to be searchable, so using it for counts isn't a problem.

The first step of crawling, for both BioCASe and DiGIR, updates the declaredCount. (NB, not for TAPIR.)

There are at least some datasets where it's easy to see duplication, since we have exactly 2× or 3× the declaredCount. This installation has several cases: https://www.gbif.org/installation/35ad703f-c23a-44c9-b3d9-b2fec65298f1 but also one where we don't as many as are declared by the count.

[ahahn-gbif]

Good to know it works reliably now; that was not always the case. The temptation to map the dataset title as a static value (applying to all records, but then not searchable) has often led to confusion in the past.

Do we store the declaredCount value anywhere that would allow integrating it as a target number for BioCASe datasets, i.a. into the console at https://management-tools.gbif.org/overcrawls? So far, it has been a manual check on known cases, directly polling the endpoint for the count of a specific dataset. It would be great if we could spot these cases more easily, and possibly even kick off a (manually triggered) delete for "records older than..." if the count of the last indexing run matches that target.