Empty "Closure of links" file

[coret]

I have made a file (using the CIV ontology) containing all civil registries from SAMH (available as download via Open Archives). I've run all functions, but the last step - closure - doesn't seem to compute the transitive closure of all detected links. The file links-individuals.csv is empty isn't created.

$ java -jar burgerLinker.jar --function closure --inputData output/smh.civ.hdt --outputDir output

=======================
Welcome to burgerLinker
=======================

WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
START: Computing the transitive closure
START: Loading HDT dataset...
*DONE: Loading HDT dataset [runtime: 1.21 s]

--------
- Number of statements in this dataset: 10876758
- Number of distinct subjects: 1737181
- Number of distinct predicates: 19
- Number of distinct objects: 4956465
--------



Saving individual links (Between_D_M) for file: output/between_d_m-maxLev-3-fixed/results/between-D-M-maxLev-3-fixed.csv
Between_D_M 100% │█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████│ 84952/84952 links (0:00:01 / 0:00:00)

Saving individual links (Between_M_M) for file: output/between_m_m-maxLev-3-fixed/results/between-M-M-maxLev-3-fixed.csv
Between_M_M 100% │█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████│ 46802/46802 links (0:00:00 / 0:00:00)

Saving individual links (Between_B_D) for file: output/between_b_d-maxLev-3-fixed/results/between-B-D-maxLev-3-fixed.csv


Saving individual links (Between_B_M) for file: output/between_b_m-maxLev-3-fixed/results/between-B-M-maxLev-3-fixed.csv
Between_B_M 100% │█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████│ 63232/63232 links (0:00:01 / 0:00:00)

Saving individual links (Within_B_M) for file: output/within_b_m-maxLev-3-fixed/results/within-B-M-maxLev-3-fixed.csv
Within_B_M 100% │██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████│ 18576/18576 links (0:00:00 / 0:00:00)

Saving individual links (Within_B_D) for file: output/within_b_d-maxLev-3-fixed/results/within-B-D-maxLev-3-fixed.csv
Within_B_D 100% │██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████│ 63938/63938 links (0:00:01 / 0:00:00)

Starting: Closure of links in file: output/closure/results/links-individuals.csv

Closure 100% │███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████│ 630560/630560 links (0:00:00 / 0:00:00)
Number of individuals from Invidual to Class: 10
Number of individuals from Class to Individuals: 10
Number of max individuals in Eq Class: 10
Saving closure results to file...
Finished saving closure results to file!
Estimated number of certificates to be processed is: 10876758
Reconstructing dataset after transitive closure 100% │███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████│ 10876758/10876758 cert. (0:00:09 / 0:00:00)
*DONE: Computing the transitive closure [runtime: 17.65 s]

=====================
*DONE: burgerLinker [runtime: 17.67 s]
=====================

$ less output/closure/results/links-individuals.csv
output/closure/results/links-individuals.csv: No such file or directory

There is a finalDataset.nt file, but this just contains all input data. My 'test person' is "abraham johannes loon" (birth, marriage and death certificate) but in finalDataset.nt I encounter all 3 events with different person URIs.

What am I missing here?

[rlzijdeman]

hi, Bob, thanks for sharing this. @raadjoe @ivozandhuis @RubenSchalk any thoughts before looking into the code itself?

[raadjoe]

Hi Bob and Richard, sorry for the delayed response. I will have a look at the code and test it with the SAMH dataset, and will get back to you soon (next week most probably).

[ivozandhuis]

I'll test it with my own mini-dataset (of three registrations) this weekend.

[ivozandhuis]

I experience something similar.
This is what I did.

The last step results in this error message:

java.lang.NumberFormatException: For input string: "person_hga_F037247C-F765-4A1A-81BD-BB2BEF36F5A6_Person2053123967"
        at java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
        at java.base/java.lang.Integer.parseInt(Integer.java:652)
        at java.base/java.lang.Integer.parseInt(Integer.java:770)
        at iisg.amsterdam.burgerlinker.LinksCSV.saveIndividualLink(LinksCSV.java:326)
        at iisg.amsterdam.burgerlinker.processes.Closure.saveLinksIndividuals_Within_B_M(Closure.java:435)
        at iisg.amsterdam.burgerlinker.processes.Closure.saveIndividualLinksToFile(Closure.java:368)
        at iisg.amsterdam.burgerlinker.processes.Closure.computeClosure(Closure.java:74)
        at iisg.amsterdam.burgerlinker.processes.Closure.<init>(Closure.java:64)
        at iisg.amsterdam.burgerlinker.Controller.computeClosure(Controller.java:399)
        at iisg.amsterdam.burgerlinker.Controller.runProgram(Controller.java:109)
        at iisg.amsterdam.burgerlinker.App.run(App.java:91)
        at iisg.amsterdam.burgerlinker.App.main(App.java:60)

[coret]

@ivozandhuis please change the id's of the persons to an integer (I had the same issue). Then the closure will run, curious about your outcome.

Integer as id's is a documented requirement, which I'm not happy with btw. I have to 'synthesize' numeric identifiers somehow for huge number of records.

[ivozandhuis]

Unfortunately that does not seem to work, slightly different error message, thought:

java.lang.NumberFormatException: null
        at java.base/java.lang.Integer.parseInt(Integer.java:614)
        at java.base/java.lang.Integer.parseInt(Integer.java:770)
        at iisg.amsterdam.burgerlinker.LinksCSV.saveIndividualLink(LinksCSV.java:326)
        at iisg.amsterdam.burgerlinker.processes.Closure.saveLinksIndividuals_Within_B_M(Closure.java:435)
        at iisg.amsterdam.burgerlinker.processes.Closure.saveIndividualLinksToFile(Closure.java:368)
        at iisg.amsterdam.burgerlinker.processes.Closure.computeClosure(Closure.java:74)
        at iisg.amsterdam.burgerlinker.processes.Closure.<init>(Closure.java:64)
        at iisg.amsterdam.burgerlinker.Controller.computeClosure(Controller.java:399)
        at iisg.amsterdam.burgerlinker.Controller.runProgram(Controller.java:109)
        at iisg.amsterdam.burgerlinker.App.run(App.java:91)
        at iisg.amsterdam.burgerlinker.App.main(App.java:60)

[coret]

FYI

You can now get N-Triples and Turtle in Civil Registries Schema directly from Open Archives via:
curl -H 'Accept: application/n-triples+civ' https://www.openarch.nl/hga:F6619DCA-BE62-4A37-B09D-960CB8900393
curl -H 'Accept: text/turtle+civ' https://www.openarch.nl/hga:F6619DCA-BE62-4A37-B09D-960CB8900393

or by appending /nt:civ or /ttl:civ to the record URL (eg. https://www.openarch.nl/hga:F6619DCA-BE62-4A37-B09D-960CB8900393) in your browser:

• https://www.openarch.nl/hga:F6619DCA-BE62-4A37-B09D-960CB8900393/nt:civ
• https://www.openarch.nl/hga:F6619DCA-BE62-4A37-B09D-960CB8900393/ttl:civ

[coret]

Now a minimal, repeatable test to show the closure seems not to be working is as easy as:

rm -Rf output_test
mkdir output_test

curl -H 'Accept: application/n-triples+civ' https://www.openarch.nl/smh:d20c645a-b053-6ded-5fc8-fec1d28d3b7d > output_test/closure_test.nt
curl -H 'Accept: application/n-triples+civ' https://www.openarch.nl/smh:d7cedde4-39fe-dc10-dff7-cfb26a8e1b90 >> output_test/closure_test.nt
curl -H 'Accept: application/n-triples+civ' https://www.openarch.nl/smh:aa01e7c9-017e-3014-6110-9b64af482d60 >> output_test/closure_test.nt
curl -H 'Accept: application/n-triples+civ' https://www.openarch.nl/smh:604f5fe2-6e44-ec00-c44d-bd14018e14ff >> output_test/closure_test.nt

java -jar burgerLinker.jar --function ConvertToHDT --inputData output_test/closure_test.nt --outputDir output_test

java -jar burgerLinker.jar --function Within_B_M --inputData output_test/closure_test.hdt --outputDir output_test --format CSV --maxLev 3 --fixedLev
java -jar burgerLinker.jar --function Within_B_D --inputData output_test/closure_test.hdt --outputDir output_test --format CSV --maxLev 3 --fixedLev

java -jar burgerLinker.jar --function closure --inputData output_test/closure_test.hdt --outputDir output_test

[raadjoe]

The bug should be now fixed in the latest release. The issue was in the required type of literals for registrationID that needs to be explicitly specified in the HDT library. The previous version required these registrationIDs to be of type xsd:int, now the tool allows as well registrationIDs of type xsd:integer. I will verify later what needs to be modified in the code to remove completely the restriction on the type of registrationID.

I tested the closure function on the links detected from the SAMH dataset, and now the resulted dataset finalDataset.nt in the closure/results directory replaces identifiers of individuals that are linked with a unique identifier (considering as well the transitive characteristic of these identity links).

Of course let me know if you have more problems when computing the closure.

[ivozandhuis]

Thanks Joe! Unfortunately the above test-script by Bob does not work for me yet. Bob: does it work for you? Maybe there is something wrong with my Java-installation?

[coret]

On the first run of the small test mentioned above I was presented with an error, something like "91108331000 is not an integer" (maximum of 32-bit int is only 2.147.483.647). I adjusted the Open Archives numbering scheme so these synthetic, non-persistent event and individual numbers fall in the integer range...

On the second run with the downloaded 0.6 release of burgerLinker I did get a links-individuals.csv file with contents:

565548082,1822166623,sameAs,W_B_M,22,3,1822166620,565548080,0,0,2,2,2,28
565548085,1822166622,sameAs,W_B_M,22,3,1822166620,565548080,1,1,1,1,1,28
565548083,1822166621,sameAs,W_B_M,22,3,1822166620,565548080,0,0,2,1,1,28
1766929781,1822166623,sameAs,W_B_D,22,3,1822166620,1766929780,0,0,2,2,2,87
1766929783,1822166622,sameAs,W_B_D,22,3,1822166620,1766929780,0,0,1,1,1,87
1766929782,1822166621,sameAs,W_B_D,22,3,1822166620,1766929780,0,0,2,1,1,87

But in finalDataset.nt I just see all of these 'duplicate' individuals, no merged ones with new identifiers? So no closure yet.

I'm currently running burgerLinker on the SAMH dataset (10M+ statements, takes a bit longer...)

FYI:

$ java -version
openjdk version "11.0.15" 2022-04-19
OpenJDK Runtime Environment (build 11.0.15+10-post-Debian-1deb11u1)
OpenJDK 64-Bit Server VM (build 11.0.15+10-post-Debian-1deb11u1, mixed mode, sharing)

[coret]

The closure of the SAMH dataset did result in <https://iisg.amsterdam/links/person/i-...> and <https://iisg.amsterdam/links/event/i-...> URI's for "merged" persons!
But as the small test didn't, I am not entirely convinced that this closure is complete?

Although, my test-person, who is in both the small dataset as the SAMH data, doesn't appear "merged" in the closure of the small dataset, but does appear "merged" in the SAMH dataset:

finalDataset.nt:<https://iisg.amsterdam/links/person/i-123035> <http://schema.org/familyName> "loon".
finalDataset.nt:<https://iisg.amsterdam/links/person/i-123035> <http://schema.org/gender> "m".
finalDataset.nt:<https://iisg.amsterdam/links/person/i-123035> <http://schema.org/givenName> "abraham johannes".
finalDataset.nt:<https://iisg.amsterdam/links/person/i-123035> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://schema.org/Person>.
finalDataset.nt:<https://iisg.amsterdam/links/person/i-123035> <https://iisg.amsterdam/id/civ/personID> "1304896232"^^<http://www.w3.org/2001/XMLSchema#integer>.
finalDataset.nt:<https://iisg.amsterdam/links/person/i-123035> <originalRegistrationID> "person_smh_604f5fe2-6e44-ec00-c44d-bd14018e14ff_Person:3093ad6d-570c-2671-387a-752fc09ace0b".
finalDataset.nt:<https://iisg.amsterdam/links/person/i-123035> <http://schema.org/familyName> "loon".
finalDataset.nt:<https://iisg.amsterdam/links/person/i-123035> <http://schema.org/gender> "m".
finalDataset.nt:<https://iisg.amsterdam/links/person/i-123035> <http://schema.org/givenName> "abraham johannes".
finalDataset.nt:<https://iisg.amsterdam/links/person/i-123035> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://schema.org/Person>.
finalDataset.nt:<https://iisg.amsterdam/links/person/i-123035> <https://iisg.amsterdam/id/civ/personID> "1305458471"^^<http://www.w3.org/2001/XMLSchema#integer>.
finalDataset.nt:<https://iisg.amsterdam/links/person/i-123035> <originalRegistrationID> "person_smh_d20c645a-b053-6ded-5fc8-fec1d28d3b7d_Person:339ddf00-5be4-3bc7-704b-596de9716514".
finalDataset.nt:<https://iisg.amsterdam/links/person/i-123035> <http://schema.org/familyName> "loon".
finalDataset.nt:<https://iisg.amsterdam/links/person/i-123035> <http://schema.org/gender> "m".
finalDataset.nt:<https://iisg.amsterdam/links/person/i-123035> <http://schema.org/givenName> "abraham johannes".
finalDataset.nt:<https://iisg.amsterdam/links/person/i-123035> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://schema.org/Person>.
finalDataset.nt:<https://iisg.amsterdam/links/person/i-123035> <https://iisg.amsterdam/id/civ/personID> "1304964153"^^<http://www.w3.org/2001/XMLSchema#integer>.
finalDataset.nt:<https://iisg.amsterdam/links/person/i-123035> <originalRegistrationID> "person_smh_d7cedde4-39fe-dc10-dff7-cfb26a8e1b90_Person:ecefddea-4de7-11dc-4705-f3feca12fefe".

[raadjoe]

Hi Bob and Ivo,

Sorry for the previous incomplete bug fixes.
I updated the previous release with a new JAR file burgerLinker.jar (same version)

@coret The issue was in saving the links to the file, and especially when the number of lines to write on the file was smaller than 20 lines. This issue should now be fixed. I tested it on both the small and the SAMH dataset, and in both cases the individuals were merged in the finalDataset.nt file.

@ivozandhuis I checked your datasets, and you have two small issues with the data that are causing these errors:

1. The ID of individuals should be given using the ns2:personID relation (instead of ns2:persID).
2. The IDs should be typed, either using xsd:int or xsd:integer. It is annoying to type all literals, but this is required in the HDT library, otherwise the HDT search functionality does not find the person with a given identifier.

Don't hesitate to let me know if you find more issues when computing the closure.

[ivozandhuis]

@raadjoe You're a hero! Thanks.