Adding names to the whitelist does not except them

[msnackey]

Using the code suggestion from the documentation page, I tried adding a name ('Max') to the whitelist to prevent it from being deidentified:
deduce.lookup_structs['whitelist'].add_items_from_iterable(["Max"])

Unfortunately, this did not lead to the expected result of 'Max' still being represented in the deidentified text. I checked to make sure that adding it to the whitelist lookup structure did work, and indeed it was one of the items of the whitelist LookupSet. I also tried this with the placename Utrecht and had similar results. Maybe it is not intended to work with names but only (common) words?

Using deduce 3.0.3.

[LSpekschoorRumc]

Great library so far. Unfortunately this issue with lookup structures also occurs when I try to expand first names.

At first I thought it was a documentation issue because the docs say deduce.lookup_structs['first_names'].add_items_from_iterable(["naam", "andere_naam"]), while the underlying docdeid library specifies in their API that the function for LookupTrie isn't called add_items_from_iterable (as it is for LookupSet), but add_item.

When manually adding a first name string in a list to an instance of deduce, I also find it in my variable called deduce_instance.lookup_structs['first_names']. This suggests it's not due to the implementation of the trie data structure.

Inspecting the result of deduce.deidentify(text) shows that the annotation is already missing, so it is likely not going wrong in the processing or redaction steps.

Using deduce v3.0.3

[LSpekschoorRumc]

After adding ‘Pikachu’ using deduce_inst.lookup_structs['first_names'].add_item(["Pikachu"]), I inspected the object and found a difference between 'Pikachu' and another name such as 'Ilse'.

When you look at
deduce_inst.__dict__['processors'].__dict__['_processors']['names'].__dict__['_processors']['first_name_lookup'].__dict__
you'll find another dictionary. Among the available keys are '_trie' and ‘_start_words’. Both contain names, but while _trie did contain ‘Pikachu’ as expected (and consistent with other places where the trie data structure can be accessed in the deduce_inst object), _start_words didn't.

On observing a length difference of 1, it seemed that every name except ones added through the API would be present in both '_trie' and '_start_words'.

This gave an opportunity to test if manual addition of 'Pikachu' to ‘_start_words’ and calling deduce_inst.deidentify() again would help. It did!

Next step is to search the codebase of deduce and docdeid for ways to implement '_start_words' differently. Searching gave some hits.