Fix PDF.js support: integrate with internal APIs

[csillag]

Opening issue, based on discussion with @dwhly on IRC.

Unfortunately, it looks like PDF.js is using a lazy approach for rendering of the text into the DOM. That means we have the same problem as we have with scribd - see #744.

Basically, we can not anchor to text that is not yet present in the dom.

We have not recognized this until so far, because we were testing only with short PDF files, which were always rendered instantly.

But (unlike with scribd), we have a chance to fix this, since PDF.js is open source.

We could do this:

• Recognize that we are in PDF.js. (Is not, do everything as we have done until now.)
• In PDF.js, before doing anything on the DOM, run the full reattachment process on the stored text. (This needs API access from PDF.js.) (Naturally, this would kils the range selectors, but position selectors and of course fuzzy could still work.)
• As we scroll around, and PDF.js renders the text into the DOM, it should send us scroll events, and then we should do the actual attaching of annotations.
• We also need to change the heatmap, so that it shows the info, about the not-yet attached annotations, too. (Mostly on the up- and down- arrows, since everything in the viewport will be attached already.)

This requires adapting (a big part of) our anchoring process from the DOM to the in-ram stored text of PDF.js.

This will likely take at least a month or so.

[csillag]

It turns out this issue was known a long, long time ago. There is some relevant info in #89 about fixing this.

[csillag]

Here is a plan with some milestones. (Draft only)

• Get our Annotator version as close to upstream as possible without actually merging fuzzy upstream. This step is optional, but we have diverged enough already, and clearing this up gets harder and harder with each step. My impression is that we have already neglected this aspect too much. Doing this would involve:
  • Get off from the maintenance branch, and merge upstream changes
  • Where possible, modify our code so that it's identical to upstream
  • Where impossible, send PRs to upstream (this involves heavy documentation and test creation)
  • While doing all this, makes sure that our Annotator fork can still be embedded in H, and works as required
• Get a preliminary grip about PDF.js internals by reading some of the available docs
• Contact PDF.js developers, explain them our requirements, and get info about the required integration points, and verify that our planned approach is feasible - see thread here
• Run some experiments with the recommended integration points, and verify that they do what we need them to do
• In our fork of Annotator, start to build a different code-path for working inside PDF.js. First step is to detect the environment, and add some new data structures to use for virtual anchoring.
• Implement new method for scanning PDF documents. (Extract text, store page mappings, map internals of pages.)
• Text extraction API vs. Selection API harmonization 1: white-space elimination - obsoleted by Smart String Padding
• When creating new annotations over a PDF doc, create selectors (Range, TextPosition, TextQuote) with the correct data, regardless of the DOM state
• Fix re-attachment of annotations on parts that are already rendered at load time. - This feature was removed. All PDF anchoring is virtual now.
• Sort out sync / async API mess 1 - rendering (on page turns) is async; we have to wait for that
• Sort out sync / async API mess 2 - the initial text extraction is done using an async API. We have to wait for this to finish
• Modify the setupAnnotation() method (and workflow) for virtual anchoring so that when trying to get an anchor, we don't work with the DOM, but with the in-memory corpus, and store the virtual anchors.
• Virtual anchoring based on all applicatbe strategies (Position, two-phase fuzzy, one-phase fuzzy)
• When rendering a new page, check the pending virtual anchors, and execute the physical anchoring
• Solve the problem of un-rendered (deleted) pages 1: Add missing hooks to PDF.js. (Solves Chrome, and FF27+) obsoleted by generic solution below
• Solve the problem of un-rendered (deleted) pages 2: without PDF.js modification (Required for FF24-FF27)
• Support annotations that span across page (and therefore rendering) borders (This one was big, and caused API changed, down to the d-t-m level. So the changes now impact HTML documents, too.)
• Fix crash sometimes triggered by creating cross-page annotations
• Fix bugs caused by a page being un-rendered with part of a highlight - Probably done, testing pending
• Text extraction API vs. Selection API harmonization 2: smart string padding
• Fix un-attached comment functionality for PDF - Updated on top of my fix for Always-on annotations show last char of source as highlight for comments #700
• Enhance heatmap: up/down arrows: show the right number, even when some annotations are not rendered
• Enhance heatmap: up/down arrows: ability to scroll to next, even when not rendered.
• Fix heatmap: tabs are not drawn - Chrome only - This one involves some incomprehensible CSS black magic. Bad for sanity.
• Fix heatmap: colors - it's usable as it is
• Test/fix for FF - JSChannel issue: Can't connect to PDF.js pages mozilla/jschannel#20; PR: Enable connection to pdf.js environments mozilla/jschannel#21
• Fix icomoon icon mixup weirdness - FF only - Opened separate issue: Cross origin fonts on multi-frame #812.
• Add ability to jump to a page (as part of scrolling to an annotation), even if it's not rendered currently - It will make more sense to do this after Special behavior for selected annotations #800 is done.
• Once we have got this running in H, make sure (our fork of) Annotator is still working in standalone mode, too.
• Factor out the PDF document access strategy into an Annotator plugin
• Unite single-phase and two-phase anchoring codepaths (inside Annotator)
• Integrate our PDF.js fork directly into our Chrome plugin (so that it just works for Chrome) - We have a separate issue for this: Chrome support for PDFs #512

This comment is continuously updated with new information. That also includes removing boring steps when done.

[csillag]

I started to look at PDF.js examples, API and source. What is already apparent that all of it's APIs are fully asynchronous. It will be interesting (or maybe impossible) to integrate this without going async with our internal APIs, too.

We might be able to dodge this, for now, but it won't be pretty.

[csillag]

The missing hooks might not be a problem after all. On the other hand, it looks like the selection API does a particularly poor job over the PDF text rendered by PDF.js. (Randomly disappearing white-spaces, etc.)

Normally, this should not be a serious problem (thanks to fuzzy), but in this case, the problem is that there is an incompatibility between the text produced by the text extraction API, and the selection API. Not sure how to reconcile this... (updated roadmap)

[csillag]

I have just realized that the problem I am seeing has been entered to PDF.js's tracker by @dwhly about a year ago. Here: mozilla/pdf.js#1883

[dwhly]

Can you help me understand a bit better how the incompatibility between extraction and selection factors here, and why fuzzy doesn't help? Also, how much better would an improved selection API have to be to reasonably address this?

[csillag]

Q1: the problem.

This is how it goes.

We need to select a reference system to use for string offsets in PDF files.

When we are deployed over a PDF document, most of the pages are not yet rendered, so we can't read text using the selection API. The only source of information available at the time is the text extraction API. We read what is on each page (using the text extraction API), and so we create a mapping of text ranges <---> pages.

This allows us to determine where a given character position falls inside the document. The first step is to determine the page.

To find the text inside the page, we need to have the DOM rendered. Then we use the dom-text-mapper magic, as usual: we scan all the DOM elements (using the selection API), and check how they are actually rendered. This way we build a mapping from character offset ranges (inside the page) and DOM elements.

If we have the character offsets of the pages, and the character positions inside the pages, we can find anything we want.

However, the problem is that when we check the contents of a page with the Selection API, we get a different version of the text, compared to what we got earlier, with the text extraction API.

The characters are there, but the white-spaces (and therefore the offsets) are different.

We must decide which system of offsets do we want to use in our coordinates.

The problem is that during the initial loading, we don't have the selection API-style coordinates, because most of the pages are not rendered. That suggests that we standardize on the text extraction API-style coordinates. However, when we are working inside the page, we can only use what we really see, which is the selection API style.

Q2: why fuzzy does not help in this case?

Fuzzy comes into the picture when we know what we want, and we are searching for in in the doc.
But as it stands now, we might not always be able to find out what we want.
We fail in our internal calculations, not only with the anchoring.

Here is (a small part of) what we usually do, when we create our selectors:

• Start with the selected range
• Using our mappings, we go back to coordinates
• Using those coordinates, we find out the prefix and suffix
• We put those into the selectors.

However, here, when we go back to coordinates, and then from coordinates to text again, if the coordinate systems are in mismatch, then we might miss our mark totally, and save the wrong prefix in to the selector. (That's what happening now, this is how I discovered the problem.)

In theory, we could try to compensate those coordinates on the fly, using the selection API's data, but data is not always available, when the pages are not rendered. Again, it might be possible to avoid this, but this makes our code a lot more complicated.

Q3: how much improvement is needed?

Both APIs work from the same data. What we need is not improvement; it's unification. At the referenced bug, there is actual discussion about uniting the two algorithms so that their output is in sync. That's exactly what we need.

[csillag]

After some thinking, I see two ways to solve to Selection API vs Text Extraction incompatibility issues.

White-space elimination

This is the dumb and easy and solution. It goes like this: whatever method we use to get text, we drop all the white-spaces right away. No white-space, no problem. We standardise on this format (and so coordinates), and then we have a consistent reference. The drawback is that quotes will be all concatenated, and so ugly.

Smart string-padding.

This is the smart and complicated solution. In theory, we can probably go all over the text fragments received via the Text Selection API, and knowing how it should look (based on the text extraction data), we can try to find out what goes where, padding it with strings, as we go. We have to do this recursively on the DOM tree. Also, to get ranges correctly, we have to do this forwards and backwards too. And when we have the padded string offsets, we can alter the dom-text-mapper mapping information to match this information. Using this method, we can get nice results, with the proper spaces. But it's complicated, and there is some performance penalty, too.

Updating roadmap with these steps.

[csillag]

Milestone reached:

For the first time, we are now able to anchor annotations on the fly, as the PDF is rendered.
Does not show on the heatmap yet, but visible with always-on mode.
And there are other limitations, too - see roadmap.

[csillag]

@dwhly, working demo is available at http://pdf.dokku.hypothes.is/

You can use Chrome or FF. (I only tested with Chrome)
For Chrome, you have to install the PDF.js browser extension.

Here is how to test:

• Fetch the bookmarklet from dokku
• Point your browser to a PDF
• Apply bookmarklet
• You can start to create annotations
• Hopefully, they should persist across reloads.

Expect problems, especially if you open more than 20 pages. (Then it will start to un-render stale pages, which we can't handle yet.) Also, not cross-format / fuzzy stuff yet. See the roadmap above for upcoming features and status.

[csillag]

Quite a few new features are working now. (Update of the demo server is pending.)

Interesting things to test:

• Cross-format matching with fuzzy
• Long documents (However one scrolls from start to end and back, annotations on both ends should persist. Or, to be more precise, they should always re-appear when they are needed.)

See also the roadmap for info about done/missing features.