Add explorer key actions and improve auto-voice handling

[dpvc]

Overview

This PR adds a number of explorer keyboard actions from v3 that were left out in the rewrite for v4. These include:

• Z to get the full speech for a collapsed subexpression.
• V to mark a position for later retrieval.
• P to cycle through the marked positions.
• U to remove all marks and go to the initial location where the explorer was started.
• Shift + Arrow for moving in a table
• 0-9 + 0-9 to move to a specific table cell by index.
• Home to go to the top level of the expression.

Plus the new

• s to start or stop auto-voicing.

There are also changes to improve the collapse symbol for roots (since the surd character in mathjax-newcm is not centered by default). This requires a new version of the font that includes a variant that is centered; I have made the modifications, but haven't pushed a new version of the fonts yet.

I updated the help dialog to include the new keys, and changed how the modal dialog is handled: the old version used focus-out to close the dialog, but some screen readers end up giving focus out events when reading the links, so that would close the dialog prematurely. So now it uses a transparent background element to trap clicks outside the dialog box and closes on that.

The handling of auto-voicing has been improved, not only by the addition of the s key, but also by allowing any keypress or mouse down to stop the auto-voicing. That means the control key doesn't need to be mapped explicitly, since any keypress cancels the speech.

Finally, I remove the blank speech that I had included earlier, and now any node with no speech is skipped by the explorer (the data-speech-node attribute is removed, so it is never found by the explorer).

---

Details

The changes to collapse.ts are to set the mathvariant to the tex variant font so that the surd will be properly placed. It requires a new version of the font, which isn't public yet.

The explorer.ts changes are just additional CSS to handle the help dialog better (I put the keys into a kbd element with the styling that you had in the documentation pages, and I changed the ul list ot not have bullets, which were being read).

The main changes are in KeyExplorer.ts:

The changes to the dialog text is mostly addition of the new keyboard commands.

The key mappings are changed to handle the shift key for the arrows, to add the new commands, and to remove the upper-case versions of the keys, which are handled now in the key-press event handler. The map value is now an array consisting of a keymapping and a boolean, rather than having lots of explorer.active ? explorer.fn() : boolean constructs. The explorer.active test is now done in the keydown event handler when there ius a boolean in the array.

The marks, currentMark, and lastMark are used by the V, P, and U keys to handle the position marks.

The pendindIndex array is used for the cell index processing. When the number key n is pressed, this array is set to [0, n]. It works this way: when a key is pressed, the array is shifted to the left, making it [n] if the previous key was a number key, and [] if not. So if a number key is followed by a nother number key, pendingIndex will be [n] for the previous number and the new key will be the second number, and we can use these to get the indices for the cell to jumo to. If a number key is not followed by another number, then the [n] will be shifted off the array when the next key is pressed, so pressing a number key will only have a value in pendingIndex when it immediately follows a number key.

The Keydown event handler shifts the pendingIndex array, as described above, and cancels any auto-voicing in progress. The other change is to handle upper-case letters by converting them to lower-case so we don't have to have duplicate entries in the key map.

The Mousedown events clear the pending index array and cancels any auto-voicing in progress.

The controlKey() method is removed, as it is now redundant.

The homeKey() action is added.

The arrow key functions are modified to check if the shift is pressed, and if so, they call moveToNeighborCell() with index offsets, otherwise do the usual move.

The new moveToNeighborCell() function finds the cell (if any) containing the current node, and gets the indices of the cell within its table. Then it get the cell at the correct offset from that position and selects that.

Next come the position marking functions. The addMark() method pushes the current position onto the mark list and keeps the index into the list as currentMark, unless the current position is the same as the current mark, in which case, the usual text for that position is read again. That allows the "Position marked" text to be replaced by the usual speech at that point.

The prevMark() function checks if currentMark is unset (-1), and if so, sets currentMark to the last mark in the list, and if there is none, it reads either the initial position when we started exploring this expression (lastMark) or the top level of the expression. If there is a mark to be read, we jump to that mark and move currentMark to be the next one in the list, so that if we hit P multiple times in a row, we cycle through the positions. Because setCurrent() clears currentMark, we save it before setting the new position.

The clearMarks() function does what it says, clears the marks and then goes to the initial position by calling prevMark().

the autoVoice() method toggles the auto-voicing option either in the menu if there is one, or in the document options if not, then updates (to start the voicing).

The numberKey() function handles jumping to a specific cell by index. Here, if we aren't in a table, we quit with a honk. Then we change 0 to 10, if needed. Then, if there is no pending index (the "else" clause), this is the first index, so we save it in the pendingIndex as [0, n], as described above (it will shift into [n] when the next number is pressed). We speak a phrase to indicate that we have the row number and are waiting for the column number. Otherwise, if there is a pending index, that means we have a previous index and are ready to jump. We look up the table for where we are, and the get the cell with the proper index. We speak the column index (to complete the "jump to row n and column" phrase, clear the index array, and finally select the new cell after a slight delay to allow the column number to be read.

You may not like the extra "jump to..." speech, but I think there needs to be feedback for the numbers being pressed so the user knows they are being processed. It could be that they just say the number and not the whole phrase. Perhaps another menu option needs to be made for that?

The details() function implements speaking the full text of a collapsed expression. Since that test is not available in the attributes of the tree, we have to work a little bit to get this text.

First, we check if this is a collapsed node, and if not, we jsut speek the current node again. Otherwise, we look through the internal MathML for the item with the current node's semantic ID. We use that node to create a new MathML string from the subtree at that node, and add <math> around it, if needed, removing any previous data-semantic elements. Finally, we call the (new) speechFor() method of the MathItem (described below) to get the top-level speech and Braille strings for the MathML, and speak them when they arrive.

The help() function is modified to include the new transparent background (used to trap clicks outside the dialog), and to use a common dynamic close() function for the event handlers that close the dialog.

The setCurrent() function has additions to track the lastMark position and to clear the currentMark so that P will start cycling from the taop of the mark list if we do anything other than P.

The nodeId() and parentId() functions are just service functions to make getting data-semantic ids easier. The getNode() function returns the HTML node with the given semantic ID, while the childArray() method returns an array of child IDs for the given HTML node.

The tableCell() function looks through the parents of a node for the first one with role table (the cell that contains the node, if any), while cellTable() returns the HTML node for the table that contains the given cell node.

The cellPosition() function returns the (row,column) indices of a cell within a table, while cellAt() returns the HTML node for the cell at a given (row,column) position within the table, if any.

The changes to region.ts are for cancelling the auto-voicing. In particular, we want to make sure that the synchronous highlighting is removed if the speech is interrupted. (It had been being left with a permanent red background.) We add a cancelVoice() function that can be called from the KeyExplorer.

In semantic-enrich.ts, we add a toEnriched() function that can be called to enrich an arbitrary MathML string. This is used by the Z explorer key to get the full speech for a collapsed node.

Similarly, in speech.ts, we add a speechFor() function that returns the speech and Braille strings for an arbitrary mathML string. Again, this is for the Z explorer key. The speechFor() method uses a new SpeechFor() method in the GeneratorPool.tsfile, which in turn calls a newspeechFor()function inWebWorker.ts`, that returns a promise for the when worker prodiuces the speech structure.

Finally, WebWorker.ts includes the changes to not mark nodes with blank speech or blank braille.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 86.71%. Comparing base (4d1e195) to head (20068cc).

Additional details and impacted files

@@            Coverage Diff             @@
##           develop    #1295     +/-   ##
==========================================
  Coverage    86.71%   86.71%             
==========================================
  Files          337      337             
  Lines        83979    83979             
  Branches      4750     3124   -1626     
==========================================
  Hits         72826    72826             
- Misses       11130    11153     +23     
+ Partials        23        0     -23     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.