Explainers are an anti-pattern

[msporny]

Speaking as an Editor that has had to write or help with a few explainers, I find them to be an inefficient use of the WG's and the Editor's time. They effectively re-state much of what is in the specification and in a way that is not generally useful to anyone but possibly a handful of people that could have gotten the same information out of reading the use cases, charter, or introduction of the specification. They are also dropped on the floor and not updated until the next time we need to engage with the TAG or horizontal review process (document rot).

Most recently, I tried an experiment, which was to use ChatGPT's "Research Mode" to ingest the TAG's "Writing Effective Explainers", ingest the DID v1.1 specification, and generate the explainer for the v1.1 review. I then went through, line by line, to edit the result to ensure that it reflected what was requested in "Writing Effective Explainers" as well as the current interpretation of the specification, which it did. That process took about an hour, whereas writing an explainer, at least when I do it, typically takes multiple days (because it takes a lot of effort to condense a 40+ page specification down to 4 pages or so).

The result was this document: https://github.com/w3c/did/blob/main/EXPLAINER.md, which is better than the initial explainer written (and not updated for years). The TAG ended up not really reading that document, and then expected the explainer to contain things that were not requested in the "Writing Effective Explainers" document. These were the comments of concern:

I did not read it thoroughly since it was a general explanation of DID 1.1

An explainer is required for a horizontal review, so why are we required to produce things that won't be read? Was it because it was generated by an LLM? Even though it was edited by an Editor? In this particular case, the reviewer had a solid understanding of DIDs... but not reading the explainer is something I've seen happen in other reviews. What was more concerning was this exchange:

I suggest we discourage LLMs for writing explainers.
I think the way they did it is one of the least offenisve ways it can be done - Manu reviewed it. The 'alternatives considered' section doesn't match what we'd expect, but this is an OK overview of DIDs in general.
I've been reading about energy use and it seems to be that just using them doesn't use the main share of the energy; it's training them.

Using an LLM for this task saved me A LOT of time, and it produced output that was probably on par with what I would have generated anyway. I was expecting the output to be largely useless, but was pleasantly surprised that the LLM gave me something that was largely workable with a single (human) editorial pass that took 30 minutes or so. It was so successful, that I am probably going to use an LLM to write all future explainers, and want to be up-front with the TAG and all the other horizontal review boards as well (as I was this time by clearly marking the Explainer as having been generated by an LLM).

However, I'd go a step further -- instead of writing explainers, the TAG should probably ensure that the introductory portions of the specification covers the general things that are covered in an explainer, and if it doesn't, the specification should be fixed. Other items should probably just be questions in the horizontal review issue that is raised.

Having Editors generate explainers is an anti-pattern -- the spec should do what an explainer does in the introductory portions of the spec, and if it doesn't do that, those portions of the specification need to be changed. This would result in easier to read specifications at W3C and reduce the amount of time that Editors spend writing single use documents for the TAG that are not maintained.

[dlongley]

Having Editors generate explainers is an anti-pattern -- the spec should do what an explainer does in the introduction, and if it doesn't do that, the introduction of the specification needs to be changed. This would result in easier to read specifications at W3C and reduce the amount of time that Editors spend writing single use documents for the TAG that are not maintained.

This seems to me to be a good direction to head.

[torgo]

A few points:

I disagree with the statement (and thinking) that the introduction to the spec should cover everything that should be in an explainer. In particular, an explainer should cover a lot more about the user needs (motivating use cases) and the context within which the spec is written (such as alternatives considered) then what specification authors might expect to put into a spec itself. In the TAG, we've had general good outcomes from encouraging spec authors to also write explainers. Explainers should also be more "accessible" documents. We've had a lot of feedback from developers and others that W3C /tr space documents are very intimidating. An explainer is supposed to be more friendly and less formal.

However, if a spec author feels they can cover everything that we would expect to see in an explainer in the intro to the spec itself, there's no hard and fast rule saying they can't do it that way.

Also the explainer is not "for" the TAG. The explainer is for the group that's writing the spec. We see explainers as something that can eventually be used to produce developer documentation for use of the specification, so it's in the spec writer's best interest to keep it up to date.

[msporny]

I disagree with the statement (and thinking) that the introduction to the spec should cover everything that should be in an explainer.

Hmm, that's not what I was trying to convey -- perhaps I should rewrite my initial text to "introductory portions of the specification". If you can't clearly convey what your spec is trying to do to a casual reader in the top 25% of the spec, then you probably don't have a very good spec. I agree that not EVERYTHING in the explainer belongs in the introductory portions, but most of it could (and probably should). [EDIT: I have reworded the original issue to say "introductory portions"]

In particular, an explainer should cover a lot more about the user needs (motivating use cases) and the context within which the spec is written (such as alternatives considered) then what specification authors might expect to put into a spec itself.

I do think this discussion is about expectations -- I would expect much of what's in an explainer to be in the "front matter" of a specification, and for the other stuff (alternative design considerations) to be a part of what's submitted to the design review process.

In the TAG, we've had general good outcomes from encouraging spec authors to also write explainers.

This has not been my experience and it has not been the experience of the vast majority of people I've worked with that have written explainers. It's viewed as a "thing that I have to do to get horizontal review from the TAG".

Explainers should also be more "accessible" documents.

The introductory portions of a spec should be "accessible", we shouldn't move the accessible portion of why the spec exists to an external document that is rarely ever used after it is created.

We've had a lot of feedback from developers and others that W3C /tr space documents are very intimidating.

Oh, yes, they are -- and they're that way because the way we go about writing W3C specifications are problematic -- diving right into the technical content w/o giving a justification for the specifications existence to a casual reader.

An explainer is supposed to be more friendly and less formal.

Why aren't we putting (most) of this in the introductory portion of the specification?

However, if a spec author feels they can cover everything that we would expect to see in an explainer in the intro to the spec itself, there's no hard and fast rule saying they can't do it that way.

From what I gather, an Explainer is a non-negotiable requirement of the TAG's horizontal review process. Even if it was negotiable, any WG that opts to not write an Explainer runs the risk of having their horizontal review held up due to the lack of an explainer and debating whether or not it was covered in the specification.

Also the explainer is not "for" the TAG.

The only reason the groups I'm involved with create explainers is because the TAG requires it.

The explainer is for the group that's writing the spec.

That has not been my experience in the groups that I participate in.

We see explainers as something that can eventually be used to produce developer documentation for use of the specification, so it's in the spec writer's best interest to keep it up to date.

My experience is that developer documentation comes from a very different process, with the specification as input to a group of technical writers that have their own processes and communities for which they are writing documentation. It might be that much of the Explainer process is built around the idea of Web APIs and that doesn't really work well for other things that are not API-like in nature.

[jyasskin]

More later, but we have #6 to look for consensus on what should happen to an explainer after a spec is written.

[frivoal]

I agree with @msporny in thinking that specs should have good intros, explanatory text, notes, and examples, and be organized to be humanly readable (typically: general high level things upfront, fiddly details as you keep on reading), not just technically correct. A spec that immediately jumps into dry technical details (and hardly contains anything else), to me, is not a very good one. It is very hard to form an opinion on what a spec is doing if you have no idea why it's doing it, and I think it is important for implementers to understand that (how else are they going to catch mistakes or possible improvements)? Examples and notes that illustrate how this all fits together and what it's for also help with that, while simultaneously helping non-implementer readers figure out whether this technology is the right fit for the use-case they're trying to solve, and how it's meant to be used.

If such content benefits all readers of the document, then it should be in the document.

If you do that well, that covers a lot of what explainers are supposed to be, and doesn't have the downside of becoming quickly stale and unmaintained, as @msporny noted is often the case for Explainers.

One thing that even well written explanatory specs rarely cover though is why you picked that particular design, what prior art was studied, which alternatives were considered and rejected… You could stick it into an appendix, but that's rarely done too. Using explainers for that sort of things seems OK to me: the original design choices and alternatives that were considered were considered at some point in time, and while it is useful to have a long lived archive of that description, it is not obvious to me that it needs continuous maintenance in the way the rest of the spec does.

[csarven]

Just to clarify the context in which we/TAG (with me as the primary reviewer) approached this horizontal review: we focused on the most significant changes, which were class 3. I'll take the blame for not reading the Explainer thoroughly but only because it didn't seem to include what we needed for an effective review.

The Explainer gives a good general overview, but not for the core ask of the DID 1.1 horizontal review. What helped more was digging into the class 3 changes, linked issues, PRs, and even group minutes to understand both the decisions and the context. To me, that was far more useful than this particular Explainer, but I don't at all doubt its value for anyone approaching it differently.

DID had already proven itself, so the review wasn't about starting from scratch (nor was that requested). We were focused on the "diff". Not reading the Explainer thoroughly had nothing to do with it being drafted by an LLM and edited by a human. I went through the relevant material to make sure the review was effective and to flag any architectural concerns. Pardon me if I didn't communicate that clearly. I'll try to do better on that front.

[jyasskin]

My sense, which is not shared by all the people who review explainers, is that explainers are most useful for the incubation stage, where people are exploring an idea and haven't written a specification yet. You'll note that https://w3ctag.github.io/explainer-explainer/#introduction spends almost all of its time there, and only spends the last sentence on "Once the spec is written".

Several explainer sections fit naturally into the spec: Introduction, Goals, Non-goals, "proposed approach". "User-Facing Problem" and "User Research" fit less well, but they might make a good addition in general. As Florian mentioned, "Alternatives Considered" doesn't fit as well, and although I'd like to see more of it in spec appendices, I've heard objections to putting it there.

That means that our horizontal review intake process shouldn't imply that you need an explainer, and should instead ask that the spec cover the same topics. There's a suggestion in w3ctag/design-reviews#1088 (comment) to split HRs from incubation and browser-shipping reviews, and this might be an extra prod to make that happen.

And it's possible that the intake for later spec versions should be different again: explaining the overall point of the spec doesn't help a whole lot if we're only meant to review the changes. The DID 1.1 summary was good on that point, with the link to closed class-3 issues, but it didn't cover the user-facing problem, alternatives considered, etc for the changes.

[LJWatson]

@msporny, I'm curious about one of the points you make, specifically that explainers are read once by a handful of people and not again until the next horizontal review. Can you share more on the data that underpins this?

[msporny]

@LJWatson wrote:

I'm curious about one of the points you make Manu, specifically that explainers are read once by a handful of people and not again until the next horizontal review. Can you share more on the data that underpins this?

This is all anecdotal, as I don't think we have good data on Explainers in general. I'm trying to add some data points to the case of moving most of the Explainer content into the specifications themselves.

My experience has been that the WGs I've been involved with over the past 18+ years have not used an Explainer beyond its use in a charter or horizontal review. This is based on two observations:

1. The WG doesn't point people to the Explainers, but rather to their specifications or use cases. Sometimes there is an overview document if there are a large number of specifications that fit together into some larger solution, but the purpose of that document (navigating the specifications) is different from an Explainer (an approachable description of the technology).

2. We get next to no comments on the Explainer itself, even after years of document rot, which tells me that no one is reading them. The WG puts next to no effort into the Explainer after it is created. We do get comments on the specifications, though, and track those issues and fix them as the WG Charters allow... so, people are reading the WG specifications, just not the Explainer.

The thing I'm most concerned about here, aside from inefficient use of WG and Editor time, is that having an explainer seems to make it such that you don't need to make your specification approachable to a more general audience (such as a student, legislator, or a reporter). I don't understand why the TAG (which is where the requirement for an explainer comes from, AFAICT) doesn't require the specification front matter to be approachable in the same way they require the Explainer to be approachable. If the criticism is that W3C specifications are not approachable, why don't we focus on fixing that problem instead of trying to fix the problem elsewhere?

To ask a variation of the question you are raising: Where is the data to show that Explainers are read by more general audiences?

[msporny]

@frivoal wrote:

One thing that even well written explanatory specs rarely cover though is why you picked that particular design, what prior art was studied, which alternatives were considered and rejected… You could stick it into an appendix, but that's rarely done too.

Yes, agreed, but if we move just about everything else in an Explainer into the spec... maybe we don't need Explainers anymore and can instead ask different questions during Horizontal Review, like:

• "Link to the section in the specification that, to a general audience, describes the problem and the specified solution."
• "Link to the use cases for your specification."
• "Link to the design goals (and non-goals, if any) for your specification."
• "Link to the alternative solutions to the problem space." <-- don't see why that couldn't go in an appendix in the spec. If it doesn't change, no problem.

We might do this in the same way that we ask for links to the Security Considerations and Privacy Considerations sections during the SING and PING reviews.

Anything else might be best entered into the horizontal review issue itself.

[msporny]

Just to clarify the context in which we/TAG (with me as the primary reviewer) approached this horizontal review: we focused on the most significant changes, which were class 3. I'll take the blame for not reading the Explainer thoroughly but only because it didn't seem to include what we needed for an effective review.

I certainly didn't mean to imply that you did not do the right thing -- your review was exactly the sort of thing I would expect from a v1.1 review... and the request for an Explainer in the issue review template was what I was questioning the value of, especially for a point release, but the question carried over to whether we needed one or not (and if so, could we just use AI to generate it)?

The Explainer gives a good general overview, but not for the core ask of the DID 1.1 horizontal review.

Well, the FPWD horizontal review issue template asks for an Explainer, which is what I'm questioning in general.

What helped more was digging into the class 3 changes, linked issues, PRs, and even group minutes to understand both the decisions and the context.

Yes, and is what I'd expect to be the only ask for a point release for a specification. Perhaps we need a "Horizontal Review of a maintenance release of a specification" issue template?

To me, that was far more useful than this particular Explainer, but I don't at all doubt its value for anyone approaching it differently.

Yes, though I am doubting the value of the Explainer in the horizontal review process.

DID had already proven itself, so the review wasn't about starting from scratch (nor was that requested).

The horizontal review issue template requests the same stuff regardless of whether something is a point release or not. I might have missed the issue template for a horizontal review request for a point release?

Not reading the Explainer thoroughly had nothing to do with it being drafted by an LLM and edited by a human.

The part that I was responding to was the corresponding discussion in the TAG about dissuading people from using LLMs to generate this sort of documentation. I don't want to see that become a TAG position without some serious soul searching about the use of LLMs to improve the work we do here (which includes the sort of work we don't want LLMs to do). I doubt many of us question the value of spelling and grammar checkers... and soon, I hope we won't reject content entirely "written" or summarized by non-humans if it meets or exceeds the quality bar for the task at hand. In this particular case, the LLM came close and the human barely had to change much to get it over the line.

I went through the relevant material to make sure the review was effective and to flag any architectural concerns. Pardon me if I didn't communicate that clearly. I'll try to do better on that front.

You communicated that clearly and did a solid job on the review, IMHO. My concerns were not necessarily about this particular horizontal review -- it was about the Explainer requirement in general, and the Explainer requirement in a point release review.