enh: reviewer-guide - review the reviewer checklist #321
Description
After talking with @stefanv who supported a review recently, I am adding some notes about things we should consider updating in our review checklists. The big picture is to think about what items make a package usable enough. So, for instance, do all packages need a full-blown docs website? In some cases, a thorough README and contributing guide could be enough (setuptools_scm has a long readme although i didn't love having to use that and it's an extension/plugin tool).
- The word vignette is still used there twice. We should change that to short, quickstart code examples or short tutorials that quickly allow a user to get started with using your package. The r community knows the word vignette. The Python community doesn't
- Some of the items in the checklist are pretty specific. For example, the badges might not be required for a review, but they are good to have. The pyOS badge is, of course, required, but a reviewer doesn't need to ensure they have a repostatus badge, for instance. We should talk more about CI badges for tests and test coverage on Slack.
A question that we haven't discussed in our community is the scope of the review. Right now, we focus on usability from a user perspective. But a package reviewed that is super hard to build/compile locally as it's connected to a specific tool (like PDM) might make future contributions challenging as an example. Should we add a developer or contributor layer to our review?
- Because the JOSS section is OPTIONAL, we should make it a "drop-down" in the template so it can be opened and used only if the author wishes to also submit to JOSS
The quick action items here are:
- Remove the word vignette and use quickstart code snippet / tutorial instead
- Reconsider some of the super-specific items like the badges and make them a "good to consider" rather than required
- Run a community review of the updated template.
Stefan, if I missed anything, please let me know!