Docs updates, separates nix and non-nix install processes, edits to simplify for both. Update to add index for coherency for devs and reduce some redundancy. (#1379)

Author: daveoconnor

.envrc

@@ -3,6 +3,6 @@ if [ -f ".env" ]; then
   dotenv
 fi
 if [ -f "$HOME/.config/nix/nix.conf" ]; then
-  echo "Entering nix shell environment."
+  echo "Loading nix shell environment."
   use flake
 fi

README.md

@@ -6,7 +6,7 @@ A Django based website that will power a new Boost website. See the [documentati
 
 Links:
 
-- https://www.stage.boost.cppalliance.org/ - staging
+- https://stage.boost.org/ - staging
 - https://www.boost.org/ - production
 
 ---
@@ -15,54 +15,65 @@ Links:
 
 This project will use Python 3.11, Docker, and Docker Compose.
 
-Instructions to install those packages are included in [development_setup_notes.md](docs/development_setup_notes.md).
-
-**NOTE**: All of these various `docker compose` commands, along with other helpful
-developer utility commands, are codified in our `justfile` and can be ran with
-less typing.
-
-You will need to install `just`, by [following the documentation](https://just.systems/man/en/)
-
-**Environment Variables**: Copy file `env.template` to `.env` and adjust values to match your local environment. See [Environment Variables](docs/env_vars.md) for more information.
-
-```shell
-$ cp env.template .env
-```
-
-**NOTE**: Double check that the exposed port assigned to the PostgreSQL
-container does not clash with a database or other server you have running
-locally.
-
-Then run:
+There are two options for development setups, nix and native. The native setup doesn't require nix to be installed. The nix-based setup works in a similar way to a python venv, with everything encapsulated for the project. Note: it is not a vm. It handles installation of all of the application and development environment's dependencies automatically:
+
+- The python version relevant to the project
+- awscli
+- gdk
+- just
+- opentofu
+- nodejs
+- yarn
+- asciidoctor
+- asciidoctor-boost gem
+- pre-commit
+- black
+- isort
+- pip-tools
+
+
+1. Development System setup
+   1. Basic Setup Options:
+      1. [Nix-based](docs/development_setup_notes_nix.md)
+      1. [Native](docs/development_setup_notes_native.md)
+   1. [Initial Data Setup](docs/first_time_data_import.md)
+   1. [Allauth Social Login Setup](docs/allauth_setup.md)
+1. System Concepts
+   1. [GitHub Data Syncing](docs/syncing_data_with_github.md)
+   1. [Rendered Content Caching](docs/caching_rendered_content.md)
+   1. [Static Content from S3](docs/static_content.md)
+   1. ['manage' Commands](docs/commands.md)
+   1. [API](docs/api.md)
+   1. [RSS Feeds](docs/rss_feeds.md)
+   1. [Env Vars](docs/env_vars.md)
+   1. [Calendar](docs/calendar.md)
+   1. [News](docs/news.md)
+1. Instructional
+   1. [Dependency Updates](docs/dependencies.md)
+   1. [Release Reports](docs/release_reports.md)
+   1. [User Management](docs/user_management.md)
+   1. [Non-Dev Server Allauth Setup](docs/non-dev-server-allauth-setup.md)
+   1. [Admin Features](docs/admin.md)
+   1. [Mailing List Setup](docs/mailing_list.md)
+
+After going through the "Development System setup" steps above to create the Docker image, install dependencies, and start the services in `docker-compose.yml`, run:
 
 ```shell
 # start our services (and build them if necessary)
 $ docker compose up
 
 # to create database migrations
-$ docker compose run --rm web python manage.py makemigrations
+$ just makemigrations
 
 # to run database migrations
-$ docker compose run --rm web python manage.py migrate
+$ just migrate
 
 # to create a superuser
-$ docker compose run --rm web python manage.py createsuperuser
+$ just manage createsuperuser
 ```
 
-This will create the Docker image, install dependencies, start the services
-defined in `docker-compose.yml`, and start the webserver.
-
 styles.css is still missing in a local docker-compose environment. Steps to add it:
 
-```
-# One-time setup
-curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.0/install.sh | bash
-. ~/.bashrc
-nvm install 20
-nvm use 20
-npm install -g yarn
-```
-
 ```
 # Each time - rebuild styles.css
 yarn
@@ -90,13 +101,13 @@ $ docker compose down
 To run the tests, execute:
 
 ```shell
-$ docker compose run --rm web pytest
+$ just test
 ```
 
 or run:
 
 ```shell
-$ just test
+$ docker compose run --rm web pytest
 ```
 
 ## Yarn and Tailwind
@@ -125,22 +136,6 @@ currently generated by `docker compose build` and is included in the Docker imag
 
 ---
 
-## Generating Local Data
-
-To **add real Boost libraries and sync all the data from GitHub and S3**, set appropriate values in your new .env file according to [Environment Variables](docs/env_vars.md) for `GITHUB_TOKEN`, `STATIC_CONTENT_AWS_ACCESS_KEY_ID`, `STATIC_CONTENT_AWS_SECRET_ACCESS_KEY`, `STATIC_CONTENT_BUCKET_NAME`, `STATIC_CONTENT_REGION`, `STATIC_CONTENT_AWS_S3_ENDPOINT_URL` then run:
-
-```bash
-docker compose run --rm web python manage.py update_libraries --local
-```
-
-Those values can be gotten from another developer in the `#boost-website` Slack channel.
-
-The `--local` flag speeds up the command a lot by excluding the retrieval of data you generally don't need. For more information, see `update_libraries` in [Management Commands](docs/commands.md).
-
-Then as a superuser log into the admin interface, go to "Versions" and click on the "import new releases" button in the top right.
-
----
-
 ## Setting up Mailman/Hyperkitty locally
 ``shell
 sudo apt-get install sassc
@@ -190,17 +185,6 @@ We use [pre-commit hooks](https://pre-commit.com/) to check code for style, synt
 | [Ruff](https://github.com/charliermarsh/ruff) | Wrapper around `flake8` and `isort`, among other linters |
 | [Djhtml](https://github.com/rtts/djhtml) | Auto-formats Django templates |
 
-### Setup and Usage
-
-| Description | Command |
-| ---- | ------- |
-| 1. Install the `pre-commit` package using `pip` | `pip install pre-commit` |
-| 2. Install our list of pre-commit hooks locally | `pre-commit install` |
-| 3. Run all hooks for changed files before commit | `pre-commit run` |
-| 4. Run specific hook before commit | `pre-commit run {hook}` |
-| 5. Run hooks for all files, even unchanged ones | `pre-commit run --all-files` |
-| 6. Commit without running pre-commit hooks | `git commit -m "Your commit message" --no-verify` |
-
 Example commands for running specific hooks:
 
 | Hook | Example |

docs/README.md

@@ -4,7 +4,7 @@
 - [Boost Mailing List](./mailing_list.md) -Includes OAuth instructions
 - [Caching and the `RenderedContent` model](./caching_rendered_content.md)
 - [Dependency Management](./dependencies.md)
-- [Development Setup Notes](./development_setup_notes.md)
+- [Development Setup Notes](development_setup_notes_native.md)
 - [Environment Variables](./env_vars.md)
 - [Events Calendar](./calendar.md)
 - [Example Files](./examples/README.md) - Contains samples of `libraries.json`. `.gitmodules`, and other files that Boost data depends on

docs/allauth_setup.md

@@ -0,0 +1,59 @@
+### Social Login with django-allauth
+
+Follow these instructions to use the social logins through django-allauth on your local machine.
+
+See https://testdriven.io/blog/django-social-auth/ for more information.
+
+#### Github
+- Go to https://github.com/settings/applications/new and add a new OAuth application
+- Set `http://localhost:8000` as the Homepage URL
+- Set `http://localhost:8000/accounts/github/login/callback/` as the Callback URL
+- Click whether you want to enable the device flow
+   - <img src="https://user-images.githubusercontent.com/2286304/252841283-9a846c68-46bb-4dac-8d1e-d35270c09f1b.png" alt="The GitHub screen that registers a new OAuth app" width="400">
+- On completion copy the Client ID and Client Secret to the `.env` file as values of `GITHUB_OAUTH_CLIENT_ID` and `GITHUB_OAUTH_CLIENT_SECRET`.
+- Run `direnv allow` and restart your docker containers.
+
+Setup should be complete and you should be able to see an option to "Use Github" on the sign up page.
+
+To test the flow including authorizing Github for the Boost account, log into your GitHub account settings and click **Applications** in the left menu. Find the "Boost" authorization and delete it. The next time you log into Boost with this GitHub account, you will have to re-authorize it.
+
+<img src="https://user-images.githubusercontent.com/2286304/204642346-8b269aaf-4693-4351-9474-0a998b97689c.png" alt="The 'Authorized OAuth Apps' tab in your GitHub Applications" width="400">
+
+This setup process is not something that can currently be automated through terraform because of a lack of relevant Github API endpoints to create Oauth credentials.
+
+#### Google
+
+More detailed instructions at:
+
+https://docs.allauth.org/en/latest/socialaccount/providers/google.html
+
+1. Update the `.env` file with values for:
+   1. `TF_VAR_google_cloud_email` (the email address of your Google Cloud account)
+   2. `TF_VAR_google_organization_domain` (usually the domain of your Google Cloud account, e.g. "boost.org" if you will be using a @boost.org email address)
+   3. `TF_VAR_google_cloud_project_name` (optional, default: localboostdev) - needs to change if destroyed and a setup is needed within 30 days
+2. Run `just development-tofu-init` to initialize tofu.
+3. Run `just development-tofu-plan` to confirm the planned changes.
+4. Run `just development-tofu-apply` to apply the changes.
+5. Go to https://console.developers.google.com/
+   1. Search for the newly created project, named "Boost Development" (ID: localboostdev by default).
+   2. Type "credentials" in the search input at the top of the page.
+   3. Select "Credentials" under "APIs & Services".
+      1. Click "+ CREATE CREDENTIALS"
+      2. Select "OAuth Client ID"
+      3. Select Application Type: "Web application"
+      4. Name: "Boost Development" (arbitrary)
+      5. For "Authorized Javascript Origins" use:`http://localhost:8000`
+      6. For "Authorized Redirect URIs" use:
+         * `http://localhost:8000/accounts/google/login/callback/`
+         * `http://localhost:8000/accounts/google/login/callback/?flowName=GeneralOAuthFlow`
+      7. Save
+6. From the page that's displayed, update the `.env` file with values for the following:
+   - `GOOGLE_OAUTH_CLIENT_ID` should be similar to "k235bn2b1l1(...)asdsk.apps.googleusercontent.com"
+   - `GOOGLE_OAUTH_CLIENT_SECRET` should be similar to "LAJACO(...)KLAI612ANAD"
+
+Point 5 above can not be automated through terraform because of a lack of relevant Google Cloud API endpoints to create Oauth credentials.
+
+Setup should be complete and you should be able to see an option to "Use Google" on the sign up page.
+
+#### Additional Notes on allauth login flows:
+**Working locally**: If you need to run through the login flows multiple times, create a superuser so you can log into the admin. Then, log into the admin and delete your "Social Account" from the admin. This will test a fresh connection to GitHub for your logged-in GitHub user.

docs/development_setup_notes.md renamed to docs/development_setup_notes_native.md

@@ -271,72 +271,24 @@ cd website-v2
 cp env.template .env
 ```
 
-Edit the .env, adding AWS keys.
-
-Continue to the instructions in the top-level README.md file.
-
-## Local Development
-
-### Social Login with django-allauth
-
-Follow these instructions to use the social logins through django-allauth on your local machine.
-
-See https://testdriven.io/blog/django-social-auth/ for more information.
-
-#### Github
-- Go to https://github.com/settings/applications/new and add a new OAuth application
-- Set `http://localhost:8000` as the Homepage URL
-- Set `http://localhost:8000/accounts/github/login/callback/` as the Callback URL
-- Click whether you want to enable the device flow
-   - <img src="https://user-images.githubusercontent.com/2286304/252841283-9a846c68-46bb-4dac-8d1e-d35270c09f1b.png" alt="The GitHub screen that registers a new OAuth app" width="400">
-- On completion copy the Client ID and Client Secret to the `.env` file as values of `GITHUB_OAUTH_CLIENT_ID` and `GITHUB_OAUTH_CLIENT_SECRET`.
-- Run `direnv allow` and restart your docker containers.
+Edit the .env, adding AWS keys, and adjust values to match your local environment. See [Environment Variables](docs/env_vars.md) for more information.
 
-Setup should be complete and you should be able to see an option to "Use Github" on the sign up page.
+**NOTE**: Double check that the exposed port assigned to the PostgreSQL
+container does not clash with a database or other server you have running
+locally.
 
-To test the flow including authorizing Github for the Boost account, log into your GitHub account settings and click **Applications** in the left menu. Find the "Boost" authorization and delete it. The next time you log into Boost with this GitHub account, you will have to re-authorize it.
+### Pre-commit Hooks Setup
 
-<img src="https://user-images.githubusercontent.com/2286304/204642346-8b269aaf-4693-4351-9474-0a998b97689c.png" alt="The 'Authorized OAuth Apps' tab in your GitHub Applications" width="400">
+| Description | Command |
+| ---- | ------- |
+| 1. Install the `pre-commit` package using `pip` | `pip install pre-commit` |
+| 2. Install our list of pre-commit hooks locally | `pre-commit install` |
+| 3. Run all hooks for changed files before commit | `pre-commit run` |
+| 4. Run specific hook before commit | `pre-commit run {hook}` |
+| 5. Run hooks for all files, even unchanged ones | `pre-commit run --all-files` |
+| 6. Commit without running pre-commit hooks | `git commit -m "Your commit message" --no-verify` |
 
-This setup process is not something that can currently be automated through terraform because of a lack of relevant Github API endpoints to create Oauth credentials.
-
-#### Google
-
-More detailed instructions at:
-
-https://docs.allauth.org/en/latest/socialaccount/providers/google.html
-
-1. Update the `.env` file with values for:
-   1. `TF_VAR_google_cloud_email` (the email address of your Google Cloud account)
-   2. `TF_VAR_google_organization_domain` (usually the domain of your Google Cloud account, e.g. "boost.org" if you will be using an @boost.org email address)
-   3. `TF_VAR_google_cloud_project_name` (optional, default: localboostdev) - needs to change if destroyed and a setup is needed within 30 days
-2. Run `just development-tofu-init` to initialize tofu.
-3. Run `just development-tofu-plan` to confirm the planned changes.
-4. Run `just development-tofu-apply` to apply the changes.
-5. Go to https://console.developers.google.com/
-   1. Search for the newly created project, named "Boost Development" (ID: localboostdev by default).
-   2. Type "credentials" in the search input at the top of the page.
-   3. Select "Credentials" under "APIs & Services".
-      1. Click "+ CREATE CREDENTIALS"
-      2. Select "OAuth Client ID"
-      3. Select Application Type: "Web application"
-      4. Name: "Boost Development" (arbitrary)
-      5. For "Authorized Javascript Origins" use:`http://localhost:8000`
-      6. For "Authorized Redirect URIs" use:
-         * `http://localhost:8000/accounts/google/login/callback/`
-         * `http://localhost:8000/accounts/google/login/callback/?flowName=GeneralOAuthFlow`
-      7. Save
-6. From the page that's displayed, update the `.env` file with values for the following:
-   - `GOOGLE_OAUTH_CLIENT_ID` should be similar to "k235bn2b1l1(...)asdsk.apps.googleusercontent.com"
-   - `GOOGLE_OAUTH_CLIENT_SECRET` should be similar to "LAJACO(...)KLAI612ANAD"
-7. Run `direnv allow` and restart your docker containers.
-
-Point 5 above can not be automated through terraform because of a lack of relevant Google Cloud API endpoints to create Oauth credentials.
-
-Setup should be complete and you should be able to see an option to "Use Google" on the sign up page.
-
-#### Additional Notes:
-**Working locally**: If you need to run through the login flows multiple times, create a superuser so you can log into the admin. Then, log into the admin and delete your "Social Account" from the admin. This will test a fresh connection to GitHub for your logged-in GitHub user.
+Continue to the instructions in the top-level README.md file.
 
 ### Debugging
 For local development there is Django Debug Toolbar, and the option to set a debugger.