Skip to content

Conversation

RuthTurk
Copy link
Member

@RuthTurk RuthTurk commented Sep 13, 2025

🗒️ Checklist

  1. run linter locally
  2. update developer docs (API, README, inline, etc.), if any
  3. for user-facing doc changes create a Zulip thread at #Kobo support docs, if any
  4. draft PR with a title <type>(<scope>)<!>: <title> DEV-1234
  5. assign yourself, tag PR: at least Front end and/or Back end or workflow
  6. fill in the template below and delete template comments
  7. review thyself: read the diff and repro the preview as written
  8. open PR & confirm that CI passes & request reviewers, if needed
  9. delete this section before merging

📣 Summary

Add a new superuser-only endpoint, /api/v2/user-reports/, to access and filter all user usage data.

💭 Notes

  • Created a new Django app user_reports
  • Used a SQL materialized view along with some Django tables to gather and calculate all the data for the endpoint
  • Our celery task refresh-user-report-snapshot runs every 30 minutes and works to update the data in batches
  • The endpoint is only accessible if Stripe is enabled
  • Renamed api tag "Audit logs (superusers)" to "Server logs (superusers)" (Zulip thread)

👀 Preview steps

  1. ℹ️ have a superuser account and multiple regular users in multiple organizations
  2. create projects, make submissions, and buy stripe subscriptions
  3. checkout this branch and run migrations for user_reports
  4. login as a regular user
  5. 🟢 checkout /api/v2/user-reports/ and get a 403 Forbidden error
  6. login as a superuser
  7. run the celery task to refresh the data: refresh-user-report-snapshot
  8. 🟢checkout /api/v2/user-reports/ and get 200 with all the users and data
  9. verify that all data is correct
  10. make some submissions
  11. 🟢 verify that the data has been correctly refreshed

@RuthTurk RuthTurk force-pushed the dev-899-optimize-queries-for-subscriptions branch from 507182e to 3f421d1 Compare September 16, 2025 15:22
@rajpatel24 rajpatel24 requested a review from noliveleger October 6, 2025 15:07
@noliveleger noliveleger changed the title feat(api): add /user-reports/ endpoint for superusers feat(api): add /api/v2/user-reports/ endpoint for superusers Oct 7, 2025
@RuthTurk RuthTurk changed the title feat(api): add /api/v2/user-reports/ endpoint for superusers feat(api): add /api/v2/user-reports/ endpoint for superusers DEV-232 DEV-899 Oct 8, 2025
@RuthTurk RuthTurk marked this pull request as ready for review October 8, 2025 15:20
@RuthTurk RuthTurk added the API Changes related to API endpoints label Oct 8, 2025
lock_timeout = settings.CELERY_LONG_RUNNING_TASK_TIME_LIMIT + 60
lock = cache.lock(cache_key, timeout=lock_timeout)
if not lock.acquire(blocking=False, blocking_timeout=0):
logging.error('Task is already running')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was for debugging purpose. I'm wondering if we should keep this. It will probably flood Sentry logs for nothing. Maybe we can change it for logging.info(' Nothing to do, task is already running!)?
What do you think?

Comment on lines 237 to 239
with connection.cursor() as cursor:
cursor.execute('REFRESH MATERIALIZED VIEW user_reports_userreportsmv;')

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use an utility function to avoid repeating thse two lines several times?

Comment on lines 30 to 34
# Refresh materialized view
with connection.cursor() as cursor:
cursor.execute(
'REFRESH MATERIALIZED VIEW CONCURRENTLY user_reports_userreportsmv;'
)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can probably use the same helper here and in the tests.

) FILTER (WHERE sub.id IS NOT NULL),
'[]'::jsonb
)::text AS subscriptions
FROM auth_user au
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We probably want to remove anonymous user from the list (e.g. auth_user.id != settings.ANONYMOUS_ID)

@RuthTurk RuthTurk requested a review from noliveleger October 9, 2025 19:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
API Changes related to API endpoints Back end
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants