Skip to content

WiP: replace existing client-driven cache updates with a central implementation in the janitor service #341

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

WiP: replace existing client-driven cache updates with a central implementation in the janitor service #341

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions mig/lib/janitor.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
#!/usr/bin/python
# -*- coding: utf-8 -*-
#
Expand Down Expand Up @@ -26,7 +26,7 @@
# -- END_HEADER ---
#

"""Helpers for the janitor service which takes care of various recurring tasks

Check warning on line 29 in mig/lib/janitor.py

View workflow job for this annotation

GitHub Actions / Style check python and annotate 

First line should end with a period (not 's')

Check warning on line 29 in mig/lib/janitor.py

View workflow job for this annotation

GitHub Actions / Style check python and annotate 

1 blank line required between summary line and description (found 0)
like clean up, cache updates and pruning of pending requests.
"""

Expand Down Expand Up @@ -56,11 +56,11 @@
SECS_PER_HOUR = 60 * SECS_PER_MINUTE
SECS_PER_DAY = 24 * SECS_PER_HOUR

task_triggers = {}

Check failure on line 59 in mig/lib/janitor.py

View workflow job for this annotation

GitHub Actions / Style check python and annotate 

Need type annotation for "task_triggers" (hint: "task_triggers: Dict[<type>, <type>] = ...")  [var-annotated]


def _lookup_last_run(configuration, target):
"""Check if target task is pending using internal accounting for task.

Check warning on line 63 in mig/lib/janitor.py

View workflow job for this annotation

GitHub Actions / Style check python and annotate 

1 blank line required between summary line and description (found 0)
Returns the timestamp when the task was last run in UN*X epoch.
"""
_logger = configuration.logger
Expand All @@ -74,7 +74,7 @@


def _update_last_run(configuration, target, stamp):
"""Update target task pending mark using internal accounting and supplied

Check warning on line 77 in mig/lib/janitor.py

View workflow job for this annotation

GitHub Actions / Style check python and annotate 

First line should end with a period (not 'd')

Check warning on line 77 in mig/lib/janitor.py

View workflow job for this annotation

GitHub Actions / Style check python and annotate 

1 blank line required between summary line and description (found 0)
task timestamp in UN*X epoch.
Returns the same updated timestamp for the task.
"""
Expand All @@ -92,7 +92,7 @@
now,
include_dotfiles=False,
):
"""Inspect and clean up stale state files matching any of filename_pattern

Check warning on line 95 in mig/lib/janitor.py

View workflow job for this annotation

GitHub Actions / Style check python and annotate 

First line should end with a period (not 'n')

Check warning on line 95 in mig/lib/janitor.py

View workflow job for this annotation

GitHub Actions / Style check python and annotate 

1 blank line required between summary line and description (found 0)
in target_dir if they are at least expire_days old. Where filename_pattern
is a list of wildcard strings checked with fnmatch. Dot-files are excluded
from matching unless include_dotfiles is set. Directories are just skipped.
Expand Down Expand Up @@ -135,7 +135,7 @@


def clean_mig_system_files(configuration, now=None):
"""Inspect and clean up stale state files in mig_system_run.

Check warning on line 138 in mig/lib/janitor.py

View workflow job for this annotation

GitHub Actions / Style check python and annotate 

1 blank line required between summary line and description (found 0)
Returns the number of actual actions taken for central throttle handling.
"""
if now is None:
Expand All @@ -150,7 +150,7 @@


def clean_sessid_to_mrls_link_home(configuration, now=None):
"""Inspect and clean up stale state files in sessid_to_mrsl_link_home.

Check warning on line 153 in mig/lib/janitor.py

View workflow job for this annotation

GitHub Actions / Style check python and annotate 

1 blank line required between summary line and description (found 0)
Returns the number of actual actions taken for central throttle handling.
"""
if now is None:
Expand Down Expand Up @@ -494,6 +494,7 @@
handled = 0
_logger.debug("handle pending cache updates")
# TODO: actually handle vgrid/user/resource/... cache updates
# TODO: rip out the previous client-driven cache updates ones implemented
if handled > 0:
_logger.info("handled %d pending cache updates" % handled)
else:
Expand All @@ -501,7 +502,7 @@
return handled


def handle_janitor_tasks(configuration, now=None):

Check failure on line 505 in mig/lib/janitor.py

View workflow job for this annotation

GitHub Actions / Style check python and annotate 

unused function 'handle_janitor_tasks' (60% confidence)
"""A wrapper to take care of all regular janitor tasks like clean up and
cache updates.
Returns the number of actual tasks completed to let the main thread know if
Expand Down