mypy and logging.Logger subclasses

[randolf-scholz]

EDIT: I added a pull request in python/typeshed to enable workaround 2.

• issue in typeshed: Logger.getChild bad signature typeshed#6606
• pr: Logger.getChild subclass compatible typehint typeshed#6609

---

So, let's say we want to use a custom logging.Logger sub-class with additional methods e.g. trace, verbose, passed, failed that provide specialized formatting and/or use custom log-levels.

• Python3 add logging level
• How to extend the logging.Logger Class?
• How to add a custom loglevel to Python's logging facility

However, the proposed solutions, either monkey-patching or using a logging.Logger subclass both do not work well with mypy, because the standard way to get a Logger is using logging.getLogger which is unaware of the subclass. See also this blog post by Sam Hooke.

The issue

Suppose we have the following setup code: In config/base module we define a new Logger subclass:

class MyLogger(logging.Logger):
    ...

logging.setLoggerClass(MyLogger)
logger = logging.getLogger(__name__)

Then in submodules:

import logging
logger = logging.getLogger(__name__)

We will get type errors (undefined attribute) each and every time we do logger.verbose / logger.trace etc. There was already a very brief discussion here python/typeshed#1801 without any good resolution.

Possible workarounds

So I was wondering, what could be an acceptable solution? Plastering # type: ignore every time we use logger.verbose does not seem reasonable to me. I think there are two possibilities, which however currently are not quite compatible with mypy.

Workaround proposal 1 - add explicit type hint for the subclass

import logging
from mymodule.logger import MyLogger
logger: MyLogger = logging.getLogger(__name__)

This does not quite work because getLogger has the following signature in typeshed/stdlib/logging`:

def getLogger(name: Optional[str] = ...) -> Logger: ...

Instead, we would need something like

T = TypeVar("T", bound=Logger) 
def getLogger(name: Optional[str] = ...) -> T: ...

In conjunction with a change of how TypeVar works: if no type hint is given assume the type is the bound type. This was discussed here python/mypy#4236 with no resolution.

Workaround proposal 2 - Import the Logger object and create a child.

from mymodule.logger import logger   # returns actual object
logger = logger.getChild(__name__)

Here are 2 issues:

1. I am not sure what possible unintended consequences this could have compared to using getLogger.
2. We still need a signature change in typeshed/stdlib/logging: there getChild is defined as

def getChild(self, suffix: str) -> Logger: ...

However, we need something along the lines of

T = TypeVar("T", bound=Logger)
def getChild(self: T, suffix: str) -> T: ...

Cf. python/mypy#1212

[Akuli]

The second workaround seems better to me. It seems unlikely that TypeVar semantics will be changed just because of this.

I am not sure what could be possible unintended consequences this may have compared to using getLogger.

The source code for getChild is:

        if self.root is not self:
            suffix = '.'.join((self.name, suffix))
        return self.manager.getLogger(suffix)

So getLogger("foo.bar").getChild("baz") just calls getLogger("foo.bar.baz"), and there doesn't seem to be any "unintended consequences" or gotchas to be aware of.

As you said, we still need to change the definition of getChild to use a TypeVar. The runtime doesn't do anything to make child loggers to have the same type as their parent, but writing the stubs that way still seems to be the most practical type-safe way to support using custom logger classes.

[randolf-scholz]

I opened an issue in typeshed: python/typeshed#6606

[rushilsrivastava]

Is there a proper workaround for this today? I'm actually not sure how applicable Workaround 2 is for custom Logger classes

Edit: I just the linked blog post. I am certainly not keen on adding a partial typed for overriding parameters from the logging subclass

[Akuli]

I think this is the best way today...

foo.py:

import logging
from typing import TYPE_CHECKING, cast

VERBOSE = 15

class LoggerWithVerbose(logging.Logger):
    # Use the same type for verbose() as for info().
    if TYPE_CHECKING:
        verbose = logging.Logger.info
    else:
        def verbose(self, msg, *args, **kwargs):
            if self.isEnabledFor(VERBOSE):
                self._log(VERBOSE, msg, args, **kwargs)

logging.basicConfig(level=VERBOSE)
logging.addLevelName(VERBOSE, "VERBOSE")
logging.setLoggerClass(LoggerWithVerbose)
root_logger = cast(LoggerWithVerbose, logging.getLogger())

bar.py:

from foo import root_logger

logger = root_logger.getChild(__name__)
logger.verbose("hello")

If I now run python3 -c 'import bar', I get VERBOSE:bar:hello. Note that python3 bar.py runs bar.py with __name__ set to "__main__", and will instead print a confusing VERBOSE:__main__:hello.