Modernize python code for construct and add types (#609)

* construct: Fix doctests

Python 3 is working on `bytes`.

Use raw-strings to allow backslash escaped bytes.

Signed-off-by: Philipp Hahn <[email protected]>

* construct: Fix String documentation

`paddir=both` -> `center`; see
[PaddedStringAdapter](elftools/construct/adapters.py#L186).

Signed-off-by: Philipp Hahn <[email protected]>

* construct: Fix Switch documentation

Signed-off-by: Philipp Hahn <[email protected]>

* construct: Fix BitStreamWriter bytes

Since Python 3 streams work on `bytes`, not (Unicode) `str`ings.

Signed-off-by: Philipp Hahn <[email protected]>

* construct: Implement IO for BitStreams

`BitStreamReader` and `BitStreamWriter` are supposed to be used instead
of `IO[bytes]`, but actually implement a different signature: that would
force all functions currently accepting `IO[bytes]` to be changed to
`IO[bytes] | BitStreamReader` respective `IO[bytes] | BitStreamWriter`.

Instead let both inherit from `io.RawIOBase` to inherit the complete API
and implement that interface by returning the correct values and types.

See <https://docs.python.org/3/library/io.html#io.RawIOBase>.

Signed-off-by: Philipp Hahn <[email protected]>^

* construct: Use named arguments instead of kwargs

Explicitly name the named parameters in the function signature instead
of using `**kwargs`, which is a pain to type annotate.
This also allows us to get rid of the manual check for extra unknown
arguments, as the compiler will do that for us.

Signed-off-by: Philipp Hahn <[email protected]>

* construct: Initialize variables out of loop

Type checkers complain about `pos` being undefined inside the exception
handling code. Actually `ConstructError` will only be raised by
`parse()` inside the loop, so both `c` and `pos` will be assigned at
least once, but all type-checkers fail to detect this.

Initialize both `c` and `pos` before the loop to silence them.

Signed-off-by: Philipp Hahn <[email protected]>

* construct: Export used symbols

`pyright` is very picky about `Final` symbols being imported multiple
times, even when they are the same.

Explicitly name all symbole to be exported.

Signed-off-by: Philipp Hahn <[email protected]>

* construct: typing: Add / fix PEP-484 hints

- [x] `pyright`
- [x] `mypy`

Signed-off-by: Philipp Hahn <[email protected]>

* construct: Assert correct type

Signed-off-by: Philipp Hahn <[email protected]>

* construct: Assert PrefixedArray.length_field.name

Assert name of `PrefixedArray.length_field.name` is `not None`.

Signed-off-by: Philipp Hahn <[email protected]>

* construct: Fix ExprAdapter encoder/decoder

`ExprAdapter` expects two *functions* as encode/decoder and overwrites
its *methods*: Are those two functions called with `self` as the 1st
argument?

Make it clear by implementing 2 methods and calling those two functions
as functions.

Signed-off-by: Philipp Hahn <[email protected]>

* construct: Return str from StringAdapter

Directly return `str` after `decode()` as `obj` is declared as `bytes`.

Signed-off-by: Philipp Hahn <[email protected]>

* construct: Return None for Peek

`Peek._parse()` must `return None` explicitly for `mypy`.

Signed-off-by: Philipp Hahn <[email protected]>

* construct: typing: Hint static attribute name

We know that `name` is a `str`, so tell it `mypy` too.

Signed-off-by: Philipp Hahn <[email protected]>

* construct: typing: singletons

`NoDefault`, `Pass` and `Terminator` are all singletons, which are used
in various dictionaries. For type hinting the *type* is also required in
addition to the singleton *instance*.

Prefix the class with an underscore.

Hint instances with `Final[Any]` to prevent re-assignment and to exempt
them from type checking: Otherwise all code de-referencing something
from `dict[str, … | _Pass]` has to explicitly check for `_Pass` all
time, which currently it does not. Actually that is a bug, which needs
fixing at a later day.

Signed-off-by: Philipp Hahn <[email protected]>

* construct: Dispose LazyContainer

Delete instance variables instead of assigning `None`, which then
requires checks for `not None`.

Signed-off-by: Philipp Hahn <[email protected]>

* construct: typing ignore: LazyContainer.__eq__

Code explicitly checks for `AttributeError`, but `mypy` is picky here.

Signed-off-by: Philipp Hahn <[email protected]>

* construct: typing: Container

`Container` is a static typing nightmare as it acts as a `dict[str,
Any]`. In addition to that instances are also assigned additional
variables using arbitrary types. Basically `Any` goes in, `Any` comes
out with is not useful for typing.

As we know the types used by `pyelftools`, we can give `mypy` et.al.
some hints on those use cases, but have to put theme here on
`Container` as that is the base for everything.

Yikes.

Signed-off-by: Philipp Hahn <[email protected]>

---------

Signed-off-by: Philipp Hahn <[email protected]>
Signed-off-by: Philipp Hahn <[email protected]>^

Author: pmhahn

elftools/construct/__init__.py

@@ -12,24 +12,24 @@
     http://construct.wikispaces.com (including online tutorial)
 
 Typical usage:
-    >>> from construct import *
+    >>> from ..construct import *
 
 Hands-on example:
-    >>> from construct import *
+    >>> from ..construct import *
     >>> s = Struct("foo",
     ...     UBInt8("a"),
     ...     UBInt16("b"),
     ... )
-    >>> s.parse("\\x01\\x02\\x03")
-    Container(a = 1, b = 515)
-    >>> print(s.parse("\\x01\\x02\\x03"))
-    Container:
-        a = 1
-        b = 515
-    >>> s.build(Container(a = 1, b = 0x0203))
-    "\\x01\\x02\\x03"
+    >>> s.parse(b"\\x01\\x02\\x03")
+    Container({'a': 1, 'b': 515})
+    >>> print(s.parse(b"\\x01\\x02\\x03"))
+    Container({'a': 1, 'b': 515})
+    >>> s.build(Container(a=1, b=0x0203))
+    b'\\x01\\x02\\x03'
 """
+from __future__ import annotations
 
+from .lib.container import *
 from .core import *
 from .adapters import *
 from .macros import *
@@ -39,9 +39,9 @@
 #===============================================================================
 # Metadata
 #===============================================================================
-__author__ = "tomer filiba (tomerfiliba [at] gmail.com)"
-__maintainer__ = "Corbin Simpson <[email protected]>"
-__version__ = "2.06"
+__author__: str = "tomer filiba (tomerfiliba [at] gmail.com)"
+__maintainer__: str = "Corbin Simpson <[email protected]>"
+__version__: str = "2.06"
 
 #===============================================================================
 # Shorthand expressions
@@ -59,10 +59,20 @@
 #===============================================================================
 import functools
 import warnings
+from typing import TYPE_CHECKING, TypeVar
 
-def deprecated(f):
+if TYPE_CHECKING:
+    from collections.abc import Callable
+
+    from typing_extensions import ParamSpec
+
+    _P = ParamSpec('_P')
+    _T = TypeVar('_T')
+
+
+def deprecated(f: Callable[_P, _T]) -> Callable[_P, _T]:
     @functools.wraps(f)
-    def wrapper(*args, **kwargs):
+    def wrapper(*args: _P.args, **kwargs: _P.kwargs) -> _T:
         warnings.warn(
             "This name is deprecated, use %s instead" % f.__name__,
             DeprecationWarning, stacklevel=2)