ValueError with ContextVar reset within decorated coroutine

[majorgreys]

I was getting the following error while updating code for Python 3.7 and Tornado 6:

ValueError: <Token var=<ContextVar ...> at 0x1059e4fc0> was created in a different Context

I was able to reproduce this issue with the sample script below. The error only occurs when using decorate coroutines, whereas native coroutines work as expected.

import asyncio
import contextvars
from tornado import gen, ioloop

cv = contextvars.ContextVar("cv", default=False)

async def foo():
    await asyncio.sleep(1)

async def main_native():
    token = cv.set(True)
    print(f"main_native before foo: Context={contextvars.Context()}")
    await foo()
    print(f"main_native after foo: Context={contextvars.Context()}")
    cv.reset(token)
    print("main_native: OK")

@gen.coroutine
def main_decorated():
    print(f"main_decorated before foo: Context={contextvars.Context()}")
    token = cv.set(True)
    yield foo()
    print(f"main_decorated after foo: Context={contextvars.Context()}")
    cv.reset(token)
    print("main_decorated OK")

if __name__ == "__main__":
    ioloop.IOLoop.current().run_sync(main_native)
    ioloop.IOLoop.current().run_sync(main_decorated)

The output from Python 3.7 and Tornado 6.0.3 is:

main_native before foo: Context=<Context object at 0x10ee2b948>
main_native after foo: Context=<Context object at 0x10ee2b948>
main_native: OK
main_decorated before foo: Context=<Context object at 0x10ee2b7e0>
main_decorated after foo: Context=<Context object at 0x10ee2bcf0>
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/var/folders/9_/k78_39fn08991mz6r_ws9fc00000gn/T/babel-ZbUBrS/python-QPUJC9", line 29, in <module>
    ioloop.IOLoop.current().run_sync(main_decorated)
  File "/Users/tahir.butt/.virtualenvs/py37-tornado6/lib/python3.7/site-packages/tornado/ioloop.py", line 532, in run_sync
    return future_cell[0].result()
  File "/Users/tahir.butt/.virtualenvs/py37-tornado6/lib/python3.7/site-packages/tornado/gen.py", line 748, in run
    yielded = self.gen.send(value)
  File "/var/folders/9_/k78_39fn08991mz6r_ws9fc00000gn/T/babel-ZbUBrS/python-QPUJC9", line 24, in main_decorated
    cv.reset(token)
ValueError: <Token var=<ContextVar name='cv' default=False at 0x10ee1c570> at 0x10ee2bbd0> was created in a different Context

Whereas the value of contextvars.Context() is not changed within the native coroutine main_native when calling the foo coroutine, this is not true for within the decorated courtine main_decorated.

We see here where this error originates in the cPython implementation for ContextVar.reset():

https://github.com/python/cpython/blob/aacc77fbd77640a8f03638216fa09372cc21673d/Python/context.c#L295-L300

This issue reflects the fundamental problem addressed in #2716. Using that PR branch resolves the problem raised in this issue.

[bdarnell]

Sorry for the silence here; there's been some discussion of contextvars issues on #2716. I have reproduced the problem here and am working on a fix.