Python3, Twisted and Asyncio

June 24, 2018

Update: some twitterings on the subject.

After some discussions and hints on #twisted from runciter (Mark Williams) along with a Gist he’d produced, I finally figured out how Twisted’s asyncioreactor can be used to inter-operate with asyncio libraries.

This prompted me to create an example for txtorcon called web_onion_service_aiohttp.py

Let’s take a look!

Python3 and Twisted

First of all, it can still be surprising to people that Python3’s new async def and await syntax-sugar works with Twisted. The secret-sauce here is ensureDefered which turns a “coroutine” (i.e. any async def function) into a Deferred-returning function. You may also await a Deferred in such a function.

Thus, if you call ensureDeferred at the “top level” then you can just use await and async def for everything. This is great; even though it’s “just” a normal generator under the hood, async def lets you easily and definitively mark functions as “async” or not, which is a nice step forward for async in Python.

By “top level”, I mean using a react -style pattern like this:

from twisted.internet.task import react
from twisted.internet.defer import ensureDeferred
# our "real" main
async def _main(reactor):
    await some_deferred_returning_function()
# a wrapper that calls ensureDeferred
def main():
    return react(
        lambda reactor: ensureDeferred(
            _main(reactor)
        )
    )
if __name__ == '__main__':
    main()

Using a pattern like this, the rest of your code can return Deferred, use @inlineCallbacks or async def – so you can even “slowly” upgrade to Python3 style source.

asyncioreactor

The next piece of the puzzle is using asyncio libraries directly with Twisted. The way to do this currently is by installing the asyncioreactor – which implements the Twisted reactor interfaces using asyncio. (Another way would be to implement the asyncio EventLoop API using Twisted methods, but nobody has done that).

When installing custom reactors, you should do it “very early” (because if any code you import does import twisted.internet it’ll install a default reactor).

import asyncio
from twisted.internet import asyncioreactor
asyncioreactor.install(asyncio.get_event_loop())

So now Twisted will use a reactor that’s implemented using asyncio event-loops (above using the “global” event-loop), so now we can also use any asyncio library, too! (It’ll have to use the same event-loop, but most asyncio libraries just use the global one it seems).

Putting Them Together

The complete example I implemented in txtorcon is to create an Onion service using txtorcon, and then run the aiohttp web server to actually serve requests on that Onion service. This isn’t limited to server-style libraries, obviously; any mix of asyncio and Twisted can be used together.

There’s one catch, though: you basically have to know what kind of function you’re in (either asyncio or Twisted) and convert any of the “other” framework’s replies to yours. That is, if you’re “in” an asyncio function, you have to convert Deferred to Futures and if you’re “in” a Twisted function you have to convert Future instances to Deferred.

Twisted provides methods to do just that. In the example, I made a couple wrappers:

def as_future(d):
    return d.asFuture(asyncio.get_event_loop())
def as_deferred(f):
    return Deferred.fromFuture(asyncio.ensure_future(f))

Exercise: I wonder if it might be possible to create a decorator that “marks” a function as “twisted” or “asyncio” and automagically converts Deferreds or Futures as required?

Conclusion

You can read through the complete example code, which is fairly short. This is obviously a simple example, but opens some really cool and interesting possibilities for further interoperation between Twisted and asyncio libraries.

For completeness, here’s the entire example:

# This launches Tor and starts an Onion service using Twisted and
# txtorcon, and then starts a Web server using the aiohttp library.
#
# This style of interop between asyncio and Twisted requires twisted
# to use the "asyncioreactor" and for code to convert Futures/Tasks to
# Deferreds (most of which is already in Deferred)
#
# Thanks to Mark Williams for the inspiration, and this code:
# https://gist.github.com/markrwilliams/bffb9c293194d105169ea06f03484ba1
#
# note: if run in Python2, there are SyntaxErrors before we can tell
# the user nicely
import os
import asyncio
from twisted.internet import asyncioreactor
# get our reactor installed as early as possible, in case other
# imports decide to import a reactor and we get the default
asyncioreactor.install(asyncio.get_event_loop())
from twisted.internet.task import react
from twisted.internet.defer import ensureDeferred, Deferred
from twisted.internet.endpoints import UNIXClientEndpoint
import txtorcon
try:
    import aiohttp
    from aiohttp import web
    from aiosocks.connector import ProxyConnector, ProxyClientRequest
except ImportError:
    raise Exception(
        "You need aiohttp to run this example:\n  pip install aiohttp"
    )
def as_future(d):
    return d.asFuture(asyncio.get_event_loop())
def as_deferred(f):
    return Deferred.fromFuture(asyncio.ensure_future(f))
def get_slash(request):
    return web.Response(
        text="I am an aiohttp Onion service\n",
    )
def create_aio_application():
    app = web.Application()
    app.add_routes([
        web.get('/', get_slash)
    ])
    return app
async def _main(reactor):
    if False:
        print("launching tor")
        tor = await txtorcon.launch(reactor, progress_updates=print)
    else:
        tor = await txtorcon.connect(reactor,
            UNIXClientEndpoint(reactor, "/var/run/tor/control")
        )
    config = await tor.get_config()
    print("Connected to tor {}".format(tor.version))
    # here, we've just chosen 1234 as the port. We have three other
    # options:
    # - select a random, unused one ourselves
    # - put "ports=[80]" below, and find out which port txtorcon
    #   selected after
    # - use a Unix-domain socket
    #
    # we create a Tor onion service on a specific local TCP
    # port. This will launch a new Tor instance if required.
    print("Creating onion service")
    onion = await tor.create_onion_service(
        ports=[
            (80, 1234)  # 80 is the 'public' port, 1234 is local
        ],
        version=3,
        progress=print,
    )
    # we're now listening on some onion service URL and re-directing
    # public port 80 requests to local TCP port 1234.
    app = create_aio_application()
    runner = web.AppRunner(app)
    await as_deferred(runner.setup())
    site = web.TCPSite(runner, 'localhost', 1234)
    await as_deferred(site.start())
    # now we're completely set up
    print("Onion site on http://{}".format(onion.hostname))
    await Deferred()
def main():
    return react(
        lambda reactor: ensureDeferred(
            _main(reactor)
        )
    )
if __name__ == '__main__':
    main()