End-to-End Encrypted RPC/PubSub over Tor

Overview

The txtorcon documentation (clearnet) should be re-built whenever GitHub changes happen. GitHub provides a WebHook service which can notify you whenever interesting changes happen. These notifications take the form of an HTTP POST.

Ordinarily, accepting such an HTTP request and re-building the documentation would be straightforward. However, an Onion service is trying to hide its network location and giving GitHub some way to call this directly would break that. So, we will run an agent on a public IP that listens for GitHub WebHook requests and forwards them as “publish” events over Tor to the Crossbar.io router.

There will be another agent connected to the router that will subscribe to these forwarded GitHub events. This agent re-builds the documentation that is served by the txtorcon Onion service.

There are three parts in the above diagram that we will deal with here:

• configure the Crossbar.io router to work over Tor
• write the GitHub Agent (in Python)
• write the Doc Builder (in Python)

All of the code for this is on GitHub as txtorcon-documentation-builder with some excerpts from it appearing in this post.

In this post and the included code above, we set up a location-hidden Crossbar.io instance listening on a Tor onion service with a publicly-facing component that verifies and forwards GitHub WebHook requests. These requests are acted on by a second “builder” component that re-builds the txtorcon documentation. This keeps the network-location of the documentation host and builder unknown. In a bonus section, we detail how to also end-to-end encrypt the messages exchanged between the WebHook listener and the builder components.

The Crossbar Router

Crossbar.io is a Twisted application that provides a robust implementation of a “WAMP router”, routing RPC and PubSub messages between various connected clients. It supports many transports (stdin/out, Unix sockets, TCP sockets and WebSockets). For our purposes we will use WebSockets over Tor, which Crossbar natively supports.

This is configured using the onion transport type, which requires a “control” connection to the Tor daemon. This is so we can use the ADD_ONION command to install a new Onion service. This will allocate a random local-only TCP port to which the resulting traffic from Tor will be sent.

Crossbar also supports encrypting the actual RPC or PubSub payloads. Although the Tor traffic is already end-to-end encrypted, this only gets data from one client to the router. The “cryptobox” feature of Crossbar allows us to encrypt the message payloads for other clients. This uses NaCl “Box” operations. After securely exchanging two clients’ public keys, these two clients of a Crossbar router can exchange RPC and PubSub messages without the Crossbar process being able to see any of the payload. Of course, the router can still analyze any metadata, such as: how often the RPC is called; the size of the payload; between which clients the communication is happening; etcetera.

Nonetheless, being able to hide the actual payload is a nice feature.

To authenticate the clients to the router, we will use Crossbar’s “WAMP Cryptosign” authentication, which uses the NaCl “crypto_sign” algorithm to make a Curve 25519 signature of a challenge from Crossbar.

"workers": [
    {
        "type": "router",
        "transports": [
            {
                "type": "websocket",
                "endpoint": {
                    "type": "onion",
                    "port": 8080,
                    "private_key_file": "hskey",
                    "tor_control_endpoint": {
                        "type": "unix",
                        "path": "/var/run/tor/control"
                    }
                }
            }
        ]
    }
]

2017-10-01T00:00:25-0600 [Router      26361] Realm 'agent' started
2017-10-01T00:00:25-0600 [Controller  26355] Router "worker-001": realm 'realm-001' (named 'agent') started
2017-10-01T00:00:25-0600 [Router      26361] role role-001 on realm realm-001 started
2017-10-01T00:00:25-0600 [Controller  26355] Router "worker-001": role 'role-001' (named 'github') started on realm 'realm-001'
2017-10-01T00:00:25-0600 [Router      26361] role role-002 on realm realm-001 started
2017-10-01T00:00:25-0600 [Controller  26355] Router "worker-001": role 'role-002' (named 'builder') started on realm 'realm-001'
2017-10-01T00:00:25-0600 [Router      26361] WampWebSocketServerFactory starting on '[redacted]'
2017-10-01T00:00:25-0600 [Controller  26355] Router "worker-001": transport 'transport-001' started
2017-10-01T00:00:25-0600 [Router      26361] WampWebSocketServerFactory starting on 60220
2017-10-01T00:00:25-0600 [Router      26361] Uploading descriptors can take more than 30s
2017-10-01T00:00:25-0600 [Router      26361] Created hidden-service at [redacted]
2017-10-01T00:00:25-0600 [Router      26361] Created '[redacted]', waiting for descriptor uploads.
2017-10-01T00:00:59-0600 [Router      26361] Uploaded '[redacted]' to '$[redacted]'
2017-10-01T00:00:59-0600 [Router      26361] Listening on Tor onion service [redacted] with local port 60220
2017-10-01T00:00:59-0600 [Controller  26355] Router "worker-001": transport 'transport-002' started
2017-10-01T00:00:59-0600 [Controller  26355] Local node configuration applied successfully!
2017-10-01T00:05:01-0600 [Router      26361] session "2099533749194636" joined realm "agent"
2017-10-01T00:05:33-0600 [Router      26361] session "5229690000667752" joined realm "agent"

from autobahn.twisted.component import Component, run
from twisted.internet.endpoints import clientFromString
hook = Component(
    transports=[
        {
            "endpoint": clientFromString(reactor, u'tor:m6dazoly4sqnoqrm.onion:5000'),
            "url": u"ws://m6dazoly4sqnoqrm.onion:5000/",
        }
    ],
    realm=u"agent",
    authentication={
        u"cryptosign": {
            u"authid": u"agent",
            u"authrole": u"github",
            u"privkey": u"[redacted]",
        }
    }
)
if __name__ == '__main__':
    run([hook])

Receiving the WebHooks

This agent also needs to listen for the GitHub POST requests, for which we will use Klein, a modern Twisted Web development framework (sometimes likened to Flask).

A WAMP session goes through a predictable lifecycle of four possible events: connect, join, leave, disconnect (in that order). The connect and disconnect are self-explantory transport-level events. The join event fires after we’ve connected to a router and succcessfully authenticated. You may add a callback for any event by calling component_instance.on('event_name', func) or there are convenience decorators, which we will use:

from twisted.web.server import Site
import klein
# from above, 'hook' is a autobahn.twisted.component.Component instance
@hook.on_join
@defer.inlineCallbacks
def join(session, details):
    app = klein.app.Klein()
    @app.route('/webhook/github', methods=['POST'])
    def github_webhook(request):
        pass  # details removed; see repository
    site = Site(app.resource())
    ep = serverFromString(reactor, 'le:/tmp/certs:tcp:443')
    yield ep.listen(site)

This gives us a site listening on port 443 using Let’s Encrypt certificates thanks to txacme (which will also renew them). In this instance, they’ll be stored in /tmp/certs. The “see details” code includes checking the signatures from GitHub, so we can be assured these are legitimate requests.

Setting Up Github

Once the above agent is in place on a public machine (and listening on 443) we can configure GitHub to send us POSTs. Very briefly (as there are many good resources to help with this part), we ask for TLS-verified POST requests to our public machine on the URI we’re listening on.

Find this configuration in the “settings” tab of your repository:

Point it at the machine where the agent is running. This is a machine with a public IP address.

The Documentation Builder

So now we have: a Crossbar.io router on a private machine and a WebHook agent on a public machine. The Crossbar.io machine is connected only via Tor, with no incoming connections allowed (all Tor connections are outbound). (Updated: very briefly, your tor client makes an outbound circuit to one of the “rendezvous points” to which the service-providing tor has previously made its own outbound circuits; the rendezvous point glues them together. There will be a listen() call happening on localhost:random_port or a Unix socket but no incoming TCP connections from the Internet.)

The next step is to add the agent that actually re-builds the documentation on the machine serving the txtorcon documentation. This machine also is connected to the outside world only via Tor. This agent will connect to Crossbar.io via Tor and listen for “publish” events – that is, the GitHub WebHook announcements that are now being turned into WAMP “publish” messages.

For the full details, see the repository. The important parts are how we connect to the Tor onion service.

builder = Component(
    transports=[
        {
            "endpoint": clientFromString(reactor, u'tor:m6dazoly4sqnoqrm.onion:5000'),
            "url": u"ws://m6dazoly4sqnoqrm.onion:5000/",
        }
    ],
    realm=u"builder",
)
@builder.subscribe(u'webhook.github.push')
@defer.inlineCallbacks
def _github_push(**kw):
    if kw['ref'] != 'refs/heads/master':
        return
    # set PATH to our builder's venv
    # '_run' is a helper that spawns a process
    # 'disthome' points to our checkout
    yield _run(disthome, '/usr/bin/git', 'pull')
    yield _run(join(disthome, 'docs'), '/usr/bin/make', 'html')

Conclusion

We have set up a Crossbar.io listening on a Tor Project onion service with a publicly-facing component that verifies and forwards GitHub WebHook requests. These requests are acted on by a second “builder” component that re-builds the txtorcon documentation. This keeps the network-location of the documentation host unknown.

Advanced Class: end-to-end Encryption

As an additional bonus, we can also hide the payload from the Crossbar.io router. This feature uses NaCl “Box” operations to wrap all arguments and keyword arguments. Since this uses asymmetric cryptography we have a public/private keypair in the Builder agent that accepts publishes on webhook.github.*; the corresponding public key is put on the GitHub agent so that it may encrypt messages to the Builder.

Note: this cryptobox feature is still under development in Crossbar.io and notably hasn’t been audited (Issue 916) and can only use JSON as the underlying serializer (Issue 915). Additionally, it only works with Python clients and currently lacks a JavsScript, Java or C++ implementation.

The diagram below is the same as the one above, but shows the encryption keys involved. You can use the end-to-end encryption feature with a single key-pair or with key-pairs for individual URIs in the WAMP URI space. Obviously, the latter involves (potentially) many more key-pairs to manage.

Note that the key setup is the same per-URI no matter if you’re doing RPC or PubSub – although PubSub seems like “broadcast only” and should therefore only have one keypair, the second keypair (shown with dotted shading below) is for any replies (e.g. an error). Of course, if this was RPC the reply would carry the (encrypted) response data.

We also show the key-pairs used for “WAMP CryptoSign” authentication; these are also using NaCl but using the “crypto_sign” API from NaCL to sign a challenge from the router. Notice that Alice has one public + private keypair and so does Bob. Alice also has the public half of Bob’s pair, and Bob has Alice’s public half. Thus any message from Alice → Bob is encrypted to Bob’s public key and vice-versa for a message from Bob → Alice.

For the concrete use-case we’re using here, you can see the webhook setup and the corresponding builder setup. This means that the details of the actual contents of the published message (the args and kwargs) remain unknowable to the Crossbar.io router itself (the router still sees all the metadata, including the URI used).