End-to-End Encrypted RPC/PubSub over Tor

Using Crossbar/Autobahn with Tor and Encryption

The Tor Project provides an overlay network that hides the network location of TCP streams. Both clients and servers are supported. Crossbar.io provides routed Remote Procedure Calls (RPC) and Publish/Subscribe (PubSub) messaging (called Web Application Messaging Protocol or WAMP) via many transports, including WebSockets. The project has recently added native support for Tor (via txtorcon).

Overview

The txtorcon documentation (clearnet) should be re-built whenever GitHub changes happen. GitHub provides a WebHook service which can notify you whenever interesting changes happen. These notifications take the form of an HTTP POST.

Ordinarily, accepting such an HTTP request and re-building the documentation would be straightforward. However, an Onion service is trying to hide its network location and giving GitHub some way to call this directly would break that. So, we will run an agent on a public IP that listens for GitHub WebHook requests and forwards them as “publish” events over Tor to the Crossbar.io router.

There will be another agent connected to the router that will subscribe to these forwarded GitHub events. This agent re-builds the documentation that is served by the txtorcon Onion service.

Diagram of the setup

There are three parts in the above diagram that we will deal with here:

All of the code for this is on GitHub as txtorcon-documentation-builder with some excerpts from it appearing in this post.

In this post and the included code above, we set up a location-hidden Crossbar.io instance listening on a Tor onion service with a publicly-facing component that verifies and forwards GitHub WebHook requests. These requests are acted on by a second “builder” component that re-builds the txtorcon documentation. This keeps the network-location of the documentation host and builder unknown. In a bonus section, we detail how to also end-to-end encrypt the messages exchanged between the WebHook listener and the builder components.

The Crossbar Router

Crossbar.io is a Twisted application that provides a robust implementation of a “WAMP router”, routing RPC and PubSub messages between various connected clients. It supports many transports (stdin/out, Unix sockets, TCP sockets and WebSockets). For our purposes we will use WebSockets over Tor, which Crossbar natively supports.

This is configured using the onion transport type, which requires a “control” connection to the Tor daemon. This is so we can use the ADD_ONION command to install a new Onion service. This will allocate a random local-only TCP port to which the resulting traffic from Tor will be sent.

Crossbar also supports encrypting the actual RPC or PubSub payloads. Although the Tor traffic is already end-to-end encrypted, this only gets data from one client to the router. The “cryptobox” feature of Crossbar allows us to encrypt the message payloads for other clients. This uses NaCl “Box” operations. After securely exchanging two clients’ public keys, these two clients of a Crossbar router can exchange RPC and PubSub messages without the Crossbar process being able to see any of the payload. Of course, the router can still analyze any metadata, such as: how often the RPC is called; the size of the payload; between which clients the communication is happening; etcetera.

Nonetheless, being able to hide the actual payload is a nice feature.

To authenticate the clients to the router, we will use Crossbar’s “WAMP Cryptosign” authentication, which uses the NaCl “crypto_sign” algorithm to make a Curve 25519 signature of a challenge from Crossbar.

Setting Up Crossbar

After installing Crossbar via one of the methods in Crossbar’s installation guide we need to provide some relevant configuration.

First of all is our Onion service, with a stanza similar to the following:

"workers": [
    {
        "type": "router",
        "transports": [
            {
                "type": "websocket",
                "endpoint": {
                    "type": "onion",
                    "port": 8080,
                    "private_key_file": "hskey",
                    "tor_control_endpoint": {
                        "type": "unix",
                        "path": "/var/run/tor/control"
                    }
                }
            }
        ]
    }
]

The above will result in a .crossbar/hskey file being created with the private-key of the new service; on a subsequent start-up of this same configuration the same Onion service will again be launched. You can also provide an absolute path to this file if you prefer (relative paths are relative to the crossbar directory). You will have to look in Crossbar’s log file for the name of the created service (or for any errors from Tor). It will be something like m6dazoly4sqnoqrm.onion. Note that anyone with the private key can create a service listening on this onion address, so you must keep the key secret.

Now we can tell our two client services (the GitHub WebHook endpoint and the actual documentation builder) to connect with the WebSocket URI ws://m6dazoly4sqnoqrm.onion:8080/. If the crossbar service is running on the same machine as the documentation-building service, we could configure a local Unix socket for it to connect on; for brevity I’m leaving that out of this post.

If all went well, running crossbar start should result in something like this:

2017-10-01T00:00:25-0600 [Router      26361] Realm 'agent' started
2017-10-01T00:00:25-0600 [Controller  26355] Router "worker-001": realm 'realm-001' (named 'agent') started
2017-10-01T00:00:25-0600 [Router      26361] role role-001 on realm realm-001 started
2017-10-01T00:00:25-0600 [Controller  26355] Router "worker-001": role 'role-001' (named 'github') started on realm 'realm-001'
2017-10-01T00:00:25-0600 [Router      26361] role role-002 on realm realm-001 started
2017-10-01T00:00:25-0600 [Controller  26355] Router "worker-001": role 'role-002' (named 'builder') started on realm 'realm-001'
2017-10-01T00:00:25-0600 [Router      26361] WampWebSocketServerFactory starting on '[redacted]'
2017-10-01T00:00:25-0600 [Controller  26355] Router "worker-001": transport 'transport-001' started
2017-10-01T00:00:25-0600 [Router      26361] WampWebSocketServerFactory starting on 60220
2017-10-01T00:00:25-0600 [Router      26361] Uploading descriptors can take more than 30s
2017-10-01T00:00:25-0600 [Router      26361] Created hidden-service at [redacted]
2017-10-01T00:00:25-0600 [Router      26361] Created '[redacted]', waiting for descriptor uploads.
2017-10-01T00:00:59-0600 [Router      26361] Uploaded '[redacted]' to '$[redacted]'
2017-10-01T00:00:59-0600 [Router      26361] Listening on Tor onion service [redacted] with local port 60220
2017-10-01T00:00:59-0600 [Controller  26355] Router "worker-001": transport 'transport-002' started
2017-10-01T00:00:59-0600 [Controller  26355] Local node configuration applied successfully!
2017-10-01T00:05:01-0600 [Router      26361] session "2099533749194636" joined realm "agent"
2017-10-01T00:05:33-0600 [Router      26361] session "5229690000667752" joined realm "agent"

Setting Up a Crossbar Client

On a machine with a public IP address, we will run the GitHub WebHook client. This will do two things:

  • listen on 443 for HTTPS connections from GitHub;
  • and connect to Crossbar via Tor turning any WebHook calls into “publish” events

The documentation builder will subscribe to these topics and trigger builds when the source code changes. (Note again that Crossbar has native support for this, but we don’t want Crossbar to be on a public IP address for this exercise).

We will use Klein as our Web server and txacme to get a (free!) Let’s Encrypt certificate for our server.

Using the “Component” API in Crossbar, configuration of our client will look something like this (in Python, using Twisted):

from autobahn.twisted.component import Component, run
from twisted.internet.endpoints import clientFromString
hook = Component(
    transports=[
        {
            "endpoint": clientFromString(reactor, u'tor:m6dazoly4sqnoqrm.onion:5000'),
            "url": u"ws://m6dazoly4sqnoqrm.onion:5000/",
        }
    ],
    realm=u"agent",
    authentication={
        u"cryptosign": {
            u"authid": u"agent",
            u"authrole": u"github",
            u"privkey": u"[redacted]",
        }
    }
)
if __name__ == '__main__':
    run([hook])

One thing to note is that the “endpoint” configuration is an actual Twisted IStreamClientEndpoint instance, and the tor: prefix is added by txtorcon.

Receiving the WebHooks

This agent also needs to listen for the GitHub POST requests, for which we will use Klein, a modern Twisted Web development framework (sometimes likened to Flask).

A WAMP session goes through a predictable lifecycle of four possible events: connect, join, leave, disconnect (in that order). The connect and disconnect are self-explantory transport-level events. The join event fires after we’ve connected to a router and succcessfully authenticated. You may add a callback for any event by calling component_instance.on('event_name', func) or there are convenience decorators, which we will use:

from twisted.web.server import Site
import klein
# from above, 'hook' is a autobahn.twisted.component.Component instance
@hook.on_join
@defer.inlineCallbacks
def join(session, details):
    app = klein.app.Klein()
    @app.route('/webhook/github', methods=['POST'])
    def github_webhook(request):
        pass  # details removed; see repository
    site = Site(app.resource())
    ep = serverFromString(reactor, 'le:/tmp/certs:tcp:443')
    yield ep.listen(site)

This gives us a site listening on port 443 using Let’s Encrypt certificates thanks to txacme (which will also renew them). In this instance, they’ll be stored in /tmp/certs. The “see details” code includes checking the signatures from GitHub, so we can be assured these are legitimate requests.

Setting Up Github

Once the above agent is in place on a public machine (and listening on 443) we can configure GitHub to send us POSTs. Very briefly (as there are many good resources to help with this part), we ask for TLS-verified POST requests to our public machine on the URI we’re listening on.

Find this configuration in the “settings” tab of your repository:

How to find GitHub settings

Point it at the machine where the agent is running. This is a machine with a public IP address.

GitHub WebHook endpoint settings

The Documentation Builder

So now we have: a Crossbar.io router on a private machine and a WebHook agent on a public machine. The Crossbar.io machine is connected only via Tor, with no incoming connections allowed (all Tor connections are outbound). (Updated: very briefly, your tor client makes an outbound circuit to one of the “rendezvous points” to which the service-providing tor has previously made its own outbound circuits; the rendezvous point glues them together. There will be a listen() call happening on localhost:random_port or a Unix socket but no incoming TCP connections from the Internet.)

The next step is to add the agent that actually re-builds the documentation on the machine serving the txtorcon documentation. This machine also is connected to the outside world only via Tor. This agent will connect to Crossbar.io via Tor and listen for “publish” events – that is, the GitHub WebHook announcements that are now being turned into WAMP “publish” messages.

For the full details, see the repository. The important parts are how we connect to the Tor onion service.

builder = Component(
    transports=[
        {
            "endpoint": clientFromString(reactor, u'tor:m6dazoly4sqnoqrm.onion:5000'),
            "url": u"ws://m6dazoly4sqnoqrm.onion:5000/",
        }
    ],
    realm=u"builder",
)
@builder.subscribe(u'webhook.github.push')
@defer.inlineCallbacks
def _github_push(**kw):
    if kw['ref'] != 'refs/heads/master':
        return
    # set PATH to our builder's venv
    # '_run' is a helper that spawns a process
    # 'disthome' points to our checkout
    yield _run(disthome, '/usr/bin/git', 'pull')
    yield _run(join(disthome, 'docs'), '/usr/bin/make', 'html')

Conclusion

We have set up a Crossbar.io listening on a Tor Project onion service with a publicly-facing component that verifies and forwards GitHub WebHook requests. These requests are acted on by a second “builder” component that re-builds the txtorcon documentation. This keeps the network-location of the documentation host unknown.

Advanced Class: end-to-end Encryption

As an additional bonus, we can also hide the payload from the Crossbar.io router. This feature uses NaCl “Box” operations to wrap all arguments and keyword arguments. Since this uses asymmetric cryptography we have a public/private keypair in the Builder agent that accepts publishes on webhook.github.*; the corresponding public key is put on the GitHub agent so that it may encrypt messages to the Builder.

Note: this cryptobox feature is still under development in Crossbar.io and notably hasn’t been audited (Issue 916) and can only use JSON as the underlying serializer (Issue 915). Additionally, it only works with Python clients and currently lacks a JavsScript, Java or C++ implementation.

The diagram below is the same as the one above, but shows the encryption keys involved. You can use the end-to-end encryption feature with a single key-pair or with key-pairs for individual URIs in the WAMP URI space. Obviously, the latter involves (potentially) many more key-pairs to manage.

Note that the key setup is the same per-URI no matter if you’re doing RPC or PubSub – although PubSub seems like “broadcast only” and should therefore only have one keypair, the second keypair (shown with dotted shading below) is for any replies (e.g. an error). Of course, if this was RPC the reply would carry the (encrypted) response data.

Diagram of the setup

We also show the key-pairs used for “WAMP CryptoSign” authentication; these are also using NaCl but using the “crypto_sign” API from NaCL to sign a challenge from the router. Notice that Alice has one public + private keypair and so does Bob. Alice also has the public half of Bob’s pair, and Bob has Alice’s public half. Thus any message from Alice → Bob is encrypted to Bob’s public key and vice-versa for a message from Bob → Alice.

For the concrete use-case we’re using here, you can see the webhook setup and the corresponding builder setup. This means that the details of the actual contents of the published message (the args and kwargs) remain unknowable to the Crossbar.io router itself (the router still sees all the metadata, including the URI used).

When the Bad Stuff Happens

If our public (probably VPS-hosted) machine is compromised, an attacker will learn:

  • the onion address of our router (but not its clearnet location)
  • an authentication keypair
  • a public encryption key (for the publishes)
  • an encryption keypair (for any replies to the publishes)

Learning the onion address of our router isn’t catastrophic, but does then allow the attacker to make connections (over Tor only) because they have control of a valid authentication keypair. However, the only “publish” events that our Builder will accept will be for webhook.github.* – so our attacker can publish fake GitHub webhook events.

When the compromise is noticed, we can revoke the public half of the authentication keypair, disallowing further authentication – that is, the attacker can no longer connect to the WAMP router (via Tor). We can also remove the keypair from the Builder agent so it will no longer accept webhook.github.* events, disallowing the attacker from making fake GitHub webhook events.

After a compromise ideally we would also make a new Onion address – this might not always be desireable or possible, though; you may wish to have a long-lived .onion address (that other users have written down) and it may take a while to notify them of a new one. This isn’t a huge deal; the attacker still can’t authenticate nor encrypt any fake publish message, but they can continue to contact the router (via the Onion address they’ve learned, so only via Tor).

So, this is not completely fool-proof; an attacker can still feed us fake events until we notice the compromise – if the builder isn’t careful it might be tricked into publishing something we didn’t intend. However, the risk is pretty low – the Builder will be pulling from a public repo and is essentially using the WebHook “just” to trigger some pre-defined action. It can thus be very important to consider what kinds of events or RPC calls you respond to – if the Builder instead was listening for an arbitrary command to run (for example) our attacker could do a lot more damage (e.g. run “rm -rf /” or worse).

Additionally, structuring different realms with fine-grained permissions will help a lot here; as set up, the GitHub Agent can only publish to webhook.github.* – if instead it could publish to any WAMP URI at all it could interfere with other agents connected to this Crossbar.io router.

Advanced Class Conclusion

We have two different pieces of end-to-end encryption happening with this setup:

  • Onion service encryption to the Crossbar.io router
  • cryptobox encryption between the two agents

These work together to provide different layers of defense: the router can be assured it is receiving the exact bytes the client sent over Tor; the client can be assured it is contacting the correct computer (because of Onion service properties); and the Builder is assured it’s receiving events from the WebHook agent (and not some other client that’s managed to connect and authenticate to the correct realm).

Besides the above, we also have authentication: the Crossbar.io router only accepts connections from clients that control the correct cryptosign keypair.