Teserakt partners with P3KI

We’re happy to join forces with P3KI GmbH, provider of the P3KI Core decentralized PKI solution for access delegation applications. We like what P3KI is doing because we think that it’s useful, well-engineered, and developed with security in mind.

The P3KI Core product can be seen as a kind of PKI but with more decentralization, and with precise expression of permission levels thanks to a mathematically-verified language, which adresses most authorization and authentication challenges.

Concretely, this partnership consists in bundling the P3KI’s authorization management and Teserakt’s key management services in a single appliance, simplifying and reducing the cost of integration of these two complementary products. More details are available in our slide deck.

For any questions, please get in touch at [email protected]. See also P3KI’s announcement.

Transport Security and LwM2M

Network World recently published an article on the differences between the MQTT and LwM2M protocols for businesses considering which to adopt. While there are a number of interesting differences between the standards, our main interest is of course security. In particular we are interested in end-to-end encryption.

This term has become popular with the rise of secure messaging software such as Signal and Wire, and products that have adopted the Double Ratchet protocol such as WhatsApp and Facebook Messenger. In these contexts, end-to-end means from person to person: sender to their intended recipient(s). In our machine-to-machine world we mean exactly the same thing: when an IoT device communicates with other IoT devices, end-to-end for us means that any brokers, proxy servers or other intermediaries cannot decrypt or alter the messages.

With this definition in mind, we can take a closer look at the article and specifications. On the topic of what is provided in the existing standards, the article has this to say:

Modern revisions of the MQTT specification support features such as end-to-end encryption using either transport layer security (TLS) for the entire channel or payload encryption… For message security, CoAP and LWM2M typically utilize datagram transport layer security (DTLS) for data payload encryption. Unlike MQTT and TLS, CoAP messages encrypt the payload of the message rather than the entire message transport.

Indeed, the MQTT specification does suggest the use of TLS for transport encryption. It does not provide encryption for payloads or application messages (unless the transport itself is encrypted), although it indicates that users may wish to implement this themselves:

TLS [RFC5246] can provide encryption of data sent over the network…. An application might independently encrypt the contents of its Application Messages.

So on the subject of end-to-end encryption, MQTT leaves any implementation up to vendors.

LwM2M is a set of several standards comprising several parts—at the time of writing version 1.1 is the most recent edition. In the transport standard document, the section on session security begins as follows:

CoAP over UDP [CoAP] is secured using the Datagram Transport Layer Security (DTLS) 1.2 protocol [RFC6347], as defined in Section 9 of [CoAP]

So, LwM2M uses a variant of TLS called Datagram Transport Layer Security, or DTLS, suitable for use with UDP and other streaming transports. Like TLS, this protocol operates a handshake and establishes trust based on X.509 PKI Infrastructure. CoAP messages are then encrypted as DTLS application data, or in TLS parlance, records. This differs from MQTT only in that DTLS is used; DTLS is used because unlike MQTT, LwM2M supports UDP and DTLS is a suitable protocol for this transport type. All in all, this is not controversial.

As at the broker in MQTT, TLS is removed in LwM2M whenever a proxying device or change of transport protocol is required (e.g. HTTP to SMS). LwM2M is interesting because it also provides a protocol called OSCORE. Quoting again from the transport security section of the transport specification:

Object Security for Constrained RESTful Environments [OSCORE] is an application layer security protocol protecting CoAP message exchanges. OSCORE is applicable to protocol messages which can be mapped to CoAP or a subset of CoAP, including HTTP and LwM2M. The use of OSCORE with LwM2M is optional. It MAY be used for protecting the communication between a LwM2M Client and the LwM2M Bootstrap-Server. It MAY be used between a LwM2M Client and a LwM2M Server.

In this context, OSCORE allows for sender and recipient IDs independently of their role in the network. However, it seems from the Core LwM2M specification that the use of OSCORE in LwM2M is solely for client-server communications. In defining object types, the documentation says:

This LwM2M Object provides the keying material and related information of a LwM2M Client appropriate to access a specified LwM2M Server using OSCORE.

This may be because the purpose of LwM2M seems to be explicitly focused on the management of devices and is less focused on messaging between devices. If a device were to communicate information to an LwM2M server that then needed to be re-transmitted to another device and the server did not “need-to-know”, we would not consider this end-to-end cryptography. On the other hand, where the goal is to communicate between LwM2M server and client, OSCORE does indeed meet our definition of end-to-end—any proxy on the path would not need (or be able) to remove the cryptography, guaranteeing confidentiality on the message path. The OSCORE specification explains this succinctly:

In scenarios with intermediary nodes such as proxies or gateways, transport layer security such as (D)TLS only protects data hop-by- hop. As a consequence, the intermediary nodes can read and modify any information. The trust model where all intermediary nodes are considered trustworthy is problematic, not only from a privacy perspective, but also from a security perspective

LwM2M further specifies the cryptographic algorithm choices:

OSCORE is defined for use with an Authenticated Encryption with Additional Data (AEAD) algorithm and a HMAC-based Key Derivation Function (HKDF) [RFC5869]. OSCORE mandates the implementation of the AEAD algorithm AES-CCM-16-64-128 (also known as CCM*) as defined in [RFC8152], and specifies the use of HMAC with SHA-256 as default.

These are very interesting choices. In particular, as things stand, they offer fairly good security. CCM used alone is vulnerable to nonce-misuse, in particular on embedded devices where random number generation may not be ideal, so not addressing this also seems to be a shortcoming. There is also an open mention of “other algorithms” as necessary to match the “state of the art” but these are not specified. Overall, however, assuming implementations use these algorithms correctly and keys are appropriately generated and provisioned these choices are well regarded. Our own solution, E4, offers slightly stronger security guarantees as a minimum—we offer nonce-misuse resistance.

OSCORE relies on a master key with 128 bits of entropy to derive further encryption keys and initialization vectors using a hash-based key derivation function:

OSCORE depends on a pre-established random Master Secret (Section 12.3) used to derive encryption keys, and a construction for making (key, nonce) pairs unique (Appendix D.3).

This is also uncontroversial and is intended to allow multiple keys for different purposes to be generated from a single source of entropy. The specification leaves the question open as to whether a single master secret should be used or multiple should be allowed to enable “forward secrecy”—in case this is desired, the specification suggests “a key establishment protocol”, giving an example specification annex to OSCORE.

A major feature of modern secure messengers we wanted to replicate for E4 was secure group messaging, scalable to large numbers of members. E4 supports this natively. OSCORE appears to have an extension draft for group messaging. LwM2M calls group messaging Observe-Composite. It is unclear how LwM2M proposes OSCORE be used in this scenario.

We have not covered the scenarios in which you may want to use each of the protocols in question, but it should be clear from this brief summary that they do serve slightly different use cases. We think it is very encouraging that standards bodies are considering the need for end-to-end encryption solutions and the cryptographic choices in OSCORE are reasonable. We are in particular encouraged by the move away from TLS, which is not suitable for such environments. DTLS in particular sees less scrutiny than TLS, while both protocols are complicated to deploy and manage. (D)TLS also involves an expensive handshake setup stage per hop, which is expensive where bandwidth is at a premium and slows communications down unnecessarily.

One criticism we would offer is that LwM2M is a complicated standard spanning multiple documents for multiple purposes and some components of the end-to-end encryption are referenced from further standards. In our own product, we strive for simplicity as an explicit goal, keeping unknowns and optional choices to the absolute minimum while guaranteeing the highest cryptographic and software security possible.

You may have been wondering how Teserakt’s E4 compares to OSCORE and DTLS/TLS in terms of protocol integration. The answer is that we are protocol-agnostic. Our reference implementation works on top of MQTT, however, we are working on integrations with other transport protocols. E4 could be used for application layer security in LwM2M. We have a clear set of choices for key rollover. OSCORE itself is not tied to the architecture of LwM2M, but its use in LwM2M focuses on client-server. E4 does not assign roles to the sender or recipient and is suitable as a building block for any application, whether communicating between two endpoints or communicating as part of a group.

In summary, LwM2M is an interesting development. From a security perspective, the adoption of end-to-end cryptography is very encouraging, although we would slightly tweak the cryptographic choices. There appears to be work to do at the interface between LwM2M and OSCORE to support group messaging.

Express fuzzing of MQTT brokers

MQTT brokers seem to lend themselves well to fuzzing because they often implement their own versions of the MQTT protocol, which includes parsing of MQTT control packets. These packets often include length–value encodings, a typical source of bugs if not carefully implemented. For example, a PUBLISH packet’s control header encodes the topic length over two bytes; if the parser naively attempts to read as many bytes as specified regardless of the packet’s actual length, an out-of-bound memory access could occur—the same kind of bug that caused Heartbleed.

With the intuition that such bugs and other parsing bugs could exist, we set out to run some dumb fuzzing on MQTT brokers. We started by manually crafting malformed packets but quickly discovered mqtt_fuzz, a small project from F-Secure that does exactly what we wanted, namely blind fuzzing of common MQTT control packets.

We started fuzzing brokers written in C—more sensitive to memory corruption bugs—Mosquitto and Bevywise. The former is the most popular open-source broker, and unsurprisingly our straightforward methodology didn’t yield any result. But we got several Bevywise crashes after running the fuzzer for about 20 seconds, apparently caused by the above length–value pattern (thanks to zx2c4 for helping us reverse engineering the binaries). A debugger would for example show the following:

* thread #15, stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
    frame #0: 0x000000010b6e3bc0 libmqttbroker.so`on_recv_publish_func + 368
libmqttbroker.so`on_recv_publish_func:
->  0x10b6e3bc0 <+368>: movzbl (%r15,%rax), %edx
    0x10b6e3bc5 <+373>: movb   %dl, (%rbx,%rax)
    0x10b6e3bc8 <+376>: movslq %ecx, %rax
    0x10b6e3bcb <+379>: incl   %ecx
Target 0: (Broker) stopped.

We didn’t investigate further and can’t tell whether these specific bugs are exploitable for more than remotely crashing a broker. But being able to crash a remote broker by sending a string such as 8206000200e0bfadf3a081f3a0812b2f762fa9bdcb90f3a0819001cd2b2f762f8523cab000 is already a problem.

We quickly looked at some other brokers, found to be resilient to our basic fuzzer: In EMQ and VerneMQ, both written in Erlang, certain malformed packets would crash session processes (by design), without interrupting the service. In HiveMQ, written in Java, we spotted an uncaught index-out-of-bound exception, which did not interrupt the service (HiveMQ directly fixed and credited us for the observation). In Mosca, written in JavaScript, we didn’t notice any abnormal behavior.

The largest part of this project was done in a Geneva–Lausanne train ride (~40min), so there’s probably much more to find, starting with using a less dumb fuzzer (afl, libFuzzer?) and looking at other brokers. Feel free to let us know your findings!

 

Introduction to MQTT

You’ll read on our homepage that Teserakt’s first product brings end-to-end encryption to the MQTT protocol. But before describing our solution, let’s find out how MQTT works and where it’s used.

The MQTT site describes it as “a machine-to-machine (M2M)/’Internet of Things’ connectivity protocol”, although the Message Queue Telemetry Transport existed long before the term IoT was coined, namely since the late 90’s. Today MQTT is for example used for cloud infrastructure telemetry, in critical infrastructure systems such as dams or power plants, in connected cars, and in Facebook messenger.

One reason behind the success of MQTT is its simplicity. MQTT involves three types of entities:

  • Publishers, which create messages, tag them with a topic, and send them to a broker.
  • Subscribers, which subscribe to one or more topics, and then receive all messages tagged with these topics from a broker.
  • Broker servers, which dispatch messages from publishers to subscribers, and manage clients’ authentication and authorizations.

A client device can act as both publisher and subscriber. Publishers are for example sensors sending telemetry data to a back-end that runs the business logic (analytics, reporting, and so on). The broker and the back-end sometimes run on the same infrastructure.

In the example below, the clients on the left publish messages and the client on the right subscribes to the topics associated to these messages. After requesting the broker to subscribe to a topic, a client receives a SUBACK message from the broker, which confirms that the broker has registered the subscription.

You’ll notice that in the PUBLISH messages, we’ve set the “retain” flag to true, which means that the broker will retain the last message associated with a given topic, and will send it to a client even if it subscribes after the message was published. The diagram can thus be read in different ways: with the PUBLISH messages happening before the SUBSCRIBE messages, or the other way around.

You can play with MQTT by installing the mosquitto open-source broker and its suite of tools. First, start the broker locally by running the mosquitto command (add -d to run it as a background process). This will start an MQTT broker instance listening on port 1883. Then you can start a client instance that will connect to this broker and subscribe to topic mqttest:

mosquitto_sub -h localhost -p 1883 -t mqttest

In another terminal, publish a message to this topic:

mosquitto_pub -h localhost -p 1883 -t mqttest -m hello

The message hello will then appear on the first terminal running the subscribed client instance. You can try first publishing the message with the flag -r (telling the broker to retain the message) and subscribe afterwards. Don’t forget to stop your broker after running this test.

Mosquitto is a popular open-source broker, but if you use MQTT in production environments and need high availability and/or technical support, several options are available. Vendors such as HiveMQ and VerneMQ offer enterprise-ready broker software, and cloud platforms of AWS, Google, and IBM offer hosted MQTT brokers (note that MQTT is the only non-web protocol supported by their IoT platforms).

MQTT is super simple, but not very secure by default. Most deployment will (at best) run client–broker connections in TLS tunnels, for example in order to protect authentication credentials from eavesdropping. In subsequent posts we’ll show you how our technology addresses MQTT’s inherent security limitations, and we’ll also talk about brokers’ software security.