Releasing mqttinfo and scanning all open brokers with BinaryEdge

Today we’re releasing mqttinfo, a command-line utility we wrote to collect information on the configuration and behavior of a remote MQTT broker service. mqttinfo also attempts to guess the product type of the underlying software, in the same spirit as nmap’s OS detection – but with fewer and simpler heuristics at the moment.

mqttinfo is available at https://github.com/teserakt-io/mqttinfo in version 0.1.0, with almost no documentation, probably dozens of bugs, but we hope it’ll be useful nonetheless to MQTT users! We’ll do our best to fix any bugs found and to add new features.

As an example, here’s the output of mqttinfo when analyzing the open test server of the popular open-source broker mosquitto:

As you can see, mqttinfo supports MQTT versions 3.1.1 and 5, which are respectively the most common version and the recently standardized one. In this example, mqttinfo directly connects without any authentication because the service is open, but you can also pass a username and password to analyze authenticated brokers, as documented:

In the above example, the option -j will tell mqttinfo to append the results in JSON to a file ./mqttinfo.json, for example:

{"Host":"test.mosquitto.org","Port":1883,"Username":"","Password":"","V4":true,"V5":true,"V4Anonymous":true,"V4PublishSYS":true,"V4FilterSYS":true,"V4SubscribeAll":true,"V4InvalidTopics":false,"V4InvalidUTF8Topic":false,"V4QoS1":true,"V4QoS2":true,"V4QoS3Response":false,"V5Anonymous":true,"V5PublishSYS":false,"V5FilterSYS":true,"V5SubscribeAll":true,"V5InvalidTopics":false,"V5InvalidUTF8Topic":false,"V5QoS1":true,"V5QoS2":true,"V5QoS3Response":false,"TypeGuessed":"mosquitto","Failed":false,"Error":""}

It’s of course interesting to analyze one broker service, for example to detect misconfigurations or shortcomings of the broker you’ll be using in production. But it’s even better to analyze many brokers, such as all brokers available on internet. To do this efficiently and reliably, we worked with our friends at BinaryEdge, who offered to help after they noticed mqttinfo’s teaser.

BinaryEdge deployed mqttinfo as a module of their scanning platform, and ran it on all the hosts with port 1883 open (the port typically used for MQTT unencrypted connections). The scan took several hours—notably because mqttinfo’s software detection can take up to 20 seconds—and the main observations are the following:

  • Of all the hosts with TCP port 1883 open, 23287 authorized an anonymous connection over MQTT v3.1.1, by sending back a CONNACK packet with reason code 0x00. Many of these brokers may be there for testing and/or open on purpose, but many are evidently used by real devices, as BinaryEdge had already observed.
  • Of these 23287 hosts, about 70% seemed to be running Mosquitto as a broker—but keep in mind that our product detection is based on heuristics and can be easily fooled, so take this number with a grain of salt.
  • 71,5% of the hosts authorized clients to publish to $SYS topics (a behavior that we discussed in a previous post), a number only slightly greater than that of the mosquitto, and indeed almost all hosts identified as mosquitto share this property.
  • 77.5% of the hosts accepted an invalid UTF-8 string as a topic, whereas the standard states that “If a Server or Client receives a Control Packet containing ill-formed UTF-8 it MUST close the Network Connection”. We used the string \xc3\x28 to test this, and of course could not test all invalid UTF-8 strings.
  • 2.3% of the hosts accepted the invalid topic “A+”, which includes the wildcard character “+” in a way that is invalid, as per the standard.
  • A bit less than 1% of the hosts did not seem to support QoS level 1, with a similar number of instances not supporting QoS2. Surprisingly, about 4.5% of the broker instances (1055) accepted a QoS level of 3, which is not defined by the standard and should be rejected as invalid.
  • 88.5% of the hosts authorized clients to subscribe to all topics at once, by subscribing to the topic filter “#”.
  • Only about 10% of the hosts (2430) supported MQTT v5, of which only approximately 1570 seemed to support QoS levels greater than 0.

We’ve been running mqttinfo on several versions of the main broker products, and have often noticed different behaviors, not only across products, as expected, but also across versions. Furthermore, and in part due to differences in the norms defined by the standards, MQTT v3.1.1 and v5 interfaces may also behave differently even in a same version of a broker.

We have reported some of our observations to broker product maintainers, who often helped us understand MQTT better and sometimes corrected our understanding. We’d like to thank in particular HiveMQ and VerneMQ for their insightful feedback.

Last but not least, we’d like to thank Tiago from BinaryEdge and his team for offering to extend their worldwide scan of MQTT brokers with mqttinfo, and for rapidly providing us with the results!

Is MQTT secure? (A report)

Today we publish our report “Is MQTT secure?”, available here as a PDF, or directly below. (Such a report ought to be called a white paper, but having lost much of its prestige – and meaning – lately, that term didn’t sound right to us.)

When someone asks if something is secure it is not always easy to answer. People want a yes or a no even if they are incapable of explaining what “secure” means in the context. The last thing they want to hear is “it depends” – yet that’s often the least wrong answer.

This post elaborates on what “it depends” involves in the context of MQTT’s security, with a focus on confidentiality. Although we will focus on MQTT here, the same ideas apply to any protocol or communication between devices. Let’s start with some general background on the notion of security for online services.

“Secure”?

Imagine if a group of people are communicating using a social network or chat program, how then might we define security? There are a number of ways to look at this problem:

  1. Security of the service itself. Does the social network or chat server have any logic errors or exploitable code that might allow an unauthenticated user, or authenticated user not part of the group, to interfere with the messages of the group?
  2. Transport security to the service. Most internet packets take multiple hops between a user and the destination server, whether inside the destination social network’s network or outside of it. Can we trust these parties not to interfere with packets? Might they record them? Might other parties insert themselves into this path and do the same?
  3. End-to-end security. This ensures confidentiality and integrity between the communicating participants such that no intermediaries, even the social network or chat server, can view or tamper with communications. All they know are who is communicating with whom.
  4. Sender and recipient untraceability. This is where we do not know who is sending to whom at all, a.k.a. anonymity. We list this for completeness, but it’s hard to implement effectively and for this reason rarely guaranteed. The obvious real world example of this is Tor.

So how do common service providers fare when considering these security criteria?

Most services provide transport security through the use of TLS (you might know this as SSL – TLS is the protocol that has replaced SSL but the names are often used interchangeably). Whether you upload a photo on Facebook or Google services, or chat using Slack, you are doing this via TLS and subject to TLS being secure, so are your communications. Since the mass surveillance revelations by Edward Snowden from 2013 onwards, there has been an increasing push to deploy encryption and in particular TLS: Chrome now marks unencrypted sites as insecure, there’s Let’s Encrypt, “end-to-end encryption” has a Wikipedia page; Snowden made crypto jump from subculture to mainstream, which is probably good, all things considered.

The security of online service providers is, however, never a total guarantee (although we know they employ word class engineers and teams to guarantee the best level of protection they can). Things are even worse when you move away from companies that understand the internet. Most penetration testers can tell you horror stories of out of date software, poorly written and possibly outright insecure code, lack of hardening and no process to manage this for critical production software.

TLS’ limitations

Transport security is a great goal, but it cannot protect you if the security of the service in question is compromised, or if some of the service’s operators decide to look into your data. So point 2 (transport security) does not mitigate point 1 (service security). Specifically, suppose we have the following sequence of events:

  1. User Alice uploads a photo to a social networking website. Her browser negotiates TLS with the service and her photo is secured in transit.
  2. The service stores the photo unencrypted in one of their data centres.
  3. User Bob wishes to view the photo Alice just posted – perhaps it was a skiing trip to Verbier? He requests the photo from the service and being authorized by Alice already, he receives the photo. His browser likewise successfully negotiates and uses TLS.

We can visualize this in the following diagram:

The communication from Alice to the service and the service to Bob is secure, but the service itself has an unencrypted copy of Alice’s photo. So if a logic error exists, or the service is otherwise compromised, that photo can be extracted by the attacker.

End-to-end security comes in to play at this stage: leaving out the details, if Alice encrypts a photo for Bob, then the service in question can no longer view the photo. They can only disrupt its transmission, as the diagram below shows:

The machine-to-machine case

In the most popular machine-to-machine (M2M) protocol, MQTT, messages are sent to a server, called the broker, before being forwarded to their recipient(s). Most broker software support TLS, so you can get transport security today. This is excellent news and ensures that messages cannot be read by intermediate internet nodes.

What TLS does not protect against is, of course, the compromise of the broker. This could happen for a whole number of reasons, from outright logic errors in the software to misconfiguration by the user. For example, a vulnerability such as the one we found in a popular broker software may be leveraged to get access to the host running the broker, thereby granting the attacker’s access to all MQTT messages processed by the broker.

As stated above, TLS also leaves data exposed to operators of the broker, whom may not be trustworthy enough for critical applications – for example, if access to a confidential data stream is worth $1M to a competitor, said competitor could bribe or blackmail an operator for a fraction of this cost. Corporate espionage is a very real risk for many companies today and has never been easier. Companies that manage critical national infrastructure are particularly at risk.

Even worse, some devices rely on publicly managed brokers. When sharing a broker with other service providers in this way, while all information may be secure when talking to the broker, the only thing preventing other clients accessing your devices’ information is the access control implementation on the broker. If such a public broker is compromised or implements access control poorly, those messages can easily be read and modified. As we’ve recently observed, access control are not always safe by default, which increases the risk.

Increasingly, vehicles are becoming connected devices too. The data shared by vehicles might range from engine performance information that is of interest to competing manufacturers, to personal information such as owner location and the journeys they make. This information almost certainly comes under the purview of ever stringent data protection law (for example the GDPR).

This is why there is a need for end-to-end security in the M2M, V2X, and IoT spaces. As increasingly amounts of sensitive data are sent between autonomous nodes, such as vehicle locations or presence in private homes, it becomes even more important to ensure that even in the event of relay, proxy or broker compromise, only the intended recipients of those messages can read the data.

Intermission – MQTT & TLS

As far as confidentiality is concerned, MQTT only briefly mentions using TLS. The standard says: “The MQTT protocol requires an underlying transport that provides an ordered, lossless, stream of bytes from the Client to Server and Server to Client.”, and then states that, although the underlying transport is TCP/IP, TLS and WebSocket are “also suitable”.

In the MQTT standard’s Security section, we read that “it is strongly recommended that Server implementations that offer TLS SHOULD use TCP port 8883”. However, the standard does not recommend – let alone strongly – that TLS be used. Instead, the same section notes that “[as] a transport protocol, MQTT is concerned only with message transmission and it is the implementer’s responsibility to provide appropriate security features. This is commonly achieved by using TLS.” Fair enough.

(The above excerpts are taken from MQTT 3.1.1’s specification, but the latest one, 5.0, includes almost identical text.)

Cryptography of things

Implementing state-of-the-art cryptography in resource-constrained environments can present a technical challenge. In some constrained environments, the device may not be powerful enough to run public-key cryptography and/or to include a TLS implementation. Such environments can for example include some RFID chips or 8-bit AVR processors. But most of the time you’ll have symmetric cryptography algorithms such as AES and SHA-256. Yet crafting a protection scheme compatible with the system constraints isn’t always straightforward – for example, how do you add authentication when you can’t extend the payload size?

In more extreme cases – legacy hardware, high throughput constraints, and so on – standard crypto may still be insufficient. For such cases, NIST is running a competition that aims to standardize so-called lightweight cryptography algorithms. Note that the niche field of lightweight cryptography is all but new, and that many lightweight crypto schemes already exist; they’re just not blessed by NIST. (Fun fact: the MSc and PhD theses of a Teserakt founder were about lightweight crypto, back in 2006 and 2009, before it was cool!)

Protecting messages with cryptography is generally a solved problem, and in our experience the trickiest part is the case-specific engineering problem. Specifically, a non-trivial problem is to create a system that remains safe even if the device is partially compromised, if its pseudorandom generator fails, or if its clock is not accurate, yet consumes minimal energy and does not incur significant performance hits.

The hard part

In the same way that you can’t solve the halting problem, or that you can’t generate randomness deterministically, you can’t create trust with cryptography, no matter how much cryptography you use – to cite Jon Callas. In machine-to-machine networks, trust often simply consists in the association between a device identity and a cryptographic key. From there, we’ve got two main classes of problems related to key management:

First, problems that relates to key provisioning and trust: How do unique keys end up on devices? At what stage of the production chains are they generated and provisioned to the device? Can a device generate its own keys? How would they share their public or secret keys with other parties? What trust model should be used, pre-shared keys, trust-on-first-use, centralized and hierarchical PKI, or P3KI‘s decentralized model? There are no generic good answers to these questions – again, it depends.

Second, how do you encrypt, or more generally, how do you protect messages once every devices has its key? How are session keys derived from identity keys? Should there be a notion of session at all? How can you ensure that previous communications won’t be compromised if a key is compromised later (forward secrecy)? What about the opposite (backward secrecy – if a current key is compromised can we recover and protect future messages)? How do you realize secure group messaging among large sets of devices, as opposed to humans? How can you replace humans’ operations, and how can you leverage the absence of humans to better automate things? These questions and many others are mostly unexplored territory, and are our focus at Teserakt. We don’t claim to have all the right answers (we definitely don’t), but are doing our best to find the right trade-offs to provide the highest security at the lowest cost.

Is MQTT secure?

After reading the above you’ll now realize that the question is not the right one to ask. MQTT is not insecure, but it’s not secure, because it’s not designed to be secure. A slightly better question to ask could be “Are MQTT broker services secure?”, but then again it depends. A better question would be “Are MQTT broker services, and their related software, secure enough for application X?”.

An even better question would be “Do you have enough trust in the broker software, in the organization that runs it and its personnel, in said organization’s ability to perform security updates (not only of the broker but also of related software, such as hypervisors) to leave all your messages exposed to espionage and tampering today and a few years from now?”

Many organizations will acquiesce, and rightfully so, noting that they care more about availability than confidentiality and integrity. That could for example be the case of weather measurement data.

But many other organizations will reply that no, they think that risk is too high to be neglected or accepted. This might be the right answer when data transmitted relate to private individuals’ activity, to critical infrastructure systems, to safety monitoring systems, to proprietary technology (which may be reverse engineered only from metadata), to software updates, to geolocation information, and many other types of information that needs be protected to ensure a sustainable business.

The future

We would like to think the internet was always ready for e-commerce and that the cryptography and security of systems protecting it have always been fit for purpose. However, the truth is that our understanding of secure systems and our expectations have evolved over time. SSL Version 1 never made it outside of Netscape and SSH version 1 is considered badly broken. We have iterated these protocols over the last several decades to arrive at today’s level of transport security.

Security in personal messaging has likewise evolved from obscure, niche origins (such as OTR over what was originally called Jabber). The first attempt to bring end-to-end security to this space was TextSecure and RedPhone around 2012, a time in which all other messaging applications relied only on transport security. These applications were merged into Signal. Following the success of Signal, the protocol was licensed by and integrated into WhatsApp – among others – and is now used daily by around 1.5 billion people. Alternative products such as Wire, Telegram, and Facebook Messenger either launched with similar security standards, or rolled them out on their platforms.

Teserakt’s vision is that the same evolution is happening in the embedded, IoT, V2X, and M2M spaces. Today, messaging protocols such as MQTT and Kafka are at best providing transport security. We want the state of the art of end-to-end-security to apply to the smart home, to critical national infrastructure, to the cars you drive, and to the medical devices you use.

Solving the hard problem with E4

If you need end-to-end security, you should contact us to try our product, E4, which we believe is the best option today for integrating solid encryption and key management in IoT systems. We have solved the hard problems so that you don’t have to.

Securing the mosquitto MQTT broker

Summary: The mosquitto broker authorizes publication to SYS topics by default, which is a potential security risk. We explain how to address this issue and provide some background on MQTT brokers’ default security.

MQTT is a publish/subscribe messaging protocol: when a device sends out a message, it won’t explicitly send it to a given device or address, but instead the sender tags the message with a string called a topic. All devices that have previously subscribed to that topic will then receive the message; pretty dumb logic, but efficient and enough for most IoT and machine-to-machine networks.

In MQTT, topic names usually follow a topic hierarchy. For example, data from the New-York State power grid can be accessed using MQTT under topics such as “ny-power/upstream/fuel-mix/Hydro” or “ny-power/upstream/fuel-mix/Wind”. MQTT also supports a wildcard symbol: if a device subscribes the the topic “ny-power/#”, then it will receive all the messages published under topic names starting with “ny-power/”.

In MQTT, the server implementing the publish/subscribe logic—as well as authorization and authentication mechanisms—is called a broker. The best known MQTT broker is arguably mosquitto, an open-source broker part of the Eclipse Foundation and an iot.eclipse.org project.

The first thing you have to be careful with when using mosquitto is that by default it is open to anonymous connections—that is, it allows anyone to connect to it and use it. Such an open broker would allow anyone to see the messages published by subscribing to their topics. A number of open brokers are available for testing, and you can also find open brokers evidently used for production and critical systems. In contrast, an enterprise-oriented broker such as VerneMQ will not allow you to connect to it without having first created authorized credentials, which is the safest default configuration.

Many systems and services will default to an insecure configuration, because it’s more convenient and because such services will often run in closed networks, where the risk is lower. For example, popular services such as ElasticSearch default to unauthenticated, unencrypted connections. Also, some MQTT brokers are only available via VPN, which minimizes the risk of open brokers. Insecure defaults are thus not always a big deal, as long as they’re documented and that the risk is obvious to most users.

Recently we observed another case of insecurity-by-default, which is the point of this post: so-called SYS topics. These are a special class of topics under which the broker publishes data, typically for monitoring purposes. SYS topics are not a formal standard but are an established practice in MQTT brokers. For example, mosquitto defines SYS topics such as:

$SYS/broker/clients/maximumThe maximum number of clients that have been connected to the broker at the same time.

$SYS/broker/clients/totalThe total number of active and inactive clients currently connected and registered on the broker.

$SYS/broker/messages/receivedThe total number of messages of any type received since the broker started.

As explained by HiveMQ, provider of an enterprise MQTT broker: “These topics are special meta topics which are used by the broker to publish information about the broker itself and its MQTT client sessions. All SYS-Topics start with $SYS and are read-only for MQTT clients, so publishing to these topics is prohibited.” We indeed verified that the HiveMQ broker prevents publication to SYS topics in its default configuration. The aforementioned VerneMQ does it too.

Furthermore, the MQTT standard states (as non-normative comment) that “Applications cannot use a topic with a leading $ character for their own purposes”, which suggests that publication SYS topics should be restricted to the broker.

Mosquitto, however, authorizes by default any client to publish messages under SYS topics, and therefore to potentially send fake status data to consumers. In particular, mosquitto’s public broker test.mosquitto.org allows clients to publish to $SYS/#. You can for example send fake data as follows, where I sent an invalid number of connected clients:

First, subscribe to the topic $SYS/broker/client/connected, (where the dollar sign must be escaped in my terminal):


$ mosquitto_sub -h test.mosquitto.org -t "\$SYS/broker/clients/connected" -d
Then, in another terminal, publish an invalid number of connected clients under that topic (here, 414141414141):

$ mosquitto_pub -h test.mosquitto.org -t "\$SYS/broker/clients/connected" -m 414141414141 -d

The first terminal will then show that it received correct values (968, 970), followed by the incorrect one.:


Client mosqsub|80540-id received PUBLISH (d0, q0, r0, m0, '$SYS/broker/clients/connected', ... (3 bytes))
968
Client mosqsub|80540-id received PUBLISH (d0, q0, r0, m0, '$SYS/broker/clients/connected', ... (3 bytes))
970
Client mosqsub|80540-id received PUBLISH (d0, q0, r0, m0, '$SYS/broker/clients/connected', ... (12 bytes))
414141414141

 

We reported this observation to mosquitto maintainers, who responded that this permissive behavior is intended and well known. Unlike opening the service to anonymous connections, however, authorizing publication to SYS topics doesn’t have any usability or convenience benefit, so we respectfully challenge this design choice.

It is therefore the responsibility of users to prevent the publish capabilities to SYS topics. This can be done via access control lists (ACL), as for example documented here.

To conclude, if you run an MQTT broker make sure that:

  1. Your service is not open to anyone (well-known risk);
  2. Only the broker can publish to SYS topics (also note that SYS topics should not be used as a source of production metrics).

Introducing PQGo, experimental post-quantum cryptography in Go

TL;DR: We published cgo wrappers for post-quantum crypto at https://github.com/Teserakt-io/PQGo.

Post-quantum cryptography is about public-key algorithms that wouldn’t be broken by a quantum computer—unlike RSA, Diffie-Hellman, and elliptic curve cryptography, which all rely on the hardness of problems that are hard with a classical computer but become easy for quantum algorithms. For more on this topic, we refer to our presentation at BSides Lisbon 2017.

Quantum computers that would break our crypto aren’t coming anytime soon, but as an insurance against a breakthrough in quantum engineering, NIST is running a public competition to standardize post-quantum crypto in the next five years. This project sparked greater interest from researchers and industry, who see the opportunities from the publicity and investments around the quantum computer threat—often blatantly exaggerated.

To respond to the demand from customers and as a research interest, we at Teserakt investigated the integration of post-quantum crypto in our products. Our products’ reference code is in Go, whereas implementations of post-quantum crypto are in C. We therefore wrote a Go package using cgo to call the C code, which we adapted from reference implementations of some NIST submissions.

Our package github.com/Teserakt-io/PQGo currently supports the signature scheme Dilithium and the KEMs Kyber and Round5. As explained in the README, this choice isn’t a general endorsement nor a recommendation, but is based on some of our functional and performance requirements.

We tried to simplify the API and to hide some of its quirks, such as the fact that the C signature verification function will sometimes write at the message pointer’s address more data than the message’s actual length. Our package includes benchmark routines, but remember that we took the reference implementations rather than the optimized implementation, so an algorithm’s measured performance may be far from its optimal performance.

Please let us know your feedback by reporting issues on GitHub, submitting PRs, or writing us! We’d be happy to integrate more algorithms, please let us know your requests.

WARNING: Experimental code, don’t use in production or for anything that matters. (We know from experience that such statements don’t stop people from using the code in production or in their blockchain, but don’t say we didn’t warn you.)

 

Teserakt partners with VerneMQ

We’re happy to announce our partnership with Octavo Labs AG, the company behind the open-source MQTT broker VerneMQ. At Teserakt we have reviewed part of VerneMQ’s internals and source code, and also tested it in combination with our MQTT encryption solution. We really liked VerneMQ for reasons including its:

  • Security: Although Teserakt’s E4 solution aims to protect against compromised brokers, we prefer to use a secure broker. VerneMQ follows best security practices, minimizes the risk of memory corruption bugs, and is one of the few brokers with secure-by-default settings.
  • Scalability: Thanks to its Erlang-based design, VerneMQ easily scale horizontally and vertically by fully utilizing multicore architectures.
  • Ease of use: VerneMQ is simple to build, run, and configure, and offer lots of options as well as a comprehensive documentation.

As described in more detail in our slide deck, the partnership consists in the following actions:

  • Octavo Labs includes Teserakt’s encryption technology in their MQTT offering, to allow their customers to directly deploy end-to-end secure MQTT in their infrastructure.
  • Teserakt selects VerneMQ as its first approved broker, which concretely means that VerneMQ will be part of our continuous integration process and test suites, and will be recommended to Teserakt customers (other brokers may be approved after a technical assessment from our team).

We’d like to thank the VerneMQ team for this collaboration! For any questions, please get in touch at [email protected].

 

Teserakt partners with P3KI

We’re happy to join forces with P3KI GmbH, provider of the P3KI Core decentralized PKI solution for access delegation applications. We like what P3KI is doing because we think that it’s useful, well-engineered, and developed with security in mind.

The P3KI Core product can be seen as a kind of PKI but with more decentralization, and with precise expression of permission levels thanks to a mathematically-verified language, which adresses most authorization and authentication challenges.

Concretely, this partnership consists in bundling the P3KI’s authorization management and Teserakt’s key management services in a single appliance, simplifying and reducing the cost of integration of these two complementary products. More details are available in our slide deck.

For any questions, please get in touch at [email protected]. See also P3KI’s announcement.

Transport Security and LwM2M

Network World recently published an article on the differences between the MQTT and LwM2M protocols for businesses considering which to adopt. While there are a number of interesting differences between the standards, our main interest is of course security. In particular we are interested in end-to-end encryption.

This term has become popular with the rise of secure messaging software such as Signal and Wire, and products that have adopted the Double Ratchet protocol such as WhatsApp and Facebook Messenger. In these contexts, end-to-end means from person to person: sender to their intended recipient(s). In our machine-to-machine world we mean exactly the same thing: when an IoT device communicates with other IoT devices, end-to-end for us means that any brokers, proxy servers or other intermediaries cannot decrypt or alter the messages.

With this definition in mind, we can take a closer look at the article and specifications. On the topic of what is provided in the existing standards, the article has this to say:

Modern revisions of the MQTT specification support features such as end-to-end encryption using either transport layer security (TLS) for the entire channel or payload encryption… For message security, CoAP and LWM2M typically utilize datagram transport layer security (DTLS) for data payload encryption. Unlike MQTT and TLS, CoAP messages encrypt the payload of the message rather than the entire message transport.

Indeed, the MQTT specification does suggest the use of TLS for transport encryption. It does not provide encryption for payloads or application messages (unless the transport itself is encrypted), although it indicates that users may wish to implement this themselves:

TLS [RFC5246] can provide encryption of data sent over the network…. An application might independently encrypt the contents of its Application Messages.

So on the subject of end-to-end encryption, MQTT leaves any implementation up to vendors.

LwM2M is a set of several standards comprising several parts—at the time of writing version 1.1 is the most recent edition. In the transport standard document, the section on session security begins as follows:

CoAP over UDP [CoAP] is secured using the Datagram Transport Layer Security (DTLS) 1.2 protocol [RFC6347], as defined in Section 9 of [CoAP]

So, LwM2M uses a variant of TLS called Datagram Transport Layer Security, or DTLS, suitable for use with UDP and other streaming transports. Like TLS, this protocol operates a handshake and establishes trust based on X.509 PKI Infrastructure. CoAP messages are then encrypted as DTLS application data, or in TLS parlance, records. This differs from MQTT only in that DTLS is used; DTLS is used because unlike MQTT, LwM2M supports UDP and DTLS is a suitable protocol for this transport type. All in all, this is not controversial.

As at the broker in MQTT, TLS is removed in LwM2M whenever a proxying device or change of transport protocol is required (e.g. HTTP to SMS). LwM2M is interesting because it also provides a protocol called OSCORE. Quoting again from the transport security section of the transport specification:

Object Security for Constrained RESTful Environments [OSCORE] is an application layer security protocol protecting CoAP message exchanges. OSCORE is applicable to protocol messages which can be mapped to CoAP or a subset of CoAP, including HTTP and LwM2M. The use of OSCORE with LwM2M is optional. It MAY be used for protecting the communication between a LwM2M Client and the LwM2M Bootstrap-Server. It MAY be used between a LwM2M Client and a LwM2M Server.

In this context, OSCORE allows for sender and recipient IDs independently of their role in the network. However, it seems from the Core LwM2M specification that the use of OSCORE in LwM2M is solely for client-server communications. In defining object types, the documentation says:

This LwM2M Object provides the keying material and related information of a LwM2M Client appropriate to access a specified LwM2M Server using OSCORE.

This may be because the purpose of LwM2M seems to be explicitly focused on the management of devices and is less focused on messaging between devices. If a device were to communicate information to an LwM2M server that then needed to be re-transmitted to another device and the server did not “need-to-know”, we would not consider this end-to-end cryptography. On the other hand, where the goal is to communicate between LwM2M server and client, OSCORE does indeed meet our definition of end-to-end—any proxy on the path would not need (or be able) to remove the cryptography, guaranteeing confidentiality on the message path. The OSCORE specification explains this succinctly:

In scenarios with intermediary nodes such as proxies or gateways, transport layer security such as (D)TLS only protects data hop-by- hop. As a consequence, the intermediary nodes can read and modify any information. The trust model where all intermediary nodes are considered trustworthy is problematic, not only from a privacy perspective, but also from a security perspective

LwM2M further specifies the cryptographic algorithm choices:

OSCORE is defined for use with an Authenticated Encryption with Additional Data (AEAD) algorithm and a HMAC-based Key Derivation Function (HKDF) [RFC5869]. OSCORE mandates the implementation of the AEAD algorithm AES-CCM-16-64-128 (also known as CCM*) as defined in [RFC8152], and specifies the use of HMAC with SHA-256 as default.

These are very interesting choices. In particular, as things stand, they offer fairly good security. CCM used alone is vulnerable to nonce-misuse, in particular on embedded devices where random number generation may not be ideal, so not addressing this also seems to be a shortcoming. There is also an open mention of “other algorithms” as necessary to match the “state of the art” but these are not specified. Overall, however, assuming implementations use these algorithms correctly and keys are appropriately generated and provisioned these choices are well regarded. Our own solution, E4, offers slightly stronger security guarantees as a minimum—we offer nonce-misuse resistance.

OSCORE relies on a master key with 128 bits of entropy to derive further encryption keys and initialization vectors using a hash-based key derivation function:

OSCORE depends on a pre-established random Master Secret (Section 12.3) used to derive encryption keys, and a construction for making (key, nonce) pairs unique (Appendix D.3).

This is also uncontroversial and is intended to allow multiple keys for different purposes to be generated from a single source of entropy. The specification leaves the question open as to whether a single master secret should be used or multiple should be allowed to enable “forward secrecy”—in case this is desired, the specification suggests “a key establishment protocol”, giving an example specification annex to OSCORE.

A major feature of modern secure messengers we wanted to replicate for E4 was secure group messaging, scalable to large numbers of members. E4 supports this natively. OSCORE appears to have an extension draft for group messaging. LwM2M calls group messaging Observe-Composite. It is unclear how LwM2M proposes OSCORE be used in this scenario.

We have not covered the scenarios in which you may want to use each of the protocols in question, but it should be clear from this brief summary that they do serve slightly different use cases. We think it is very encouraging that standards bodies are considering the need for end-to-end encryption solutions and the cryptographic choices in OSCORE are reasonable. We are in particular encouraged by the move away from TLS, which is not suitable for such environments. DTLS in particular sees less scrutiny than TLS, while both protocols are complicated to deploy and manage. (D)TLS also involves an expensive handshake setup stage per hop, which is expensive where bandwidth is at a premium and slows communications down unnecessarily.

One criticism we would offer is that LwM2M is a complicated standard spanning multiple documents for multiple purposes and some components of the end-to-end encryption are referenced from further standards. In our own product, we strive for simplicity as an explicit goal, keeping unknowns and optional choices to the absolute minimum while guaranteeing the highest cryptographic and software security possible.

You may have been wondering how Teserakt’s E4 compares to OSCORE and DTLS/TLS in terms of protocol integration. The answer is that we are protocol-agnostic. Our reference implementation works on top of MQTT, however, we are working on integrations with other transport protocols. E4 could be used for application layer security in LwM2M. We have a clear set of choices for key rollover. OSCORE itself is not tied to the architecture of LwM2M, but its use in LwM2M focuses on client-server. E4 does not assign roles to the sender or recipient and is suitable as a building block for any application, whether communicating between two endpoints or communicating as part of a group.

In summary, LwM2M is an interesting development. From a security perspective, the adoption of end-to-end cryptography is very encouraging, although we would slightly tweak the cryptographic choices. There appears to be work to do at the interface between LwM2M and OSCORE to support group messaging.