☰
Navigation :

MQTT

MQTT, or Message Queuing Telemetry Transport, is a great protocol of when working with microcontrollers and sending relative small data payloads across networks. It was designed for the Internet of Things in mind, allowing for the fact that connections will invariably break and recover as devices move in and out of network range.

The best MQTT library for the mote mini is a fork of Nick O'Leary's PubSubClient by Ian Tester (@imroy) (BTW Nick O'Leary is also the lead developer for the nodeRED project). We are using @imroy's version as it supports the definition of custom port numbers. You need to install this library manually from github:

A Brief Introduction

MQTT is a protocol – an agreed-upon set of communication methods that devices can use to send data between each other. Think of this as two devices saying: “OK, let's use English as the language to talk to one another. We'll do whatever we need to do on our ends to communicate with our sensors/peripherals, but when I send the data over to you, it'll be in English.”

The key things to note are that you'll need the following ‘actors’ in a MQTT topology:

graph LR; style Z fill:#f9f,stroke:#666,stroke-width:1px A[mote A] --- Z("MQTT server
(running on IoTa server)") B[mote B] --- Z C[any device supporting MQTT] --- Z Z --- D[mote C] Z --- E[mote D]

You need an MQTT server to handle incoming/outgoing messages. The server is like a post office – it routes and/or broadcasts messages depending on who the incoming message it meant for. One difference, as hinted in the previous sentence, is the ‘broadcast’: you can create a message that is meant for an entire fleet of devices through what is known as wildcarding.

HiveMQ does a great job of describing wildcarding – be sure to read it.

MQTT in the Arduino IDE environment

To use MQTT with your mote / mote mini, you will need to have WiFi properly set up. Once your mote has Internet connectivity (or local network where your server is running on), you can then configure the MQTT events and listeners.

1. Set up WiFi

The following code provides a simplified, annotated version of the WiFiClient example found in the Arduino IDE (File > Examples > ESP8266WiFi > WiFiClient). It also has additional lines specific to our server configuration that you should take note of:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

#include <ESP8266WiFi.h>            // include the WiFi connection library

#ifndef STASSID
#define STASSID "your-ssid"         // enter your WiFi SSID here
#define STAPSK  "your-password"     // and your WiFi password
#endif

const char* ssid     = STASSID;
const char* password = STAPSK;

void setup() {
  Serial.begin(115200);

  // We start by connecting to a WiFi network
  Serial.println();
  Serial.println();
  Serial.print("Connecting to ");
  Serial.println(ssid);

  /* Explicitly set the ESP8266 to be a WiFi-client, otherwise, it by default,
     would try to act as both a client and an access-point and could cause
     network-issues with your other WiFi-devices on your WiFi-network. */
  WiFi.mode(WIFI_STA);
  WiFi.begin(ssid, password);

  while (WiFi.status() != WL_CONNECTED) {       // wait until we connect
    delay(500);
    Serial.print(".");
  }

  Serial.println("");
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());     // if all good, Serial Monitor should show your mote's IP address
}

void loop() {
  delay(1000);    // nothing to do in this basic connection test!
}

Upload and test this code on your mote mini – remember to open the Serial Monitor at 115200 baud to listen in to what your mote mini is doing. If you get the WiFi connected message along with a local IP address, you are good to proceed to the next step.

2. Create a MQTT connection

Let's build on top of the above example and throw in the MQTT libraries. We will connect to the MQTT server that's running on iota.mdit.space.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92

#include <ESP8266WiFi.h>            // include the WiFi connection library
#include <PubSubClient.h>           // include the MQTT library

#ifndef STASSID
#define STASSID "your-ssid"         // enter your WiFi SSID here
#define STAPSK  "your-password"     // and your WiFi password
#endif

#define MQTT_USER "!!!!"            // refer to #motemini Slack channel for credentials
#define MQTT_PWD  "!!!!"            // refer to #motemini Slack channel for credentials
#define MQTT_PORT 1888

const char* ssid     = STASSID;
const char* password = STAPSK;

String chipID;
String devID;

IPAddress mqtt_server(?, ?, ?, ?);   // refer to #motemini Slack channel for credentials
WiFiClient wclient;
PubSubClient mqtt_client(wclient, mqtt_server, MQTT_PORT);

void mqtt_callback(const MQTT::Publish& pub) {
  // expects non-stream data from node-RED mqtt node
  Serial.print(pub.topic());
  Serial.print(F(" => "));
  Serial.println(pub.payload_string());
}

// a custom function to process the unique chip ID of a mote/mote mini
String parseChipID(uint32_t cid) {
    // takes the last 6 chars of the chip ID
    String c = String(cid, HEX);
    String retStr;
    uint8_t l = 6 - c.length();
    for (uint8_t i = 0; i < l; i++) {
        retStr += "0";
    }
    retStr += c;
    return (retStr);
}

void setup() {
  Serial.begin(115200);
  delay(10);
  Serial.println();
  Serial.println();

  chipID = parseChipID(ESP.getChipId());
  devID = "ESP" + chipID;

  mqtt_client.set_callback(mqtt_callback);
}

void loop() {
    // moved the WiFi checking into loop since we need to constantly
    // check if we are connected!
    
    if (WiFi.status() != WL_CONNECTED) {    // WiFi isn't connected
        Serial.print("Connecting to ");
        Serial.print(ssid);
        Serial.println("...");
        WiFi.begin(ssid, password);

        if (WiFi.waitForConnectResult() != WL_CONNECTED)
            return;
        Serial.println("WiFi connected");

    } else {        // WiFi is connected

        // maintain MQTT connection by calling its loop function as often as possible
        if (mqtt_client.connected()) {
            mqtt_client.loop();
        } else {
            Serial.println("mqtt reconnecting...");
            // if MQTT client has disconnected in any way, reconnect
            if (mqtt_client.connect(MQTT::Connect(devID)
                                    .set_auth(MQTT_USER, MQTT_PWD)
                                    .set_keepalive(10)
                                    .set_clean_session(false)      // false = durable connection; subscriptions and queued messages will remain when we reconnect
                                    )) {
                Serial.println("mqtt publishing...");

                // notice the first level of the topic (before the / )
                // points to a specific group – you MUST do this to prevent
                // receiving messages from other groups
                mqtt_client.publish("group0/outTopic", "hello world");  // sends to a MQTT receive node set up with a topic name of "group0/outTopic"
                mqtt_client.subscribe("group0/inTopic");    // listens from a MQTT send node set up with a topic name of "group0/outTopic"
            }
        }
    }
}

Unpacking this code…

Seems like quite a lot of new code here! Here are the same details broken down into bite-sized sections. Please note that the following 2x sections are just excerpts of code from the above recipe, so these won't compile if you only paste the below blocks into your Arduino IDE:

2a. Initialisation

1
2
3
4
5
6
7

// initialisation
String chipID;
String devID;

IPAddress mqtt_server(?, ?, ?, ?);      // refer to #motemini Slack channel for credentials
WiFiClient wclient;
PubSubClient mqtt_client(wclient, mqtt_server, MQTT_PORT);

This first part initialises the basic WiFi and MQTT instances that will be used in the rest of the sketch.

chipID and devID are just convenient String variables that store a unique code for your mote/mote mini in the device memory. This is crucial because you might have multiple devices talking to the server, and you want unique names for each. An example of devID (yours will have a different 6-digit code after ‘ESP’):

ESP9a3c23

2b. Callback

1
2
3
4
5
6

void mqtt_callback(const MQTT::Publish& pub) {
  // expects non-stream data from node-RED mqtt node
  Serial.print(pub.topic());
  Serial.print(F(" => "));
  Serial.println(pub.payload_string());
}

Callbacks form one of the core concepts of Internet-related data services. Typically, a device (any device, from a mote to a laptop) puts out a request to some address, then continues doing its own tasks. When the response comes back, the callback function is triggered and the device process the incoming response. This is important because the Internet will not return information immediately, even if you have a very fast connection. This callback concept allows your device to not ‘hang’ and freeze up waiting for a response, which may also never arrive (for example, when your connection drops).

This block here basically prints the incoming topic and payload from the MQTT server. You can then, in this function, process the data and act on it accordingly.

2c. Custom device ID parser

A parser is just a little script that processes incoming data into something else. This function simply builds the ‘ESPxxxxxx’ device ID name from the system device information:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

// a custom function to process the unique chip ID of a mote/mote mini
String parseChipID(uint32_t cid) {
    // takes the last 6 chars of the chip ID
    String c = String(cid, HEX);
    String retStr;
    uint8_t l = 6 - c.length();
    for (uint8_t i = 0; i < l; i++) {
        retStr += "0";
    }
    retStr += c;
    return (retStr);
}

2d. void setup()

You should know what void setup() does by now – it runs code ONCE when the device starts up/resets. In here it just establishes a Serial port connection via the mote programmer board's USB port, initialises the unique device ID, and registers the callback function to trigger when it receives data from the MQTT server:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

void setup() {
  Serial.begin(115200);
  delay(10);
  Serial.println();
  Serial.println();

  chipID = parseChipID(ESP.getChipId());
  devID = "ESP" + chipID;

  mqtt_client.set_callback(mqtt_callback);
}

2e. void loop()

You should know what void loop() does, too – it runs code REPEATEDLY after setup concludes. This is where most of the interaction/processing happens. Due to its length let's break it down further into two segments:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

void loop() {
    // moved the WiFi checking into loop since we need to constantly
    // check if we are connected!
    
    if (WiFi.status() != WL_CONNECTED) {        // WiFi isn't connected
        Serial.print("Connecting to ");
        Serial.print(ssid);
        Serial.println("...");
        WiFi.begin(ssid, password);

        if (WiFi.waitForConnectResult() != WL_CONNECTED)
            return;
        Serial.println("WiFi connected");

    ...

This first section of the loop block above checks that we are still connected via WiFi – remember that devices will invariably drop off a WiFi network quite easily – walking it out of range, router resets, cat knocking the router off, etc… When it goes down, it will attempt to reconnect.

Next:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

    ...

    } else {        // WiFi is connected

        // maintain MQTT connection by calling its loop function as often as possible
        if (mqtt_client.connected()) {
            mqtt_client.loop();
        } else {
            Serial.println("mqtt reconnecting...");
            // if MQTT client has disconnected in any way, reconnect
            if (mqtt_client.connect(MQTT::Connect(devID)
                                    .set_auth(MQTT_USER, MQTT_PWD)
                                    .set_keepalive(10)
                                    .set_clean_session(false)      // false = durable connection; subscriptions and queued messages will remain when we reconnect
                                    )) {
                Serial.println("mqtt publishing...");

                // notice the first level of the topic (before the / )
                // points to a specific group – you MUST do this to prevent
                // receiving messages from other groups
                mqtt_client.publish("group0/outTopic", "hello world");  // sends to a MQTT receive node set up with a topic name of "group0/outTopic"
                mqtt_client.subscribe("group0/inTopic");    // listens from a MQTT send node set up with a topic name of "group0/outTopic"
            }
        }
    }
}

This portion above then proceeds with MQTT connection checking. Remember that WiFi and MQTT are two different things. The MQTT service needs WiFi to run; without WiFi there can be no MQTT service.

Next we perform a similar connectivity check in line 7 – if the MQTT connection has gone down, reconnect again, provided our WiFi connection is still up. mqtt_client.loop() basically ‘pings’ the MQTT server once every so often just to let it know that the device is still connected and ‘alive’ – again, very useful things to have when dealing with wireless connections.

Finally, let's take a closer look at lines 11-23 from the above code block:

Next: where and how does this data go on the MQTT server?

We now have covered basic MQTT functionality on a mote/mote mini. That's Part 1, getting the electronic device to subscribe to and publish MQTT data. Part 2 deals with processing the data on the server.

Follow through to the IoTa > IoTa + MQTT section to see how we can accept the "hello world" data on the server, and send a reply back to our device.

Once you get the basics of this relationship between device and server, you can then proceed to replace "hello world" with sensor readings, instructions to trigger physical actuators, or read data off web services! More specific examples will be shared in class.

Further notes

The use of MQTT implies that you MUST have an MQTT data broker running on a server somewhere. This means your mote will rely on a seperate server to publish and/or receive data. While this is normally not a bad thing (a Raspberry Pi running Mosquitto on a local network can easily be a server), projects that are meant to run independently, i.e. off the Internet and/or electrical grid, will probably have challenges working with any networked data protocol.