Forays into Amazon IOT

I have built many electronic devices over the years that were meant to package sensors for home automation into more refined packages – rather than having breadboards lying around all over the house. While building the data collection devices was never a problem, what always stopped me from integrating them into a neat and usable system was the frontend UI and the backend DB. Clearly building electronics was the fun part of the project and not the oftentimes tedious integration part.

So instead of baking my own solution for data storage and UI, I decided to give AWS IoT a shot. For this project, I picked a random sensor out of my sensor collection, hooked it up to an ESP32 Wifi-enabled microcontroller, and will connect it to my AWS account. Many tutorials have been written about this before, so here I’ll just link to the relevant ones instead of replicating the text and images again. I’ll focus on pointing out the gotchas and non-intuitive aspects of the process instead.

Before we start I want to point out a thought about security with a quote from the tutorial I will follow in this post.

Security of connected devices is of predominant concern. The widely used Transport Layer Security (TLS) version 1.2 is the de-facto standard used on the web for secure connections including banking and financial institute.There are very few micro-controllers that supports TLS 1.2 and ESP32 is one of them. Good news is that AWS IoT Core maintains TLS 1.2 and above, thus making the security robust .

Hackermoon article (https://hackernoon.com/cloud-home-automation-series-part-1-connect-esp32-to-aws-iot-with-arduino-code-lkhp36vx)

For this tutorial, I’ll use a BME680 sensor (gas, pressure, humidity, temperature) and follow this Adafruit tutorial for initial sensor testing. Besides the specific sensor, here I’ll follow the tutorial on Hackermoon with code snippets borrowed from the Hornbill collection of examples and the aws-samples collection of examples.

Basic setup

AWS IoT supports multiple protocols (MQTT, HTTPS, and LoRaWAN). Since an MQTT client is available for ESP32 we will use the MQTT protocol here. So the basic structure of the program running on an ESP32 sensor node will be the following:

  • Authentication (Wifi network, AWS client)
  • Sensor setup (BME680)
  • Repeated measurements, packaging of data, and publishing of messages

To make these sensor nodes more maintainable, the following capabilities will also be desirable.

  • Over the air (OTA) updates of the code running on the ESP32
  • Configure the sensor node via MQTT messages

The rest of this page assumes that the reader has set up the Arduino IDE, downloaded the ESP32 core, and installed the necessary libraries (see the “Resources” section below). While the Hackermoon tutorial relies on the Hornbill library, in this article, I’ll only use a basic MQTT client and the ArduinoJson library, which is sufficient to communicate with AWS IoT. The Hornbill library implements more powerful functionality but is written in C, which makes it a little less convenient to use.

Wiring example using an ESP32 controller and a BME680 sensor (credit: Adafruit)

A basic pub/sub client

The code snippets below walk through the basic program components outlined above.

#include "secrets.h"
#include <WiFiClientSecure.h>
#include <MQTTClient.h>
#include <ArduinoJson.h>
#include "WiFi.h"

These included header are all that’s required – a Wifi client, an MQTT client, and the ArduinoJSON library. The “secrets.h” file stores all credentials and certificates to connect to your local Wifi and your AWS IoT account. I’ll go into more detail how the AWS certificates are obtained later.

WiFiClientSecure net = WiFiClientSecure();

void connectToAWS() {
  // Connect to local Wifi.
  WiFi.mode(WIFI_STA);
  WiFi.begin(WIFI_SSID, WIFI_PASSWORD);

  // Wait until the Wifi client is connected.
  while (WiFi.status() != WL_CONNECTED){
    delay(500);
    Serial.print(".");
  }

  // Configure WiFiClientSecure to use the AWS IoT device credentials
  net.setCACert(AWS_CERT_CA);
  net.setCertificate(AWS_CERT_CRT);
  net.setPrivateKey(AWS_CERT_PRIVATE);
}

The above snippet sets up a secure Wifi client and sets the required AWS certificates. Note that all required credentials have to be defined in the “secrets.h” file.

// The MQTT topics that this device should publish/subscribe to.
#define AWS_IOT_PUBLISH_TOPIC   "esp32/pub"
#define AWS_IOT_SUBSCRIBE_TOPIC "esp32/sub"

// Set up the MQTT client with a buffer size of 256 bytes.
MQTTClient client = MQTTClient(256);

void setupMQTTClient() {
  // Connect to the MQTT broker on the AWS endpoint we defined earlier.
  client.begin(AWS_IOT_ENDPOINT, 8883, net);

  // Create a message handler.
  client.onMessage(messageHandler);

  while (!client.connect(THINGNAME)) {
    Serial.print(".");
    delay(100);
  }

  if(!client.connected()){
    Serial.println("AWS IoT Timeout!");
    return;
  }

  // Subscribe to a topic
  client.subscribe(AWS_IOT_SUBSCRIBE_TOPIC);
}

The above snippet sets up the MQTT client and connects it to the AWS’s MQTT endpoint. Two things are worth noting here:

  • The buffer size for the MQTT client is 256 bytes, which is what will limit the maximum message size to send. The ESP32 has 520 KB of RAM so one can easily increase the MQTT buffer size without risking to run out of memory.
  • The “messageHandler” function needs to be defined in the Arduino sketch such that incoming messages can be processed. A more advanced (and nicely packaged C++) implementation would implement the message handler in a C++ class and use a trampoline function to call it. I’ll leave that for a later post.
void publishMessage(){
  // Create a new JSON document.
  StaticJsonDocument<200> doc;

  // Populate the JSON document with sensor data.
  doc["time"] = millis();
  doc["temperature"] = 25.7;

  // Serialize the JSON document to a string.
  char jsonBuffer[512];
  serializeJson(doc, jsonBuffer); // print to client

  // Publish the serialized JSON string via MQTT.
  client.publish(AWS_IOT_PUBLISH_TOPIC, jsonBuffer);
}

void messageHandler(String &topic, String &payload) {
  Serial.println("incoming: " + topic + " - " + payload);
}

The above snippet packages the sensor data (in this snippet just dummy data) into a JSON document, serializes the document to a string and publishes the string via MQTT. This is the string that will be received at the AWS IoT backend. As a message handler I’ll just use a stub right now that only prints the received data rather than acting on them.

void setup() {
  Serial.begin(115200);
  connectToAWS();
  setupMQTTClient();
}

void loop() {
  publishMessage();
  client.loop();
  delay(1000);
}

Tying it all together are the two functions one is used to seeing in an Arduino sketch – setup() and loop(). This particular loop publishes a sensor data MQTT message once per second.

AWS Integration

A few steps are necessary to connect a device to Amazon’s AWS IoT service.

Registering the device with AWS IoT includes generating certificates for the device and completing the secrets.h file. When generating certificates (as outlined here), one can download 5 files from AWS:

  • AmazonRootCA1.pem
  • AmazonRootCA3.pem
  • bf1ce2f594906d62c02c5b8f3f657bbfd12d025711eb592320e58a062b9cc99f-certificate.pem.crt
  • bf1ce2f594906d62c02c5b8f3f657bbfd12d025711eb592320e58a062b9cc99f-private.pem.key
  • bf1ce2f594906d62c02c5b8f3f657bbfd12d025711eb592320e58a062b9cc99f-public.pem.key

For the secrets.h file, only the AmazonRootCA1.pem (AWS root certificate), the bf1ce2f594906d62c02c5b8f3f657bbfd12d025711eb592320e58a062b9cc99f-certificate.pem.crt (device certificate) and the device’s private key are required.

The AWS IoT endpoint can be found in AWS IoT Core main dashboard > Settings (Second last option in the Sidebar) > Device data endpoint > Endpoint.

The data will be available and can be seen via the AWS IoT Console > Test (in the sidebar). One has to subscribe to the topic on which the ESP publishes its data (in our example “esp32/pub“) and the data will begin showing up in the console.

Resources

AWS IoT Integration

Examples

BME680 Sensor Tutorials

Required libraries