Friday, November 24, 2017

Publishing Sensor Data to MQTT

When publishing sensor data to MQTT, two question to ask are: how often is the data published?, and with what distribution?

Here are two examples:

  • Publish temperature, humidity, and barometric pressure from one sensor every second
  • Publish and verify RFID card swipes that employees use when entering a building.

In the first example, we have a good idea of the frequency of the data as well as how that data is distributed: data is sent once every second and it is uniformly distributed - in essence, it is time series data. In the second example, the frequency of data transmission depends on the time of day. Just before the business day starts, there will be a large number of RFID card swipes as the employees "clock in", then for the rest of the day the number of RFID events will be far lower.

The answers to those two questions will determine various aspects of the backend IoT architecture: the database that should be used to store and otherwise interact with this sensor data, the "analytics pipeline" (my term, ©2017, Patriot Geek) used to analyze and visualize that data, etc.

For this blog post, we will publish time series data using our old friends, the BME280 and the ESP8266! Please see this earlier blog post for wiring and importing the proper libraries:

http://patriot-geek.blogspot.com/2017/03/nodemcu-and-digital-sensors.html

Temperature, humidity, and barometric pressure will be published once every second to a MQTT broker. Compared to industrial IoT applications, this is not a lot of data! We can thus use a database like MongoDB or even MySQL for storage. The analytics pipeline will be discussed in a later post.

The next question to ask is: how should that data be formatted? One solution is to use a JSON format: 

{
    "temperature": 71.55,
    "humidity": 41.46,
    "barometric-pressure": 29.73
}

An alternative is to use a CSV format like this: 

71.55,41.46,29.73

The JSON format contains a human-readable description of those three values, whereas the CSV format contains considerably fewer characters. Since MQTT is all about short messages, we will use the CSV format. This will also serve us well should we want to use LoRa to send the same information. If for any reason we want to reformat the CSV into JSON, we can do so later.

Here is the code that will have our ESP8266 publish this data:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
/**
 * BME280-MQTT.js
 *
 * By: Mike Klepper
 * Date: 24 November 2017
 *
 * This program demonstrates how to publish to MQTT
 *
 * See blog post on patriot-geek.blogspot.com
 * for instructions.
 */


#include "ESP8266WiFi.h"
#include "PubSubClient.h"
#include "Wire.h"
#include "Adafruit_Sensor.h"
#include "Adafruit_BME280.h"

WiFiClient wifiClient;
PubSubClient mqttClient(wifiClient);
Adafruit_BME280 bme;

const char* SSID                  = "**********";
const char* PASSWORD              = "**********";

const char* MQTT_BROKER           = "192.168.1.11";
const int   MQTT_PORT             = 1883;
const char* MQTT_TOPIC            = "home/living-room";
const char* MQTT_CLIENT_NAME      = "ESP8266Client";

const int   BME280_DELAY          = 3000;
const int   RECHECK_INTERVAL      = 500;
const int   PUBLISH_INTERVAL      = 1000;

long        lastReconnectAttempt  = 0;
long        lastPublishAttempt    = 0;


void setup() 
{
  Serial.begin(115200);
  delay(10);

  setupWifi();
  setupBME280();
  setupMQTT();
}

void loop() 
{
  long now = millis();
  
  if(!mqttClient.connected()) 
  {
    if(now - lastReconnectAttempt > RECHECK_INTERVAL) 
    {
      lastReconnectAttempt = now;
      mqttClient.connect(MQTT_CLIENT_NAME);

      if(mqttClient.connected())
      {
        // Resubscribe to any topics, if necessary
        // This is also a good place to publish an error to a separate topic!
      }
    }
  }
  else
  {
    if(now - lastPublishAttempt > PUBLISH_INTERVAL)
    {
      lastPublishAttempt = now;
      readAndPublishData();
      mqttClient.loop();
    }
  }
}


void setupWifi()
{
  Serial.println("");
  Serial.print("Connecting to ");
  Serial.print(SSID);

  WiFi.begin(SSID, PASSWORD);

  while(WiFi.status() != WL_CONNECTED) 
  {
    delay(RECHECK_INTERVAL);
    Serial.print(".");
    yield();
  }

  Serial.println("");
  Serial.println("WiFi connected");
  Serial.print("IP address: ");
  Serial.println(WiFi.localIP());
}

void setupBME280()
{
  Serial.println("");
  Serial.print("Setting up BME280");

  while(!bme.begin())
  {
    delay(RECHECK_INTERVAL);
    Serial.print(".");
    yield();
  }
  
  Serial.println("");
  Serial.println("BME280 found!");
  delay(BME280_DELAY);
}

void setupMQTT()
{
  mqttClient.setServer(MQTT_BROKER, MQTT_PORT);
  
  Serial.println("");
  Serial.print("Connecting to MQTT");
  
  while(!mqttClient.connected())
  {
    Serial.print(".");
    mqttClient.connect(MQTT_CLIENT_NAME);
    delay(RECHECK_INTERVAL);
  }
  
  Serial.println("");
  Serial.println("Connected!");
  Serial.println("");
}

void readAndPublishData()
{
  if(mqttClient.connected())
  {
    float tempC = bme.readTemperature();
    float tempF = 9.0/5.0 * tempC + 32.0;
    float humidity = bme.readHumidity();
    float pressurePascals = bme.readPressure();
    float pressureInchesOfMercury = 0.000295299830714 * pressurePascals;
  
    String msg = String(tempF) + "," + String(humidity) + "," + String(pressureInchesOfMercury);
    char msgAsCharAway[msg.length()];
    msg.toCharArray(msgAsCharAway, msg.length());
  
    Serial.print(msg);
    mqttClient.publish(MQTT_TOPIC, msgAsCharAway);
    Serial.println(" - sent");
  }
}

Here's a walkthrough of the code. The functions for initializing the wifi and the BME280 are the same as in previous posts; the new stuff is how we publish to MQTT, and how we time those publications.

14 - 18 The various libraries we'll be using
24 - 25 Name and password for wifi network
45 - 47 Connect to wifi, start the BME280, and connect to MQTT broker
120 Initialize the MQTT client, specifying the broker's IP address and port
125 While we're not connected...
128 ...attempt to connect
139 If the MQTT client is connected...
141 - 145 ...read data from the BME280 and convert to British units...
147 - 149 ...format the data as CSV, and prepare it for publication...
152 ...and publish it to the broker!
50 - 77 This loop function is based on the non-blocking reconnect example from the PubSubClient library
52 ...get the number of milliseconds since this sketch started running...
54 If we're not connected to the broker...
56 ...check to see how long it has been since we last tried reconnecting; if it has been more than a specified time...
58 ...reset the timer...
59 ...and try to reconnect
61 If we were successful...
63 - 64 ...perform any housekeeping activities, like reporting an error
68 Otherwise we're connected...
70 ...check to see how long it has been since we last published data; if sufficient time has passed...
72 ...reset the timer...
73 ...publish the data...
74 ...and process any incoming messages.

To test the application, first open a terminal window and start MQTT broker with the command:

mosquitto

Open another terminal window and start a MQTT client and subscribe to the topic home/living-room:

mosquitto_sub -h localhost -t home/living-room

Finally, watch the incoming data!

Some notes about this application:

  1. Data is transmitted in an insecure manner!
  2. A timestamp is not included in the data.
These two issues will be addressed in a later blog post.

Posted by at 7 comments:
Labels: , , , , , ,

Sunday, November 5, 2017

MicroPython on the ESP8266 - Part 2

In this entry we access MicroPython on the ESP8266 using WebREPL instead of serial REPL. WebREPL is a web interface that will allow us to run commands like we did with the serial REPL. In addition, it will allow us to move files from our computer to the ESP8266 and back!

In this short post, we will:

  1. Setup WebREPL
  2. Connect to WebREPL
  3. Upload Files to the ESP8266
  4. Download Files from the ESP8266
  5. Write a Program to Run when ESP8266 Boots


Download the WebREPL Client and Set the ESP8266 to Use It
WebREPL will be run from the desktop or notebook computer, it is not hosted on the ESP8266 itself like we did in a previous tutorial. Download and unzip the WebREPL client from Github at https://github.com/micropython/webrepl. We will see how to use it momentarily.

From the serial REPL, run the following command:

import webrepl_setup

Set a password like ABC123, then accept option to restart.



Connect to WebREPL
When the ESP8266 restarts, it creates its own network - it will have a name like MicroPython-178fee. Connect to that network. The password will be micropythoN with a capital-N

Open the HTML file called webrepl-master/webrepl.html in a browser. In the top-left corner, click the "Connect" to the ESP8266. Enter the password chosen above (ABC123) and you should be at the >>> prompt!

As you try entering commands into WebREPL, you'll see them echoed in the serial REPL - if it is still open.



Upload Files to the ESP8266
Now let us test the file upload feature...

  1. On your desktop or notebook computer, create a file called Test2.py that contains one line:
    print("Hello, from Test2.py!")
    Save the file someplace convenient.
  2. Click the "Choose File" button, choose the Test2.py file, and click "Send to device button"
  3. To make sure it arrived,
    import os
    os.listdir()
    and we see that the Test2.py was uploaded to the root directory! We run it:
    exec(open('Test2.py').read())
    and get the expected result.



Download Files from the ESP8266

Notice that there is an additional file... webrepl_cfg.py. Let's transfer it from the ESP8266 to our computer.

In the WebREPL, type that name in the text input next to "Get a file" and click "Get from device" button. The file will be downloaded. Open it with a text editor or an IDE and it will read:

PASS = 'ABC123'


Write a Program to Run when ESP8266 Boots
As discussed in Part 1, programs that we create will not automatically run when the ESP8266 boots, unless that file is named /main.py. We can use the file upload feature of WebREPL to create this file.

On your desktop or notebook computer, create a file called main.py and enter the following code:

1
2
3
4
5
6
7
8
9
def toggle(p):
    p.value(not p.value())

import time
import machine
pin = machine.Pin(2, machine.Pin.OUT)
while True:
    toggle(pin)
    time.sleep_ms(500)

This program is the same as was used in Part 1, except now it is a stored in a file. Upload the file to the ESP8266 using WebREPL, then reboot (for example by pressing the RST button on the ESP8266). The LED will start blinking!

So, there it is - how to move files between your primary computer and an ESP8266 running MicroPython.

Posted by at No comments:
Labels: , ,

MicroPython on the ESP8266 - Part 1

The ESP8266 can be programmed in several different languages - great! Besides the C++-ish language we've been using, the ESP8266 can be coded in Python, BASIC, Lua, JavaScript, and others. This post demonstrates installing MicroPython on a NodeMCU, getting to the REPL, and trying some different commands.

MicroPython is an interpreted language, so programs will not run as fast as they would when written in a compiled language. With Arduino, there is not only an IDE, but also a large collection of libraries. With MicroPython, there is no solid IDE and there aren't nearly as many libraries. However, MicroPython for the ESP8266 does support WiFi, GPIO, I2C, SPI, etc. Further, the documentation is excellent, and there are excellent tutorials available. The post is based on that documentation.

Before we start, there are several things to know. First, since Python is an interpreted language, a common way to interact with MicroPython is through a specific shell called REPL: read-evaluate-print-loop. This is an interactive environment that allows us to run single lines of code, or blocks of code. Second, Python does not use braces to delimit code blocks like C, C++, JavaScript, etc. Instead, it uses the tab character to indicate lines of code are within a block.

In this post, we will do the following:

  1. Get and Deploy the MicroPython Firmware
  2. Use the Serial Port to Get to the REPL
  3. Examine the File System
  4. Run a Simple Python Program Stored in the File System


Get and Deploy the Firmware
In order to run MicroPython, a special firmware must first be deployed to the ESP8266. We also need a program to actually perform the flashing.

Step 1: Download the firmware from this URL: http://micropython.org/download#esp8266 The file I chose was esp8266-20171101-v1.9.3.bin and it was saved in the Downloads folder. There is no need to upzip this file - instead it will be flashed to the ESP8266.

The remainder of the steps will be performed in the Mac OS X Terminal, and it assumes that Python is installed.

Step 2: Install esptool.py, which actually performs the flash. Open a Terminal window and run:

pip install esptool

Note: if you already have esptool.py installed, you can upgrade to the latest version using:
pip install esptool --upgrade

Step 3: Get the port the ESP8266 is attached to by running this command from the terminal:

ls -l /dev/tty.*
Mine is on /dev/tty.SLAB_USBtoUART

Step 4: Erase the flash memory:

esptool.py --port /dev/tty.SLAB_USBtoUART erase_flash

Step 5: Finally, deploy the new firmware:

esptool.py --port /dev/tty.SLAB_USBtoUART --baud 460800 write_flash --flash_size=detect 0 Downloads/esp8266-20171101-v1.9.3.bin


Getting to the REPL using the Serial Port
There are two ways to get to the REPL: wired using the serial port, or wireless using webREPL. We cover only the wired approach here.

On Mac OSX, use the following:

screen /dev/tty.SLAB_USBtoUART 115200
Then press Enter a few times until you get the ">>>" REPL prompt.

One command to try is:

print("Hello, world!")

Another thing to try is simple arithmetic:

1+2

To find out the version of MicroPython running on the ESP8266, enter the following two commands:

import sys
print(sys.version)
The version I have is 3.4.0.

An interesting series of commands to try is this:

import machine
pin = machine.Pin(2, machine.Pin.OUT)
pin.value()
pin.off()
pin.value()
pin.on()
pin.value()

These commands reveal that the built-in LED on the NodeMCU is "wired backwards", but that it can indeed be controlled via REPL.

Functions are defined as follows:

def toggle(p):
    p.value(not p.value())

Press Enter several times to finish the function definition.

Here is a simple program to call this toggle function once every 500 ms:

import time
import machine
pin = machine.Pin(2, machine.Pin.OUT)
while True:
    toggle(pin)
    time.sleep_ms(500)

Press Enter several times, and the LED will start blinking!

To exit this loop, press Control-C, which will generate a KeyboardInterrupt.


The File System
When the firmware is installed, a small filesystem is created. To start using it,

import os

To get the current directory,

os.getcwd()
which will return '/' - the root.

To get a list of all files and subdirectories in the current directory, use\

os.listdir()
which returns
['boot.py']

We will discuss this file shortly.

To make a directory:

os.mkdir("MyDirectory")

Move into that directory:

os.chdir("MyDirectory")

Now let's make a file while we're in that directory:

f = open("MyFile.txt", "w")
f.write("Hello, world!")
f.close()
Just after the write command is executed, the number "13" is displayed, indicating that 13 characters were written to MyFile.txt.

Now if we list directory contents using os.listdir(), we see:

['MyFile.txt']

Now open the file, read the content, then close it:

f = open("MyFile.txt")
f.read()
f.close()
and 'Hello, world!' is displayed.

Files are removed as follows:

os.remove('MyFile.txt')

And os.listdir() returns []. We move up one level with

os.chdir('..')

Remove the directory with

os.rmdir('MyDirectory')

Let's examine boot.py using

f = open('boot.py')
f.read()
f.close()
Here's the result:
# This file is executed on every boot (including wake-boot from deepsleep)\n#import esp\n#esp.osdebug(None)\nimport gc\n#import webrepl\n#webrepl.start()\ngc.collect()\n

Replacing the '\n' with newlines shows:

# This file is executed on every boot (including wake-boot from deepsleep)
#import esp
#esp.osdebug(None)
import gc
#import webrepl
#webrepl.start()
gc.collect()

The majority of the lines in this file have been commented-out, but notice the reference to WebREPL, we'll save that for a later blog post!


Saving and Running a Simple Python Program
Now we create a simple one-line MicroPython program and execute it!

f = open("Test.py", "w")
f.write('print("Hello, world!")')
f.close()

To run this program, do the following:

exec(open("Test.py").read())

We get the expected output. Note that this method of running Python programs does not necessarily make global and local variables available to Test.py.

Finally, to exit: press Control-a then Control-\ and in response to the "Really quit and kill all your windows [y/n]" prompt, press y.

If the ESP8266 is rebooted, Test.py will NOT run. As it goes, the ESP8266 running MicroPython will first execute code in boot.py, then it will execute main.py, if that file exists. We will demonstrate this in the next blog post.

So now we know how to manually create a Python program, but it is terribly inconvenient. This points to the problem of how to conveniently enter and run MicroPython programs on our ESP8266. We will (partially) solve that problem in the next post!

Posted by at No comments:
Labels: , ,
Subscribe to: Posts (Atom)