Sunday, November 26, 2017

Node-RED: Displaying MQTT Data in a Dashboard

Last time on Patriot Geek, our intrepid blogger built a MQTT client in Node-RED. Can we now present the incoming data in a useful and attractive manner? Stay tuned!

To visualize the data being sent by the BME280, we will use the "node-red-dashboard" module, which adds various nodes that generate interactive dashboards. Here's the dashboard we'll be building:

This post is divided into the following sections:

  1. Install the dashboard extension
  2. Understand dashboard nomenclature
  3. Design Overview
  4. The MQTT Subscriber Flow
  5. Create Dashboard Tabs and Groups
  6. The Dashboard Flow
  7. Test the Dashboard
  8. Conclusions and Source Code


Installation
The Node-RED IDE can be extended in various ways, from adding a single new node to adding a family of new nodes as well as adding significantly to the user interface. That's what the Node-RED Dashboard module does: it adds 16 new nodes, a new sidecar tab, and makes other changes to the IDE.

To install, click the Menu Button and choose "Manage palette". In the property sheet that opens, click the "Install" tab. Do a search for "Dashboard", and scroll until you see the "node-red-dashboard" package, and click the "install" button, then click the red "Done" button. Note: there is no need to install the "node-red-dashboard-es" package.


Dashboard Nomenclature
The use of the word "Dashboard" in Node-RED is kind of confusing, because it is used in several ways:

  • The module we just installed
  • The sidebar tab
  • The section of the palette, and the nodes inside it
  • The UIs that are built with those nodes
In addition, there are two specific terms used to described the parts of the dashboards we build: "tab" and "group".

A tab is a top-level part of the dashboard, and every dashboard must have one. When a dashboard has more than one, a menu will be shown on the left side of the title bar. Each tab has one or more "groups", which are... groups of UI controls.

Here's a tab with three groups inside it:

Here's the menu showing the two available tabs:


Design Overview
We will implement the MQTT data dashboard using two workspace tabs: one called "MQTT Subscriber" which will be a flow similar to that in the previous blog post; the other called "MQTT Dashboard" which will be for the dashboard nodes.

This design approach follows the Model-View-Controller (MVC) design pattern. The dashboard user interface is a view, and the MQTT client is essentially a controller - it processes incoming events and passes data to the view.

How do we send messages from the controller tab over to the view tab? There are a pair of nodes for doing this: the Link Out node and the Link In node. These are found under the "output" and "input" sections of the palette, respectively.


The MQTT Subscriber Flow
Create two new tabs by pressing the New Tab button. Double click on the first, and set the name to be "MQTT Subscriber". Double click the second and name it "MQTT Dashboard". Click back on the "MQTT Subscriber" tab.

Again, the MQTT Subscriber flow is a modification of the one in the previous post - all we'll be doing is adding a Link Out node and removing the two Debug nodes. Here's what the final result will look like:

After deleting the two Debug nodes, add a new Link Out node, which is found in the "Input" section of the palette. Wire it as shown. Double click it and set the name to be "To Dashboard" and click "Done". For quick reference, here are the properties of all four nodes in this flow:

Node Type Property Value
MQTT In
Server: 192.168.1.11:1883
Topic: home/living-room
QoS: 0
CSV
Columns: temperature,humidity,barometric-pressure
Name: Parse CSV into JSON
Function
Name: Add Time Info
Function:
1
2
3
4
var now = new Date();
msg.payload["timestamp"] = now.getTime();
msg.payload["fomratted-time"] = now.toUTCString();
return msg;
Link Out
Name: To Dashboard

Create the Dashboard's Tabs and Groups
Before drawing the view flow, set up the tabs and groups for the dashboard. We will have only one tab called "Living Room Data". Inside that tab will be three groups, called "Temperature", "Humidity", and "Barometric Pressure". Here's how this will look in the Layout subtab when we're done:

Click the dashboard sidebar tab, then choose "Layout" sub tab, and use the "+ tab" button to create the tab. Set the name to be "Living Room Data". This determines the text shown in the title bar of the dashboard.

Then, hover-over the tab and click the "+ group" button three times. Hover-over each and click the "edit" button and set the names to be "Temperature", "Humidity", and "Barometric Pressure". Make sure they're all under the "Living Room Data" tab.

Click the "Site" subtab, and set the web page's title to "MQTT Data". This will be shown in the browser title bar.

While we're here, click the Theme subtab, and set the style to be "Dark". Why? Because Dark UIs Matter:


The Dashboard Flow
Click the "MQTT Dashboard" tab so we can build the dashboard flow. On it, drop one Link In node, one Debug node, three Change nodes, two Chart nodes, one Gauge node, and three Text nodes. The Chart, Gauge, and Text nodes and found in the "dashboard" section of the palette, and the Link In node is found under the "Input" section. Wire them up as follows:

We want the Link In node to receive data from the Link Out node on the other tab. Double click the Link In node, and the name we set to the Link Out node ("To Dashboard") is listed there as a checkbox. Click that checkbox. Also, set the node's name to be "From Subscriber":

Click the Link In node. When it has been successfully connected to the Link Out in the other tab, it is displayed like this, but only when selected:

Double click the Debug node, and set the out to be "complete msg object"

Now, the Chart and Gauge nodes expect their input to be in a particular format: the input must be numeric and the incoming msg.payload must be set equal that numeric value. Trouble is, the data coming from the MQTT subscriber tab (via the Link In node) is a JSON object like this:

{
    "temperature": 71.55,
    "humidity": 41.46,
    "barometric-pressure": 29.73
}

That's the purpose of the three Change nodes, to extract the numeric values from msg.payload, and replace msg.payload with those values.

Double click the first Change node, and set the properties as follows:

Update the second Change node so that it uses msg.payload.humidity. Give it the name "Extract Humidity", and set msg.payload to be msg.payload.humidity.

Name the third Change node to be "Extract Pressure", and set msg.payload to be msg.payload["barometric-pressure"]. Note the different syntax here - "barometric-pressure" is a perfectly fine JSON field name, but it is an invalid JavaScript field name!

Notice the dichotomy between visual programming languages and most traditional languages: altering msg.payload in the first Change node only has consequences for the "downstream" nodes!

The Gauge node and the Chart nodes are now getting the right data!

Now we configure the charts and the gauge. Double click the Chart node that's attached to the "Extract Temperature" Change node, and set the properties as follows:

The Chart attached to the "Extract Pressure" Change node will be set in a similar fashion, except that it will be set to appear in the "Barometric Pressure [Living Room Data]" group, and the name will be "Pressure Chart".

For the Gauge node, set the properties as shown:

Note that range for the Humidity Gauge is set to be from 0 to 100. This is because humidity runs from 0% to 100%. It is also possible to preconfigure the y-axis range on the Chart nodes but the chart will draw values outside that range as a flat line.

Our last task is to configure the Text nodes. Set their properties as follows:

Node Property Value
Text node for Temperature
Group: Temperature [Living Room Data]
Label: Temperature:
Value format: {{msg.payload}} °F
Layout: "row-left"
Name: Temperature Text
Text node for Humidity
Group: Humidity [Living Room Data]
Label: Humidity:
Value format: {{msg.payload}} %
Layout: "row-left"
Name: Humidity Text
Text node for Pressure
Group: Barometric Pressure [Living Room Data]
Label: Barometric Pressure:
Value format: {{msg.payload}} InHg
Layout: "row-left"
Name: Pressure Text

Here's what the flow will now look like:


Test the Dashboard
To see the result of all this, be sure that the MQTT broker is running, and open a new browser window and go to:

http://127.0.0.1:1880/ui
Data will start appearing in the gauge and charts. If there's not a lot of variability in the data, try (get this) breathing on the BME280! Here's what the dashboard should look like:
 

It works on an iPhone 5c, too:


Conclusions and Source Code
This post demonstrated how to make a dashboard that displays MQTT data coming from the BME280 and ESP8266. Something else demonstrated was how to use two separate tabs for the application, one for the controller, the other for the user interface. This use of the Model-View-Controller (MVC) design pattern adds some clarity to our flows.

The source code for the modified MQTT Subscriber flow is as follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
[
    {
        "id": "fdb325cd.8b5748",
        "type": "mqtt in",
        "z": "553bcc1e.c97b54",
        "name": "",
        "topic": "home/living-room",
        "qos": "0",
        "broker": "a481358c.031c18",
        "x": 128,
        "y": 218,
        "wires": [
            [
                "3c175b0f.97a134"
            ]
        ]
    },
    {
        "id": "3c175b0f.97a134",
        "type": "csv",
        "z": "553bcc1e.c97b54",
        "name": "Parse CSV into JSON",
        "sep": ",",
        "hdrin": "",
        "hdrout": "",
        "multi": "one",
        "ret": "\\n",
        "temp": "temperature,humidity,barometric-pressure",
        "x": 340,
        "y": 218,
        "wires": [
            [
                "eae5505.8945fb"
            ]
        ]
    },
    {
        "id": "eae5505.8945fb",
        "type": "function",
        "z": "553bcc1e.c97b54",
        "name": "Add Time Info",
        "func": "var now = new Date();\nmsg.payload[\"timestamp\"] = now.getTime();\nmsg.payload[\"fomratted-time\"] = now.toUTCString();\nreturn msg;",
        "outputs": 1,
        "noerr": 0,
        "x": 554,
        "y": 218,
        "wires": [
            [
                "3a476f0b.d6bea"
            ]
        ]
    },
    {
        "id": "3a476f0b.d6bea",
        "type": "link out",
        "z": "553bcc1e.c97b54",
        "name": "To Dashboard",
        "links": [
            "492fdf69.9cb38",
            "4e7d2e3c.c3427"
        ],
        "x": 684,
        "y": 218,
        "wires": []
    },
    {
        "id": "a481358c.031c18",
        "type": "mqtt-broker",
        "z": "",
        "broker": "192.168.1.11",
        "port": "1883",
        "clientid": "",
        "usetls": false,
        "compatmode": true,
        "keepalive": "60",
        "cleansession": true,
        "willTopic": "",
        "willQos": "0",
        "willPayload": "",
        "birthTopic": "",
        "birthQos": "0",
        "birthPayload": ""
    }
]

Notice how there is an extra node in the JSON that is not shown in the workspace - this is where info about the MQTT broker is stored. Nodes like this are called "configuration nodes".

And here's the source code for the Dashboard flow:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
[
    {
        "id": "4e7d2e3c.c3427",
        "type": "link in",
        "z": "6fc2e4b5.acb35c",
        "name": "",
        "links": [
            "3a476f0b.d6bea"
        ],
        "x": 55,
        "y": 140,
        "wires": [
            [
                "43a49c20.b2b534",
                "2d87fae4.e4c7b6",
                "ec5f8a6c.7cef28",
                "432ea3c2.6ca94c"
            ]
        ]
    },
    {
        "id": "43a49c20.b2b534",
        "type": "debug",
        "z": "6fc2e4b5.acb35c",
        "name": "",
        "active": true,
        "console": "false",
        "complete": "true",
        "x": 210,
        "y": 40,
        "wires": []
    },
    {
        "id": "2d87fae4.e4c7b6",
        "type": "change",
        "z": "6fc2e4b5.acb35c",
        "name": "Extract Temperature",
        "rules": [
            {
                "t": "set",
                "p": "payload",
                "pt": "msg",
                "to": "payload.temperature",
                "tot": "msg"
            }
        ],
        "action": "",
        "property": "",
        "from": "",
        "to": "",
        "reg": false,
        "x": 260,
        "y": 100,
        "wires": [
            [
                "5294196b.20d158",
                "d925dd10.68321"
            ]
        ]
    },
    {
        "id": "ec5f8a6c.7cef28",
        "type": "change",
        "z": "6fc2e4b5.acb35c",
        "name": "Extract Humidity",
        "rules": [
            {
                "t": "set",
                "p": "payload",
                "pt": "msg",
                "to": "payload.humidity",
                "tot": "msg"
            }
        ],
        "action": "",
        "property": "",
        "from": "",
        "to": "",
        "reg": false,
        "x": 250,
        "y": 200,
        "wires": [
            [
                "f65a9085.ae81",
                "78782f2d.880f1"
            ]
        ]
    },
    {
        "id": "432ea3c2.6ca94c",
        "type": "change",
        "z": "6fc2e4b5.acb35c",
        "name": "Extract Pressure",
        "rules": [
            {
                "t": "set",
                "p": "payload",
                "pt": "msg",
                "to": "payload[\"barometric-pressure\"]",
                "tot": "msg"
            }
        ],
        "action": "",
        "property": "",
        "from": "",
        "to": "",
        "reg": false,
        "x": 240,
        "y": 300,
        "wires": [
            [
                "ee70b8b0.828fc8",
                "c65aff2.f086b"
            ]
        ]
    },
    {
        "id": "5294196b.20d158",
        "type": "ui_chart",
        "z": "6fc2e4b5.acb35c",
        "name": "Temperature Chart",
        "group": "637855dc.2229bc",
        "order": 0,
        "width": 0,
        "height": 0,
        "label": "",
        "chartType": "line",
        "legend": "false",
        "xformat": "HH:mm:ss",
        "interpolate": "linear",
        "nodata": "",
        "dot": false,
        "ymin": "",
        "ymax": "",
        "removeOlder": "5",
        "removeOlderPoints": "",
        "removeOlderUnit": "60",
        "cutout": 0,
        "useOneColor": false,
        "colors": [
            "#1f77b4",
            "#aec7e8",
            "#ff7f0e",
            "#2ca02c",
            "#98df8a",
            "#d62728",
            "#ff9896",
            "#9467bd",
            "#c5b0d5"
        ],
        "useOldStyle": false,
        "x": 530,
        "y": 80,
        "wires": [
            [],
            []
        ]
    },
    {
        "id": "ee70b8b0.828fc8",
        "type": "ui_chart",
        "z": "6fc2e4b5.acb35c",
        "name": "Pressure Chart",
        "group": "9728196c.705ff8",
        "order": 0,
        "width": 0,
        "height": 0,
        "label": "",
        "chartType": "line",
        "legend": "false",
        "xformat": "HH:mm:ss",
        "interpolate": "linear",
        "nodata": "",
        "dot": false,
        "ymin": "",
        "ymax": "",
        "removeOlder": "5",
        "removeOlderPoints": "",
        "removeOlderUnit": "60",
        "cutout": 0,
        "useOneColor": false,
        "colors": [
            "#1f77b4",
            "#aec7e8",
            "#ff7f0e",
            "#2ca02c",
            "#98df8a",
            "#d62728",
            "#ff9896",
            "#9467bd",
            "#c5b0d5"
        ],
        "useOldStyle": false,
        "x": 520,
        "y": 280,
        "wires": [
            [],
            []
        ]
    },
    {
        "id": "f65a9085.ae81",
        "type": "ui_gauge",
        "z": "6fc2e4b5.acb35c",
        "name": "Humidity Gauge",
        "group": "f468b186.a264b",
        "order": 0,
        "width": 0,
        "height": 0,
        "gtype": "gage",
        "title": "Gauge",
        "label": "%",
        "format": "{{value}}",
        "min": 0,
        "max": "100",
        "colors": [
            "#00b500",
            "#e6e600",
            "#ca3838"
        ],
        "seg1": "",
        "seg2": "",
        "x": 520,
        "y": 180,
        "wires": []
    },
    {
        "id": "d925dd10.68321",
        "type": "ui_text",
        "z": "6fc2e4b5.acb35c",
        "group": "637855dc.2229bc",
        "order": 0,
        "width": 0,
        "height": 0,
        "name": "Temperature Text",
        "label": "Temperature:",
        "format": "{{msg.payload}} °F",
        "layout": "row-left",
        "x": 530,
        "y": 120,
        "wires": []
    },
    {
        "id": "78782f2d.880f1",
        "type": "ui_text",
        "z": "6fc2e4b5.acb35c",
        "group": "f468b186.a264b",
        "order": 0,
        "width": 0,
        "height": 0,
        "name": "Humidity Text",
        "label": "",
        "format": "{{msg.payload}} %",
        "layout": "row-left",
        "x": 520,
        "y": 220,
        "wires": []
    },
    {
        "id": "c65aff2.f086b",
        "type": "ui_text",
        "z": "6fc2e4b5.acb35c",
        "group": "9728196c.705ff8",
        "order": 0,
        "width": 0,
        "height": 0,
        "name": "Pressure Text",
        "label": "Barometric Pressure:",
        "format": "{{msg.payload}} InHg",
        "layout": "row-left",
        "x": 520,
        "y": 320,
        "wires": []
    },
    {
        "id": "637855dc.2229bc",
        "type": "ui_group",
        "z": "",
        "name": "Temperature",
        "tab": "d376d602.4e3538",
        "order": 1,
        "disp": true,
        "width": "6"
    },
    {
        "id": "9728196c.705ff8",
        "type": "ui_group",
        "z": "",
        "name": "Barometric Pressure",
        "tab": "d376d602.4e3538",
        "order": 3,
        "disp": true,
        "width": "6"
    },
    {
        "id": "f468b186.a264b",
        "type": "ui_group",
        "z": "",
        "name": "Humidity",
        "tab": "d376d602.4e3538",
        "order": 2,
        "disp": true,
        "width": "6"
    },
    {
        "id": "d376d602.4e3538",
        "type": "ui_tab",
        "z": "",
        "name": "Living Room Data",
        "icon": "dashboard",
        "order": 2
    }
]

Here, we have four extra configuration nodes: one for the tab and three for the groups.

Saturday, November 25, 2017

Node-RED: Subscribing to a MQTT Topic

For our next Node-RED flow, we will build a MQTT client that will subscribe to BME280 sensor data published by an ESP8266. The hardeare and code for publishing that data is described in an earlier post.

The messages published to the MQTT topic home/living-room are CSV strings containing temperature, humidity, and barometric pressure in British units. Here's an example of such a message:

71.55,41.46,29.73

Our goal for this post is to build a Node-RED flow that subscribes to that topic, and parses that data into JSON like this:

{
    "temperature": 71.55,
    "humidity": 41.46,
    "barometric-pressure": 29.73
}

For this flow, we'll be using the following three types of nodes:

MQTT In
Connects to a MQTT server and subscribes to a specified topic
CSV
Converts CSV data into JSON
Debug
Displays messages in the debug tab

These nodes will be connected as follows:

Double-click the MQTT In node and set the following properties:

Server: 192.168.1.11:1883
Topic: home/living-room
QoS: 0

Next, double-click the CSV node and set the following properties:

Columns: temperature,humidity,barometric-pressure
Separator: Comma
Name: Parse CSV into JSON

Finally, double-click each of the Debug nodes and set the following properties:

Output: complete msg object
to: debug tab

The CSV node uses the column names for the field names, and assigns values to those fields in the order of the values in the incoming CSV. The end result is set to be the message's payload. This is how the messages look in the debug tab:

Pretty-printing the top entry in the debug list shows that the msg object has the following structure:

{
  "topic": "home/living-room",
  "payload":
  {
    "temperature": 73.8,
    "humidity": 37.12,
    "barometric-pressure": 29.6
  },
  "qos": 0,
  "retain": false,
  "_msgid": "1c091ae3.456e45"
}

Great, we've subscribed to a MQTT topic and formatted the incoming CSV data as a JSON object!

Wouldn't it be nice if there were timestamps in these objects? As explained in the post where we built and coded the sensor, our little ESP8266 doesn't have a clock running on battery, nor are we getting time from a network time service. Instead, we will add the timestamp to the data once it arrives in our Node-RED flow.

To accomplish this, we will use a Function node, which will run a block of JS. Wire this Function node into the rest of the flow as follows:

Double-click on the Function node, and set the name to be Add Time Info. In the "Function" textarea, add the following code:

1
2
3
4
var now = new Date();
msg.payload["timestamp"] = now.getTime();
msg.payload["fomratted-time"] = now.toUTCString();
return msg;

All that this code does is to add two additional properties to msg.payload. The JavaScript .getDate() method returns the number of milliseconds since January 1, 1970, and the .toUTCString() formats that data into a human-readable form in the GMT timezone.

Clear the debug tab, deploy the edited flow, and examine the incoming data. Here's an example of what the final result will be, once we pretty-print it:

{
  "topic": "home/living-room",
  "payload": 
  {
    "temperature": 73.76,
    "humidity": 37.35,
    "barometric-pressure": 29.6,
    "timestamp": 1511588927869,
    "formatted-time": "Sat, 25 Nov 2017 05:48:47 GMT"
  },
  "qos": 0,
  "retain": false,
  "_msgid":"3867cbee.a2cfa4"
}

Here's the source code for this last flow:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
[
    {
        "id": "ee409dc6.f511b",
        "type": "mqtt in",
        "z": "e260f053.ca405",
        "name": "",
        "topic": "home/living-room",
        "qos": "0",
        "broker": "a481358c.031c18",
        "x": 120,
        "y": 160,
        "wires": [
            [
                "f90ccd18.b0f84",
                "b7c62375.17de5"
            ]
        ]
    },
    {
        "id": "f90ccd18.b0f84",
        "type": "debug",
        "z": "e260f053.ca405",
        "name": "",
        "active": false,
        "console": "false",
        "complete": "true",
        "x": 770,
        "y": 160,
        "wires": []
    },
    {
        "id": "b7c62375.17de5",
        "type": "csv",
        "z": "e260f053.ca405",
        "name": "Parse CSV into JSON",
        "sep": ",",
        "hdrin": "",
        "hdrout": "",
        "multi": "one",
        "ret": "\\n",
        "temp": "temperature,humidity,barometric-pressure",
        "x": 380,
        "y": 220,
        "wires": [
            [
                "5097ac3c.7d4b04"
            ]
        ]
    },
    {
        "id": "adf30d5c.857be",
        "type": "debug",
        "z": "e260f053.ca405",
        "name": "",
        "active": true,
        "console": "false",
        "complete": "true",
        "x": 770,
        "y": 220,
        "wires": []
    },
    {
        "id": "5097ac3c.7d4b04",
        "type": "function",
        "z": "e260f053.ca405",
        "name": "Add Time Info",
        "func": "var now = new Date();\nmsg.payload[\"timestamp\"] = now.getTime();\nmsg.payload[\"fomratted-time\"] = now.toUTCString();\nreturn msg;",
        "outputs": 1,
        "noerr": 0,
        "x": 600,
        "y": 220,
        "wires": [
            [
                "adf30d5c.857be"
            ]
        ]
    },
    {
        "id": "a481358c.031c18",
        "type": "mqtt-broker",
        "z": "",
        "broker": "192.168.1.11",
        "port": "1883",
        "clientid": "",
        "usetls": false,
        "compatmode": true,
        "keepalive": "60",
        "cleansession": true,
        "willTopic": "",
        "willQos": "0",
        "willPayload": "",
        "birthTopic": "",
        "birthQos": "0",
        "birthPayload": ""
    }
]

What are we going to do with all this data flowing in? That's the topic of the next post!

Node-RED: Calling REST Services and Building Our Own API

The goal for this tutorial is to build a Node-RED flow that grabs weather JSON from Weather Underground and presents the results in a web page. Our Node-RED service will accept requests of the form

/weather?zip=<<zipcode>>
It will call the Weather Underground API for that zipcode, then return the data formatted as HTML. Only simple error handling will be implemented.


The Weather Underground API
Weather Underground provides an API that provides current weather conditions based on zipcode, etc. To use this API, it will be necessary to create a free developer account, called the "Stratus Plan", which allows for 500 API calls per day and at most 10 calls per minute.

Use that account to generate an API key. Using that key along with their conditions API, weather data for zipcode 18901 can be retrieved as follows:

http://api.wunderground.com/api/<<API_key_goes_here>>/features/conditions/q/18901.json

Here's an example response. The returned JSON is overkill, and there doesn't seem to be a way to specify which fields to return. The fields we'll be using are highlighted in yellow:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
{
  "response": 
  {
    "version": "0.1",
    "termsofService": "http://www.wunderground.com/weather/api/d/terms.html",
    "features": 
    {
      "conditions": 1
    },
    "error": 
    {
      "type": "unknownfeature"
    }
  }, 
  "current_observation": 
  {
    "image": 
    {
      "url": "http://icons.wxug.com/graphics/wu2/logo_130x80.png",
      "title": "Weather Underground",
      "link": "http://www.wunderground.com"
    },
    "display_location": 
    {
      "full": "Doylestown, PA",
      "city": "Doylestown",
      "state": "PA",
      "state_name": "Pennsylvania",
      "country": "US",
      "country_iso3166": "US",
      "zip": "18901",
      "magic": "1",
      "wmo": "99999",
      "latitude": "40.31000137",
      "longitude": "-75.12999725",
      "elevation": "128.6"
    },
    "observation_location": 
    {
      "full": "Doylestown Harvey, Doylestown, Pennsylvania",
      "city": "Doylestown Harvey, Doylestown",
      "state": "Pennsylvania",
      "country": "US",
      "country_iso3166": "US",
      "latitude": "40.312313",
      "longitude": "-75.135078",
      "elevation": "289 ft"
    },
    "estimated": {},
    "station_id": "KPADOYLE20",
    "observation_time": "Last Updated on November 23, 8:12 PM EST",
    "observation_time_rfc822": "Thu, 23 Nov 2017 20:12:54 -0500",
    "observation_epoch": "1511485974",
    "local_time_rfc822": "Thu, 23 Nov 2017 20:12:56 -0500",
    "local_epoch": "1511485976",
    "local_tz_short": "EST",
    "local_tz_long": "America/New_York",
    "local_tz_offset": "-0500",
    "weather": "Clear",
    "temperature_string": "29.3 F (-1.5 C)",
    "temp_f":29.3,
    "temp_c":-1.5,
    "relative_humidity": "62%",
    "wind_string": "Calm",
    "wind_dir": "South",
    "wind_degrees":182,
    "wind_mph":0.0,
    "wind_gust_mph":0,
    "wind_kph":0,
    "wind_gust_kph":0,
    "pressure_mb": "1017",
    "pressure_in": "30.04",
    "pressure_trend": "-",
    "dewpoint_string": "18 F (-8 C)",
    "dewpoint_f":18,
    "dewpoint_c":-8,
    "heat_index_string": "NA",
    "heat_index_f": "NA",
    "heat_index_c": "NA",
    "windchill_string": "29 F (-2 C)",
    "windchill_f": "29",
    "windchill_c": "-2",
    "feelslike_string": "29 F (-2 C)",
    "feelslike_f": "29",
    "feelslike_c": "-2",
    "visibility_mi": "10.0",
    "visibility_km": "16.1",
    "solarradiation": "1",
    "UV": "0.0","precip_1hr_string": "0.00 in ( 0 mm)",
    "precip_1hr_in": "0.00",
    "precip_1hr_metric": " 0",
    "precip_today_string": "0.00 in (0 mm)",
    "precip_today_in": "0.00",
    "precip_today_metric": "0",
    "icon": "clear",
    "icon_url": "http://icons.wxug.com/i/c/k/nt_clear.gif",
    "forecast_url": "http://www.wunderground.com/US/PA/Doylestown.html",
    "history_url": "http://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=KPADOYLE20",
    "ob_url": "http://www.wunderground.com/cgi-bin/findweather/getForecast?query=40.312313,-75.135078",
    "nowcast": ""
  }
}

Our Node-RED flow should be able to handle some error conditions. Here's one example of an error response from the conditions API:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
{
  "response":
  {
    "version": "0.1",
    "termsofService": "http://www.wunderground.com/weather/api/d/terms.html",
    "features":
    {
      "conditions": 1
    },
    "error":
    {
      "type": "querynotfound",
      "description": "No cities match your search query"
    }
  }
}

It was generated by requesting weather for a non-existent zipcode, like this:

http://api.wunderground.com/api/<<API_key_goes_here>>/features/conditions/q/99999.json


The Flow
Now, onto our goal of writing a Node-RED flow to call that API and format the response as pretty HTML!

For the flow, we will be using the following components:

HTTP In
Listen for requests of the form /weather?zip=<<zipcode>>
Function
Extract the zipcode out of the request query params, and build the Weather Underground API call
HTTP Request
Make the API call and when we get a response...
Switch
... check to see if it was an error
Template
Extract data from the response JSON and format it into an HTML page
HTTP Response
Return the HTML generated by the template node
Debug
Show API URL as well as the Weather Underground response

The nodes are connected with wires as follows:

Note that the two Debug nodes near the top are redundant, since both are outputting the msg object! If we wanted to, we could wire the outputs from the HTTP In node and the "Set URL" Function node into one Debug node.

The code for the Set URL Function node is as follows:

1
2
msg.url = "http://api.wunderground.com/api/<<API_key_goes_here>>/features/conditions/q/" + msg.req.query.zip + ".json";
return msg;

In the HTTP Request node, set the return type to be "a parsed JSON object".

In the "Check for Error" Switch node, if msg.payload.response.error.type equals "unknownfeature" we route to the "success" template, otherwise we route to the error template. Here's what the Switch node configuration form looks like:

The template for when Weather Underground successfully returned data is as follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
<!DOCTYPE html>
<html>
    <head>
        <title>Weather!</title>
        <style>body {font-family: sans-serif}</style>
    </head>
    <body>
        <h1>Weather in {{payload.current_observation.display_location.full}}</h1>
        <h2>{{payload.current_observation.weather}} <img src="{{payload.current_observation.icon_url}}" /></h2>

        Temperature: {{payload.current_observation.temperature_string}}<br />
        Humidity: {{payload.current_observation.relative_humidity}}<br />
        Pressure: {{payload.current_observation.pressure_in}} inHg<br />
        Wind: {{payload.current_observation.wind_string}}<br />

        <h6>{{payload.current_observation.observation_time}}</h6>

        <a href="{{{payload.current_observation.image.link}}}" target="_blank"><img src="{{{payload.current_observation.image.url}}}" /></a>
    </body>
</html>

Mustache template tags are used in the HTML body. Notice that "triple mustaches" are used on line 18 since we don't want Mustache to escape the URL.

When an error condition is detected, the flow will use this template to report an error:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
<html>
    <head>
        <title>Weather!</title>
        <style>body {font-family: sans-serif}</style>
    </head>
    <body>
        <h1>Error Getting Weather</h1>
        <h2>{{payload.response.error.description}}</h2>
    </body>
</html>

Open a browser and go to http://127.0.0.1:1880/weather?zip=18901, and something like the following page will be presented:

But if you go to http://127.0.0.1:1880/weather?zip=99999, you'll get the error page:

Finally, here is the complete source code to that flow, minus my API key, of course!

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
[
  {
    "id": "4ece1ecb.8096b",
    "type": "http in",
    "z": "7651fb0.bb59804",
    "name": "",
    "url": "/weather",
    "method": "get",
    "upload": false,
    "swaggerDoc": "",
    "x": 110,
    "y": 120,
    "wires": [
      [
        "658ac2c5.bc83cc",
        "ca471dea.0d8d"
      ]
    ]
  },
  {
    "id": "869b03f7.a180c",
    "type": "http request",
    "z": "7651fb0.bb59804",
    "name": "Get Weather",
    "method": "GET",
    "ret": "obj",
    "url": "",
    "tls": "",
    "x": 270,
    "y": 280,
    "wires": [
      [
        "b763dd13.c9185",
        "b54ab768.1f6298"
      ]
    ]
  },
  {
    "id": "7b4c5ed6.4422d",
    "type": "template",
    "z": "7651fb0.bb59804",
    "name": "Show Weather",
    "field": "payload",
    "fieldType": "msg",
    "format": "handlebars",
    "syntax": "mustache",
    "template": "<!DOCTYPE html>\n<html>\n    <head>\n        <title>Weather!</title>\n        <style>body {font-family: sans-serif}</style>\n    </head>\n    <body>\n        <h1>Weather in {{payload.current_observation.display_location.full}}</h1>\n        <h2>{{payload.current_observation.weather}} <img src=\"{{payload.current_observation.icon_url}}\" /></h2>\n\n        Temperature: {{payload.current_observation.temperature_string}}<br />\n        Humidity: {{payload.current_observation.relative_humidity}}<br />\n        Pressure: {{payload.current_observation.pressure_in}} inHg<br />\n        Wind: {{payload.current_observation.wind_string}}<br />\n\n        <h6>{{payload.current_observation.observation_time}}</h6>\n\n        <a href=\"{{{payload.current_observation.image.link}}}\" target=\"_blank\"><img src=\"{{{payload.current_observation.image.url}}}\" /></a>\n    </body>\n</html>",
    "output": "str",
    "x": 700,
    "y": 240,
    "wires": [
      [
        "7a405c32.7c1134"
      ]
    ]
  },
  {
    "id": "7a405c32.7c1134",
    "type": "http response",
    "z": "7651fb0.bb59804",
    "name": "",
    "statusCode": "",
    "headers": {},
    "x": 890,
    "y": 280,
    "wires": []
  },
  {
    "id": "7b473466.a14c6c",
    "type": "debug",
    "z": "7651fb0.bb59804",
    "name": "",
    "active": true,
    "console": "false",
    "complete": "true",
    "x": 450,
    "y": 120,
    "wires": []
  },
  {
    "id": "658ac2c5.bc83cc",
    "type": "debug",
    "z": "7651fb0.bb59804",
    "name": "",
    "active": true,
    "console": "false",
    "complete": "true",
    "x": 310,
    "y": 40,
    "wires": []
  },
  {
    "id": "ca471dea.0d8d",
    "type": "function",
    "z": "7651fb0.bb59804",
    "name": "Set URL",
    "func": "msg.url = \"http://api.wunderground.com/api/<<API_key_goes_here>>/features/conditions/q/\" + msg.req.query.zip + \".json\";\nreturn msg;",
    "outputs": 1,
    "noerr": 0,
    "x": 280,
    "y": 120,
    "wires": [
      [
        "869b03f7.a180c",
        "7b473466.a14c6c"
      ]
    ]
  },
  {
    "id": "b763dd13.c9185",
    "type": "debug",
    "z": "7651fb0.bb59804",
    "name": "",
    "active": true,
    "console": "false",
    "complete": "false",
    "x": 470,
    "y": 400,
    "wires": []
  },
  {
    "id": "b54ab768.1f6298",
    "type": "switch",
    "z": "7651fb0.bb59804",
    "name": "Check for Error",
    "property": "payload.response.error.type",
    "propertyType": "msg",
    "rules": [
      {
        "t": "eq",
        "v": "unknownfeature",
        "vt": "str"
      },
      {
        "t": "neq",
        "v": "unknownfeature",
        "vt": "str"
      }
    ],
    "checkall": "true",
    "outputs": 2,
    "x": 480,
    "y": 280,
    "wires": [
      [
        "7b4c5ed6.4422d"
      ],
      [
        "db257584.8aa478"
      ]
    ]
  },
  {
    "id": "db257584.8aa478",
    "type": "template",
    "z": "7651fb0.bb59804",
    "name": "Show Error",
    "field": "payload",
    "fieldType": "msg",
    "format": "handlebars",
    "syntax": "mustache",
    "template": "<html>\n    <head>\n        <title>Weather!</title>\n        <style>body {font-family: sans-serif}</style>\n    </head>\n    <body>\n        <h1>Error Getting Weather</h1>\n        <h2>{{payload.response.error.description}}</h2>\n    </body>\n</html>",
    "output": "str",
    "x": 690,
    "y": 320,
    "wires": [
      [
        "7a405c32.7c1134"
      ]
    ]
  }
]