Publish RuuviTag sensor data on The Things Network

In this tutorial we use a LoPy or FiPy to track temperature and humidity from a location with no WiFi and (in my case) no power supply socket and publish the RuuviTag sensor data on The Things Network with the MicroPython RuuviTag Scanner.

This tutorial uses settings specifically for connecting to The Things Network within the European 868 MHz region. For another usage, please see the settings.py and node.py files for relevant sections that need changing.

Note

The code in this tutorial is not bounded to The Things Network and can also be used with other LoRaWAN networks.

This tutorial is made available under the Creative Commons Attribution 4.0 International (CC BY 4.0) license. Example code is made available under the MIT License.

Hardware

_images/ruuvitag-lorawan.jpg
  • 1 x LoPy or FiPy from pycom
  • 1 x LoRa Antena
  • 1 x Expansion Board
  • 1 x Case
  • 1 x Lithium Ion Polymer Battery or battery pack
  • 1 or more RuuviTags

TL;DR

If you are familiar with MicroPython, LoPy, RuuviTag, TTN or just want to get started now, you can get the up to date snippets from the tutorial repository on GitHub. Modify the settings.py file and copy all *.py files to your device.

LoRaWAN limitations

LoRaWAN is limited on how much data can be send over the network. You can read more about the limitations on:

In this tutorial we only want temperature and humidity from our tags. To save space we use the RuuviTag Data Format 5. But you can pack the data in any format you like or add more sensor data.

Note

A good goal is to keep the payload under 12 bytes. For temperature and humidity we need 2 bytes for each and one extra byte as a identifier, if we plan to scan more than one tag. In sum we have 5 bytes for each tag. As we don’t need to send updates every minute we can add two or three more tags to the payload and send the measurements every 5 or 10 minutes.

Device setup

If you are new to LoPy/FiPy, I recommend you start with updating your device firmware and go on with REPL & uploading code. To upload code you can use FTP, the Pymakr Plugins or use something like the mpfshell. I prefer mpfshell. With mpfshell you can easily manage the data on your device similar to an ftp client program and also access the REPL prompt.

After your are familiar with your device and updated to the latest firmware, it’s time to install the MicroPython RuuviTag Scanner. Copy the ruuvitag directory from the repository to your device /flash/lib/ directory.

In example with mpfshell on Linux with the device connected to ttyUSB0:

  1. Clone the repository:

    git clone https://github.com/rroemhild/micropython-ruuvitag.git
    
  2. Go to the module directory:

    cd micropython-ruuvitag
    cd ruuvitag
    
  3. Copy the files to your device

    mpfshell ttyUSB0
    cd lib
    md ruuvitag
    cd ruuvitag
    mput .
    

TTN Device Registration

Before we can start hacking, we need to add our new device to The Things Network.

  1. Start a MicroPython REPL prompt.

  2. Run the following commands in the REPL:

    >>> import ubinascii
    >>> from network import LoRa
    >>> lora = LoRa(mode=LoRa.LORAWAN)
    >>> print(ubinascii.hexlify(lora.mac()).upper().decode('utf-8'))
    

This should print your Device EUI like:

70B3D5499C89D4CE
  1. Follow the steps to register your device.

Note

We’ll use the Over The Air Activation (OTAA) to negotiate session keys for further communication.

Disable WiFi

To disable wifi on boot, connect to the REPL prompt and run the following commands:

>>> import pycom
>>> pycom.wifi_on_boot(False)

This will disable WiFi on your pycom board and it will persist between reboots.

Hands on code

The device is ready, our TTN is setup, finally we can start add our code. We have 3 files: settings.py, node.py and main.py. Create these files on your workstation and copy them later to the device root (/flash) directory. The main.py is the file that run on each boot.

Configuration (settings.py)

This file contains settings for the device. As we use the OTAA activation copy the Application EUI and Application Key from the TTN console “device overview” to the appropriate variables.

The RUUVITAGS variable is used as a device whitelist for the RuuviTags you want to publish data. In our application we use the tuple index from the mac addresses as a device id. In example the first mac address is the device on the stable box and the second from the greenfield sites.

Note

Keep in mind that we want a small payload, only allow some tags to publish sensor data.

# LoRaWAN
LORA_FREQUENCY = 868100000
LORA_NODE_DR = 5

# OTAA activation
NODE_APP_EUI = 'Application-EUI'
NODE_APP_KEY = 'Application-Key'

# Device deepsleep time in milliseconds
NODE_DEEPSLEEP = 300000

# RuuviTags whitelist, other tags will be ignored
RUUVITAGS = (b'00:01:02:03:04:05', b'10:11:12:13:14:15')

# Bluetooth scan timeout
TIMEOUT = 10

LoRaWAN node (node.py)

This file contains the LoRaWAN network setup and don’t need to be modified for the European 868 MHz region. The LoRaWAN class abstract the network access. I’ll not get in details with this code to keep the focus on the RuuviTag code. In short this class handle the LoRaWAN setup for the European 868 MHz region, joins the network, prepare a socket and take care of the OTAA session keys.

import utime
import usocket
import ubinascii

from network import LoRa


class LoRaWANNode:

    def __init__(self, app_eui, app_key, frequency=868100000, dr=5):
        '''setup LoRaWAN for the European 868 MHz region with OTAA'''
        self.dr = dr
        self.frequency = frequency
        self.app_eui = ubinascii.unhexlify(app_eui)
        self.app_key = ubinascii.unhexlify(app_key)

        self.lora = LoRa(mode=LoRa.LORAWAN, region=LoRa.EU868)
        self.socket = None

        self.setup()

    @property
    def has_joined(self):
        return self.lora.has_joined()

    def default_channels(self):
        ''''
        Remove all non-default channels and set the 3 default channels to the
        same frequency (must be before ending the OTAA join request)
        '''
        for i in range(3, 16):
            self.lora.remove_channel(i)

        self.lora.add_channel(0, frequency=self.frequency,
                              dr_min=0, dr_max=self.dr)
        self.lora.add_channel(1, frequency=self.frequency,
                              dr_min=0, dr_max=self.dr)
        self.lora.add_channel(2, frequency=self.frequency,
                              dr_min=0, dr_max=self.dr)

    def setup(self):
        '''Try to restore from nvram or join the network with otaa'''
        self.default_channels()
        self.lora.nvram_restore()

        if not self.has_joined:
            self.join()
        else:
            self.open_socket()

    def join(self, timeout=10):
        try:
            timeout = timeout * 1000
            self.lora.join(activation=LoRa.OTAA,
                           auth=(self.app_eui, self.app_key),
                           timeout=timeout, dr=self.dr)

            if self.has_joined:
                self.lora.nvram_save()
                self.open_socket()
        except TimeoutError:
            pass

    def open_socket(self, timeout=6):
        self.socket = usocket.socket(usocket.AF_LORA, usocket.SOCK_RAW)
        self.socket.setsockopt(usocket.SOL_LORA, usocket.SO_DR,
                               self.dr)
        self.socket.settimeout(timeout)

    def reset(self):
        self.socket.close()
        self.lora.nvram_erase()
        self.join()

    def send(self, data):
        '''Send out data as bytes'''
        if self.has_joined:
            if isinstance(data, (float, str, int)):
                data = bytes([data])
            self.socket.send(data)
            utime.sleep(2)
            self.lora.nvram_save()

Main (main.py)

The main.py file scans the RuuviTags in range, prepare sensor data and send the payload to The Things Network. To extend the battery life the device goes into deepsleep mode, wake up after 5 minutes and repeats.

We start to import required modules, node, settings and the RuuviTagScanner.

import machine
import ustruct

import settings

from node import LoRaWANNode
from ruuvitag.scanner import RuuviTagScanner

Now we need to pack the sensor data. We use the data format from RuuviTag for temperature and humidity and add a tag id:

Offset Description
0 Tag ID (8bit)
1-2 Temperature in 0.005 degrees (16bit signed)
3-4 Humidity in 0.0025% (16bit unsigned)

In example for 2 RuuviTags the following payload will be send:

b'\x00\x11\x8aZ\xd0\x01\x06&y\xe0'

If we split this into parts, we get:

Stable Greenfield sites
ID Temp Hum ID Temp Hum
00 118A 5AD0 01 0626 79E0

To achieve this we add two functions to our code:

def pack_temp(temp):
    """Temperature in 0.005 degrees as signed short"""
    temp_conv = round(round(temp, 2) / 0.005)
    temp_int16 = ustruct.pack('!h', temp_conv)
    return temp_int16


def pack_hum(hum):
    """Humidity in 0.0025 percent as unsigned short"""
    hum_conv = round(round(hum, 2) / 0.0025)
    hum_int16 = ustruct.pack('!H', hum_conv)
    return hum_int16

To bring all the sensor data together we add a payload variable with an empty bytes objects, later we’ll add the packed sensor data to this object:

payload = b''

Now we initialize the RuuviTagScanner. Remember to not scan all the tags around you and add just the ones you need.

rts = RuuviTagScanner(settings.RUUVITAGS)

We are setup and can start scanning for the tags and pack the data together. You can set a higher timeout in the settings.py file if your tag is on a longer range.

The tag id is his index from the whitelist tuple we set in settings.RUUVITAGS. When you unpack the payload on the target platform you have to remember the tag position from the tuple.

for ruuvitag in rts.find_ruuvitags(timeout=settings.TIMEOUT):
    id_payload = settings.RUUVITAGS.index(ruuvitag.mac.encode())
    temp_payload = pack_temp(ruuvitag.temperature)
    hum_payload = pack_hum(ruuvitag.humidity)
    payload = payload + bytes([id_payload]) + temp_payload + hum_payload

When all tags where processed and our payload is ready, we setup up LoRaWAN and send out the payload.

node = LoRaWANNode(settings.NODE_APP_EUI, settings.NODE_APP_KEY)
node.send(payload)

At the end we send the device into deepsleep mode for 5 minutes:

machine.deepsleep(settings.NODE_DEEPSLEEP)

The complete main.py file:

import machine
import ustruct

import settings

from node import LoRaWANNode
from ruuvitag.scanner import RuuviTagScanner


def pack_temp(temp):
    """Temperature in 0.005 degrees as signed short"""
    temp_conv = round(round(temp, 2) / 0.005)
    temp_int16 = ustruct.pack('!h', temp_conv)
    return temp_int16


def pack_hum(hum):
    """Humidity in 0.0025 percent as unsigned short"""
    hum_conv = round(round(hum, 2) / 0.0025)
    hum_int16 = ustruct.pack('!H', hum_conv)
    return hum_int16


payload = b''

rts = RuuviTagScanner(settings.RUUVITAGS)

# get all data and prepare payload
print('harvest ruuvitags')
for ruuvitag in rts.find_ruuvitags(timeout=settings.TIMEOUT):
    id_payload = settings.RUUVITAGS.index(ruuvitag.mac.encode())
    temp_payload = pack_temp(ruuvitag.temperature)
    hum_payload = pack_hum(ruuvitag.humidity)
    payload = payload + bytes([id_payload]) + temp_payload + hum_payload

print('setup lorawan')
node = LoRaWANNode(settings.NODE_APP_EUI, settings.NODE_APP_KEY)

print('send payload')
node.send(payload)

print('enter deepsleep for {} ms'.format(settings.NODE_DEEPSLEEP))
machine.deepsleep(settings.NODE_DEEPSLEEP)

Now reset your device and watch the incoming application data on the TTN console.

TTN Payload Format Decoder

On The Things Network Console we can now decode our payload with the following javascript example. Remember the position from each tag from the settings.RUUVITAGS variable, we will give them names in the decoded output.

function Decoder(bytes, port) {
  var ruuvitags = {};
  var tagname = "";
  var tags = bytes.length / 5;

  for (i=0;i<tags;i+=1) {
    var temperature = (bytes[1] << 8) | bytes[2];
    var humidity = (bytes[3] << 8) | bytes[4];

    if (bytes[0] === 0) {
      tagname = "stable";
    }
    else if (bytes[0] === 1) {
      tagname = "greenfield";
    }

    ruuvitags[tagname] = {
        "humidity": parseFloat((humidity * 0.0025).toFixed(2)),
        "temperature": parseFloat((temperature * 0.005).toFixed(2))
    };

    bytes.splice(0, 5);
  }

  return ruuvitags;
}
  1. Find out how many tags are included in the payload. Since we know that one tag use 5 bytes for payload, we divide the payload length with 5.
  2. Iterate over the payload and unpack the data. In position zero we find the tag number from our tuple. We do a simple if-else match and give each RuuviTag a name.
  3. Add the RuuviTag name with the measurements to the ruuvitags object, remove the first 5 bytes from bytes and iterate over the next payload data.
  4. Return the decoded data.

When you take a look on the TTN Console Device Overview in the data section, you can see the incoming unpacked payload.