Helium is a multi network server with a decentralized packet routing system. This is really clever and allows anyone to use the public infrastructure as private LoRaWAN compatible network. That way you get benefit of a worldwide coverage and, in the same time, the ability to protect your raw data from anyone looking at them. You can also create some other public network server, as we are doing with Helium-Iot.eu

The objective of a such public service is to offer a shorter route for your European devices and as a consequence a shorter response time for downlink. It also ensure your data to stay in Europe, something important for personal data like tracking, health or for industrial applications.

For a better understanding, let’s take a look at the Helium network architecture:

Thanks to the miner components incorporated in the hotspots, the traffic from the devices is directly routed to the right network server. Each of the network servers belongs to a, operator, it can be you or me or any established telecom operator. This is basically really cool !

That’s why we have decided with the company I work for to take a look at this business and launched Helium-IoT.eu. So you can connect your devices to Helium using our console https://console.helium-iot.eu

In the next page of this post I will explain how to become your own operator for making your private network. This is a bit complex operations, so if you want your own network server, as part of our services we are proposing to make it for you and host it. We also have solutions to migrate existing LoRaWan networks to Helium. Just let us know by contacting me with the contact link.

So, let’s be more technical to understand all of this.

Some overall information before getting started

The power of the blockchain is to allow anyone to be part of it with no specific contract other than the smart-contract. No negotiation, no long discussion with layers, no better price to obtain than any other … You just apply the common rules and get into it. In a world of big telecom companies, where all the b2b prices are secrets it makes a big difference and allows you to distribute communication over a network of more than 25.000 hotspot, worldwide, in less than a week where any technical stuff matter.

So, what we need to do to be a Helium operator is to get a OUI, basically a network operator identifier. Then we need to get some of the DevADDR to associate to the device we are going to route. After that, we have to host a router, connect it to the pair-to-pair network, run a console and start receiving IoT device data.

The Helium Network Server runs with some transactions with associated economics (always in DC, stable over time).

• OUI Creation – $100
• DevAddr Association – $100 each / minimum 8
• State_Channel – $0.35 by default every 45 blocks (about 45 minutes) = $12 / day, should be changed
• IoT Device Attachment (filter update) – $0.40 on every device addition update (can be grouped, run on every 10 minutes)
• IoT Device Communication – $0.00001 per message or 24 bytes

The communication are always paid by the router, not the device itself. In fact we can consider that a device is owned by a router, so communication are paid by that router when accepting each of the packets.

You should read State Channel documentation. Apparently, State channel is a sub chain to track the transactions related to the communication between hotspot and router. The duration of the state channel impact the frequency of Packet Transfer payment. 2 state channels are opened, the second one have a life cycle twice of the setting made for the first one. this ensure you have one state channel when renewing the previous one. When the state channel is open, on top of the cost, there is a “state channel amount” DC blocked to to ensure you will pay for the communications during the state_channel duration. At end of the state channel duration you get it back. Once this amount in consumed, the state channel is renewed. This is about 1000 DC. (by default, this means the real price of transferring 24 bytes of data is not 1DC but 18.5DC… because 2 time this amount of DC is reserved). All of this will be part of the parameters.

The DevAddr are multiplexed over different devices, there is nothing defined currently but a factor of 10 to 500 active devices per DevAddr should make sense.

In term of security, the network server own the encryption keys and is in charge of decrypting the payload. This is the way it is able to differentiate the different devices behind the same DevAddr: only one of them have the right key to get a valid frame after decryption. This is also why Helium allows to have a public network acting as a private network. Only the right network server is manipulating the decrypted payload.

As you can see the traffic stay encrypted from the device to the Network Server. The Device data is not stored in the blockchain and none of the component between device to Network Server knows the secret keys. If you own the Network Server (router+console), you are the only one to access the keys.

Let’s implement it

The following part of the blog post describes the way I’ve created my own Network Server. For doing this I had to use different wallet version, some not yet officially published. The wallet should soon been updated (if not yet) to includes the fixes and this documentation should be ok. But currently it could not work on every steps.

By following this tutorial, you are going to spend more than $900 in DC / HNT. if you make something wrong or if I mis-document something you can loose this money. So you decide or not to follow this tutorial, you decide to do it on your own and I won’t be responsible of any loss you could have.

As a reminder, we are providing on helium-iot.eu Helium Network Servers as a Service to avoid all this complexity.

Create a Wallet

A wallet is needed to execute the operation to get the OUI and DevAddr range. The binary version was not working on my centos due to library incompatibility.

Install RUST

Following the recommendations on rust-lang.org, this is processing the installation automatically.

[~] curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Source the rust environement:

[~] source $HOME/.cargo/env

Install the needed dependencies

[~] yum install gcc openssl-devel

Build Wallet

Download the wallet sources from helium wallet Github releases ; uncompress it, then enter the directory and run compilation:

[~helium] cargo build --release

Now you can run the wallet from target/release and run a first test command

[~helium] cd target/release/ 
[~wallet] ./helium-wallet -h  # It shows help

Create the wallet

You can create a basic wallet (1 key file) or a sharded wallet (multiple keys) (see helium github). Here is the creation of a basic wallet.

[~wallet] ./helium-wallet create basic

The creation process asks for a password… do not loose it. And it generate a wallet.key file you need to backup this file.

The creation of the wallet will terminate with a print of your wallet address:

+-----------------------------------------------------+------------+...
| Address                                             | Balance    | Data
+-----------------------------------------------------+------------+-...
| 13HWrdkMoooYJEYXXXXXXXXXXXZifCzwxPaCnV9XXXbC5Gso1ic | 0.00000000 | 0...
+-----------------------------------------------------+------------+-...

Transfer credits to the wallet

To operate on Helium, you need to create an OUI and at least 8 DEVADR. The OUI allows you to generate DEVEUI and DEVADR are the address used by the device once they have been joined the network. As explained in this page, you can multiplex the DEVADR. The OUI cost 10M DC, the DEVADDR also 10MDC each, you can by them by 8/16/32… So at start, you need to pay 90M DC to start being an operator. Depending on the HNT value, you will need to transfer to this wallet the right amount.

The DevAddr multiplexing select a device address in regard of the original location of the device to spread the device over the available DevAdr. When the number of DevAddr is too low compare to the number of running device, the network server will need more time to find the right device and there is a risk to be late on time for operations such as downlink which are time critical.

So you need to have the credits for the OUI/DEVADDR creation. From you hotspot wallet of binance wallet, you can make a transfert to the wallet created. I recommend to start by a 1 to 10 HNT transfer, then going further with the expected amount. Then you need to transfer the equivalent of 90M DC ($900).

Check your wallet balance to see the credit appearing:

[~wallet] ./helium-wallet balance
+-----------------------------------------------------+--------------+...
| Address                                             | Balance      | 
+-----------------------------------------------------+--------------+...
| 13HWrdkMoooYJEYXXXXXXXXXXXZifCzwxPaCnV9XXXbC5Gso1ic | 220.00000000 | 
+-----------------------------------------------------+--------------+...

Now the HNT needs to be burned in DC

[~wallet] ./helium-wallet burn --amount 2xx --payee 13HWrdkMoooYJEYXXXXXXXXXXXZifCzwxPaCnV9XXXbC5Gso1ic --commit

The transaction processing takes some time (you can check the status with the hash returned by the previous command on https://api.helium.io/v1/pending_transactions/hashValue.

Once you have 90M or more DC you can proceed to OUI creation.

Create OUI and DevADDR

Now we are going to proceed OUI + 8 DevADDR creation (this is burning the 90M DC)

[~wallet] ./helium-wallet oui create --subnet-size 8 --filter wVwCiewtCpEKAAAAAAAAAAAAcCK3fwAAAAAAAAAAAABI7IQOAHAAAAAAAAAAAAAAAQAAADBlAAAAAAAAAAAAADEAAAA2AAAAOgAAAA --commit

This is generating the OUI creation transaction indicating the Requested OUI

+------------------------+---------------------------------------------+
| Key                    | Value                                       |
+------------------------+---------------------------------------------+
| Requested OUI          | 5                                           |
+------------------------+---------------------------------------------+
| Reqeuested Subnet Size | 8                                           |
+------------------------+---------------------------------------------+

Now, we can process to the router / console installation, see next page…

Pre-requisites

Server sizing: according to my test, the router container is requiring a configuration with between 4GB to 8GB ram (it could be related to the activity and the State Channel size so, monitor this value). CPU consumption is light, 2 VCPU seems enough.

You need to have some components like docker-ce. For installing it on centos, you can follow the official installation guide. Then you need to install docker-compose:

[~] sudo curl -L "https://github.com/docker/compose/releases/download/1.28.5/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
[~] sudo chmod +x /usr/local/bin/docker-compose

Start docker (after installing)

[~] sudo systemctl start docker
[~] sudo systemctl enable docker

Clone the Repository

You need to clone the repository on https://github.com/helium/console and select the right stable release. I do not recommend to build from master. Access the list of releases on https://github.com/helium/console/releases

git clone https://github.com/helium/console.git
git tag # to list the possible tags
git checkout 2021.07.29 # as an example of release name

Open port

make sure ports 2154, 8080, 443 are open on this server to allow communications with the different protocols involved.

Start SETUPING

The system require a postgresql database, this one is included in a container during the build phase. You can modify the name and password by editing the docker-compose.yaml file. This also contains the directory where the database files will be stored on your filesystem to be persistent.

...
console:
  build:
    context: .
    args:
      socket_check_origin: "//*.helium-iot.eu"
    ...
postgres:
    image: postgres
    container_name: helium_postgres
    restart: always
    environment:
        - POSTGRES_DB=helium_console
        - POSTGRES_PASSWORD=xxx
        - POSTGRES_USER=xxx (if you setup a specific user)
    volumes:
      - "./data/postgres/:/var/lib/postgresql/data"

Then you need a Auth0 account or Magic.Link for managing user connection. Auth0 have a free subscription up-to 7000 users. This is currently, apparently, the only one identity provider supported by the open-source console. I recommend to create a new tenant with your domain in it and deactivate the social login until you completely setup it or you are going to have a big warning on the front page. Upload your logo. You need to setup the Application Type as Native. After registering your first user you need to deactivate application registration in the advanced settings. Take also a look at the 2 factor authentication parameters and reCaptcha options.

You also need a Mailgun account to let the application send email to users. This is needed as an example to validate the API keys. When creating the Mailgun account you need to validate your account by setting different DNS fields and be able to change your DNS entries for this.

Install console & router

Basically, the role of the router is to receive a data block from a device. It decides to accept it or reject it. When accepting it, this will burn the associated DC for this communication. The wallet for burning DC is your wallet.

The role of the console is to associate new devices, watch data, manage integration with a third party IoT application… This is the relation between the user and its devices.

To install the console, the best is to follow the Helium documentation described in the linked page. As a precision:

• in the .env file you need to setup your database endpoint

DATABASE_DB=helium_console
DATABASE_HOST=postgres
DATABASE_USER=helium_console
DATABASE_PASSWORD=xxx
SOCKET_CHECK_ORIGIN=https://*.helium-iot.eu

# Email settings
MAILGUN_API_KEY=xxx....xxxx-xxxxx-xxxx      (private API key)
SITE_DOMAIN_MAILGUN=helium-iot.eu           (your mailgun domain)
MAILGUN_URL=https://api.eu.mailgun.net/v3   (for Europe)
MAIL_FROM=contact@helium-iot.eu
MAIL_REPLY_TO=contact@heliu-iot.eu

# Login settings
USE_MAGIC_AUTH=true
MAGIC_PUBLIC_KEY=pk_live_xxxx
MAGIC_SECRET_KEY=sk_live_xxxx

• in the .env-router file, make sure all the variables are in the template file, sometime they are not in sync with the file from the router project. Once you verified all the variable are in the file, you should change the SEED_NODES with IP of currently running Helium Miner. The usual miner port is 44158, so make sure you use the right IP and PORT. One seed node is enough to start. You may also change the State channel expiration time (in block). This is corresponding to state_channel renewal, this operation costs 35000DC ($0,35) every-time.

ROUTER_SEED_NODES=/ip4/35.166.211.46/tcp/2154,/ip4/44.236.95.167/tcp/2154
...
# helium_console correspond to the docker container name
# can be replaced by the public endpoint but not recommended
ROUTER_CONSOLE_ENDPOINT=http://helium_console:4000
ROUTER_CONSOLE_WS_ENDPOINT=ws://helium_console:4000/socket/router/websocket
..
ROUTER_SC_OPEN_DC_AMOUNT=50000
ROUTER_SC_EXPIRATION_INTERVAL=5000
ROUTER_SC_EXPIRATION_BUFFER=15
ROUTER_XOR_FILTER_WORKER=true
ROUTER_HTTP_CHANNEL_URL_CHECK=false

ROUTER_SC_EXPIRATION is between 15 and 10080 (1 week). I currently assume the max value is 5040, I won’t detail here, see github open issue.

ROUTER_SC_OPEN_DS_AMOUNT is the number of DC blocked for the State Channel payments. So above this limit the state channel will be impossible to be used anymore and a new one open. This reduce the number of available state channel and can cause critical issues if you can’t open some new. See details on the Helium State Chanel behavior.

ROUTER_HTTP_CHANNEL_URL_CHECK avoid the router to check the domain validity before executing the http integration. (This should be enabled but I figured out some bug with it:

• In the config/release.exs file, change you host name (used for redirecting)

config :console, ConsoleWeb.Endpoint,
  server: true,
  url: [host: "console.helium-iot.eu", port: 80]

or in https (recommended)

config :console, ConsoleWeb.Endpoint,
  server: true,
  url: [scheme: "https", host: "console.helium-iot.eu", port: 443], 
  #force_ssl: [rewrite_on: [:x_forwarded_proto]],
  cache_static_manifest: "priv/static/cache_manifest.json",

• Get docker-compose.yaml from templates and copy it in console directory. Then in the file you need to setup the database according to your env file:

postgres:
    image: postgres:13.4
    container_name: helium_postgres
    restart: always
    environment:
        - POSTGRES_DB=helium_console
        - POSTGRES_PASSWORD=xxx
        - POSTGRES_USER=helium_console
    ports:
      - 5432:5432
    volumes:
      - "/opt/console/data/postgres/:/var/lib/postgresql/data"
...
router:
...
    ports:
      - 3100:3000  # to export prometheus metrics
      - ...
    volumes:
      - "/opt/console/data/router:/var/data"

• Create directory for data

[~] mkdir /opt/console

• Other settings

When a user is creating an account in the console, it automatically get a free amount of DC. By default it is 10.000 ($0,1) free communication. If you want to change this amount, you need to modify the code as it is hardcoded in it. Take a look in file lib/console_web/controllers/organization_controller.ex for the console v1 version.

From console v2, you can add the following config value in the .env file:

INITIAL_ORG_GIFTED_DC=10000

Some other setting should be in the .env file

SELF_HOSTED=true  
USER_INVITE_ONLY=true/false (if you want to filter new users)
DISABLE_USER_BURN=true/false (if you want to disable DC self crediting)
BLOCKCHAIN_API_URL=https://api.helium.wtf/v1 (if you have your own)

And in the .env-router file

# Set the timeout to wait for different hotspot replicate of data 
# frame
ROUTER_FRAME_TIMEOUT=500
# turns on/off the DNS/IP verification for integration
ROUTER_HTTP_CHANNEL_URL_CHECK=false

Build the containers

Since March 2022, you don’t need to build your console container anymore and you can use the one distributed by helium on quai.io. In the template directory you will find a docker-compose-quay.yaml you can use to directly run the console. This is the preferred way to set this up. I keep the following lines explaining how to compile for Internet history.

In the console directory tree run the docker compose script to build the container

[~console] /usr/local/bin/docker-compose build

Run the containers

[~console] /usr/local/bin/docker-compose up

The creation takes some time … (a while) so … At end, the application is running. You can stop it with a CTRL+C.

Quick access to the log later:

[~] docker logs --follow helium_router

In case you want to rebuild all that stuff (basically destroy everything)…

[~] docker system prune -a # this is destroying all what exists in docker
[~console] rm -rf ./data/postgres # this is destroying the database 

I also had some issues when changing the .env-router file content to get the router container updated. I had to:

[~] /usr/local/bin/docker-compose stop
[~] /usr/local/bin/docker-compose rm router
[~] /usr/local/bin/docker-compose up -d

Load A blockchain snaPshot

This step is not mandatory but the blockchain sync car be a bit long. So you can load a snapshot from a validator.

Get the snapshot from helium, a new one is generated every 720 blocs so it’s better to load it right after the generation:

[~] wget https://snapshots.helium.wtf/mainnet/latest-snap.json
[~] cat latest-snap.json
{ ... ,"height":1366561}
[~] wget https://snapshots.helium.wtf/mainnet/snap-1366561.gz  

Move the file to the router. The file is moved to the console/data/router directory. Then you can load it on the router:

[~] docker exec helium_router router snapshot load /var/data/snap-xxxxxx.gz

Once done you can check the chain height and verify it is near the one of your running miner.

[~] docker exec helium_router router info height

After loading a snapshot, you need to restart the router container

[~] docker restart helium_router

Backup you router swarm_key

The router use a swarm_key (like the miner) for authentication. This file is like the wallet.key for the wallet. So you need to save it in case you need to restore the router somewhere else. This file is located in:

[~console] data/router/blockchain/swarm_key 

Backup it !

Start console & router

Once everything is OK, you can run it in background instead of foreground

[~console] /usr/local/bin/docker-compose up -d
# later you can stop it with
[~console] /usr/local/bin/docker-compose stop
# end restart with
[~console] /usr/local/bin/docker-compose start

You can make some basic check after 5 minutes. Like ensuring you have some peers.

[~] docker exec helium_router router peer book -s
+--------------------------------------------+-------------------+
|                  address                   |       name        
+--------------------------------------------+-------------------+
|/p2p/<span style="color:#a30089" class="tadv-color">11wcuSbKLHDLE3XX3FbKC7KF6bgAHUDqPzv1FTE</span>|refined-...........| 
+--------------------------------------------+-------------------+

+--------------------------+
|listen_addrs (prioritized)|
+--------------------------+
|/ip4/185.3X.1XX.X/tcp/2154|
+--------------------------+

+------------------+---------------------+--------------------------
|      local       |       remote        |                   p2p    
+------------------+---------------------+--------------------------
|/ip4/172.21.0.4/tc|/ip4/24.251.6.1XX/tcp|/p2p/1122P9dzzSDcmgBxLBNj1
|/ip4/172.21.0.4/tc|/ip4/2.70.2X.4/tcp/44|/p2p/112AKGW4qMeVcwBk2KpXC
|/ip4/172.21.0.4/tc|/ip4/73.71.1XX.142/tc|/p2p/112mEVXHvkYTpHJcE5SGQ
|/ip4/172.21.0.4/tc|/ip4/74.89.1XX.175/tc|/p2p/117DmoqANL8aMJcFKjNXz
|/ip4/172.21.0.4/tc|/ip4/185.1XX.184.16/t|/p2p/11QxjZpR4Xbzb6mpjGo1F
|/ip4/172.21.0.4/tc|/ip4/194.152.2XX.36/t|/p2p/11hDybc5G2WXPHKj5ePjw
|/ip4/172.21.0.4/tc|/ip4/173.31.1XX.201/t|/p2p/11jSj6M9y5z3JTG1J3BWe
+------------------+---------------------+--------------------------

If you don’t see an IP address on the listen_addr it may be because you are behind an infrastructure that hide your public IP. In this case you need to force it manually by adding the corresponding information into .env-router file

# Set Routers NAT info
ROUTER_NAT_INTERNAL_IP=XX.XXX.XX.XXX # in the ex above 172.21.0.4
ROUTER_NAT_INTERNAL_PORT=2154
ROUTER_NAT_EXTERNAL_IP=XX.XX.XX.XX # the public IP
ROUTER_NAT_EXTERNAL_PORT=2154

Associate this router to the OUI created

Now, we need to associate the router to the OUI we have created. In this step we are going to use the wallet used to create the OUI and the router P2P address (in red on the previous figure. Replace the –oui X by the number of your OUI.

[~router] docker exec helium_router router peer addr
/p2p/11wcuSbKLHDLE3XX3FbKC7KF6bgAHUDqPzv1FTEJHDIOE
[~wallet] ./helium-wallet oui update routers --oui X --nonce 1 --address 11wcuSbKLHDLE3XX3FbKC7KF6bgAHUDqPzv1FTEJHDIOE --commit

The transaction execution can be monitored on explorer using the ID of the wallet registering the router: https://explorer.helium.com/accounts/13HWrdkMoooYJEYXXXXXXXXXXXZifCzwxPaCnV9XXXbC5Gso1ic

Add credit to your router

Ok, some element you need to understand: when a packet is sent to the router, it will accept of reject it. If the router accept it, it will have to pay the communication in DC. I means the DC are not payed by the device itself or the end-user. It is payed by the router (OUI owner). So the final user does not really own DC.

For this reason, we need to credit the router (not the wallet used for creating the OUI). This can be done by burning HNT from the wallet with the router as a payee. The router is creating a 2 state_channel with for each a reserve of 2 x ROUTER_SC_OPEN_DC_AMOUNT each. The maximum duration of the state channel will be ROUTER_SC_EXPIRATION_INTERVAL blocks (about minutes). So assuming you have ROUTER_SC_OPEN_DC_AMOUNT = 50000 and ROUTER_SC_EXPIRATION_INTERVAL = 5000 and you want the router to have DC for a month, you need to credit about 270k DC.

./helium-wallet burn --amount 9.9 --payee 11wcuSbKLHDLE3XX3FbKC7KF6bgAHUDqPzv1FTEJHDIOE --commit

You can now (wait a bit then) check the router in the explorer to see the DC owned and the state_channel_open transaction on https://explorer.helium.com/accounts/11wcuSbKLHDLE3XX3FbKC7KF6bgAHUDqPzv1FTEJHDIOE :

Now, your router is ready to proceed packets.

Expose the console on Internet

Now you need to access the console. Instead of directly exposing the port 4000, we are going to pass this traffic through a nginx and install a ssl certificate. Currently it’s really a challenge to have the console on a specific path so it is really recommended to have it on a subdomain like console.foo.com

Install NGINX

[~] sudo yum install epel-release
[~] sudo yum install nginx
[~] sudo systemctl enable nginx
[~] sudo systemctl start nginx

Configure NGINX

Create some directories ; don’t forget to give them the correct authorization when using selinux

[~] sudo mkdir -p /www/html
[~] sudo mkdir -p /www/logs
[~] sudo chown nginx:nginx /www/html /www/logs
[~] sudo setsebool -P httpd_can_network_connect 1
[~] sudo chcon -Rv --type=httpd_sys_rw_content_t /www/html
[~] sudo chcon -Rv --type=httpd_log_t /www/logs

Add a virtual host configuration file in /etc/nginx/conf.d/helium_console.conf

upstream helium_console {
   server localhost:4000;
}
server {
   gzip on;
   gzip_types text/plain text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript application/json;

   server_name helium-iot.eu;
   root /www/html;

    error_log /www/logs/error.log;
    access_log /www/logs/access.log;
 
   listen 80;
   listen [::]:80;

   location ~ ^/socket/ {
        proxy_pass http://helium_console;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }

    location  ~ {
         proxy_pass http://helium_console$request_uri;
         proxy_redirect          off;
         proxy_set_header        X-Real-IP $remote_addr;
         proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
         proxy_set_header        Host $http_host;
    }
}

Then restart nginx

[~] sudo systemctl restart nginx

Then let’s make certbot do the https configuration

[~] sudo cerbot --nginx

Now you can create your console account and then you are connected.

Connect your first device

You can follow the device attachment I’ve been described in my previous post “first steps with Helium IoT network” to add your device in the console.

Devices are not immediately active: a blockchain transaction is needed to make the Hotspots able to route them to the right router. This operation is scheduled on every 10 minutes at the router level when ROUTER_XOR_FILTER_WORKER=true

You can accelerate this by manually executing a transaction to add the devices:

docker exec helium_router router filter update --commit

This transaction has a cost 40.000DC ($0,40), it inserts all the pending devices. Once done, the device will be able to connect:

Configure monitoring

Helium router can be monitored with prometheus & grafana. You need to add the corresponding docker container with the configuration provided by helium in the router repository.

Add the prometheus definition and node exporter. Node exporter put the standard server metrics to prometheus.

  prometheus:
    image: prom/prometheus
    container_name: helium_prometheus
    network_mode: host
   restart: always
    volumes:
      - /helium/run/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - /helium/run/prometheus/data:/prometheus
  node_exporter:
    image: quay.io/prometheus/node-exporter:latest
    container_name: helium_node_exporter
   restart: always
   command:
      - '--path.rootfs=/host'
      - "--collector.disable-defaults"
      - "--collector.filesystem"
      - "--collector.netstat"
      - "--collector.meminfo"
      - "--collector.cpu"
      - "--collector.loadavg"
    network_mode: host
    pid: host
    restart: unless-stopped
    volumes:
      - '/:/host:ro,rslave'
  grafana:
    image: grafana/grafana-oss
    container_name: helium_grafana
    network_mode: host
    restart: always
   volumes:
      - /helium/run/grafana/data:/var/lib/grafana
      - /helium/run/grafana/provisioning:/etc/grafana/provisioning
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=admin
      - GF_USERS_ALLOW_SIGN_UP=false

We need to get the prometheus.yml file. Check the port used in the file and make sure is is 3100:

global:
  scrape_interval: 10s
  external_labels:
    server: helium-router

scrape_configs:
  - job_name: router
    static_configs:
      - targets: ['localhost:3100']
  - job_name: node
    static_configs:
      - targets: ['localhost:9100']

Then create the directories for persisting all data:

[ root ~ ] mkdir -p /helium/run/prometheus/data
[ root ~ ] chown nobody:nogroup /helium/run/prometheus/data
[ root ~ ] mkdir -p /helium/run/grafana/data
[ root ~ ] chown 472:root /helium/run/grafana/data
[ root /helium/run/prometheus/] wget https://github.com/helium/router/raw/master/prometheus-template.yml
[ root /helium/run/prometheus/] mv prometheus-template.yml prometheus.yml

Now you can connect to the grafana console on port 3000. Default user is admin / admin. You need to do some config in grafana:

• Connect the Prometheus data-source by going to data-source, add new, Prometheus type. Then enter the Prometheus endpoint: http://localhost:9090 and click on Save & Test button.
• Add the Helium panel by clicking on (+) then Import and copy / paste the helium json file from routeur project. Select the Prometheus data source. I had to really modify the json file to make it working so you can get my version here.

Normally, after these setup, you should get access to the grafana dashboard

Upgrade console & router

Please note that some updates will require a database upgrade, there is no script for this yet as much as I know, so your can have some trouble to manage after an update. Take a look on github migration scripts to check if some db modifications has been made since your last upgrade.

It’s critical to maintain the router up-to-date, for this you can run it manually on regular basis:

[~] docker pull quay.io/team-helium/router:latest
or
[~console] /usr/local/bin/docker-compose pull

Or script it (thank you Mikael) and cron it

#!/bin/bash
docker pull quay.io/team-helium/router:latest | grep -q 'is up to date' || /usr/local/bin/docker-compose up --detach router

Update the console:

• save your configuration files
  • config/release.exs
  • docker-compose.yaml
• update the repository

git fetch
git checkout origin/master config/releases.exs
git checkout new.tag.for.update

• replace the config/release.exs and docker-compose.yaml, verify they have not been upgraded, if they are, you need to resync. The docker-compose file is in the template directory.
• Make sure the configuration files in template directory did not include new variables, in that case, update the .env and .env-router files
• rebuild

[~] /usr/local/bin/docker-compose build

You may be ready or restarting

The database should be automatically upgraded when the console container starts but I experienced some issues on that point and had to do it manually. Basically the migration scripts are executed one by one at console start ; when a script crash the migration is stopped and the next scripts are bypassed. This can basically be a problem.The database upgrade scripts are listed in directory /console/priv/repo/migration/. You can remove the scripts crashing because they have already been executed.

To re-execute the migration scripts, you can enter the container and call the script directly:

[~] docker exec -it helium_console /bin/bash
bash-5.1$ /app/_build/prod/rel/console/bin/console eval "Console.Release.migrate"

Scripts collection are located in /app/priv/repo/migrations inside the container. The migration scripts uses ecto_sql module.

If you do not manage to fix that automatically, you need to add the corresponding migration entries in the schema_migration in the psql database. This is a bit obvious but it will fix the initial crash by avoiding the scripts to rerun.

Debug your router

You can connect to the router internal for some operations like getting the router DC balance

[~] docker exec -it helium_router _build/default/rel/router/bin/router remote_console
(router@127.0.0.1)1>

You can also use predefined command for this:

docker exec -it helium_router _build/default/rel/router/bin/router ledger balance <your router Wallet>

You also have some docker command you can use:

# See all the account balance on your router
[~] docker exec -it helium_router _build/default/rel/router/bin/router organization info all

# see all the declared devices on your router
[~] docker exec -it helium_router _build/default/rel/router/bin/router device all

# .. see other command on github/helium/router README.md file or
[~] docker exec -it helium_router _build/default/rel/router/bin/router --help

You can also directly connect to the database to get the list of the devices

# docker exec -it helium_postgres psql helium_console helium_console  --command="select id from devices;"

When you want to monitor the xor filter execution

# docker exec helium_router router filter update
 -- DRY RUN --  
 - Estimated Cost: 0 
 - Adding 0 devices 
 - Removing : (filter, num_devices)ok

Know when the next filter update will be running

# docker exec helium_router router filter timer
Running again in T- 2m 31s

Trace a particular hotspot activity

In case you need to trace a specific hotspot, you can connect into the docker container and run a trace, replace the XXXX by the device ID you want to track:

[~] docker exec -it helium_router router remote
f(),
F = fun() ->
    HotspotB58 = "XXXX",
    PubKeyBin = libp2p_crypto:b58_to_bin(HotspotB58),
    io:format("Hotspot=~s bin=~p ~n", [HotspotB58, PubKeyBin]),
    Devices = router_device_cache:get(),
    [
        io:format("DeviceID=~s OrgID=~s ~n", [router_device:id(D), maps:get(organization_id, router_device:metadata(D), undefined)])
        || D <- Devices, PubKeyBin == router_device:location(D)
    ]
end,
F().

Debug a specific device communications

In case you need to debug the router behavior for a given device, there are some specific command on the router creating a specific device’s log file.

[~] docker exec -it helium_router _build/default/rel/router/bin/router device trace --id=c563xxx-dc1x-4fxx-92xx-33axxxxx

Then you will get in the router log directory a new folder named traces where you have a file created with the device id first digits name.

The trace seems to automatically disable after 4 hours it can also being stopped with:

[~] docker exec -it helium_router _build/default/rel/router/bin/router device trace stop --id=c563xxx-dc1x-4fxx-92xx-33axxxxx

Add new devAddr to your OUI

./helium-wallet oui update request-subset --size 8,16,32... --oui <oui_id> 

In case you get a invalid_nonce failure in the transaction, you need to set the nonce value as a parameter –nonce xxx to know the right nonce value, use the https://api.helium.io/v1/ouis api, get your current nonce and add 1.

Purge the console database historical statistics

The console database is getting trace of different network events and growth month after month. You can reduce this size with the prune scripts available in console repository. In this script you will find the plsql function to add in your database to purge the tables and the sql request to execute them and clean the database.

CREATE OR REPLACE FUNCTION prune_events() RETURNS void AS $$
BEGIN
  EXECUTE format($f$delete from events where inserted_at < NOW() - INTERVAL '7 days'$f$);
  RAISE NOTICE 'Pruned events table';
END;
$$ LANGUAGE plpgsql;
SELECT prune_events();

CREATE OR REPLACE FUNCTION prune_device_stats() RETURNS void AS $$
BEGIN
  EXECUTE format($f$delete from device_stats where inserted_at < NOW() - INTERVAL '30 days'$f$);
  RAISE NOTICE 'Pruned device stats table';
END;
$$ LANGUAGE plpgsql;
select prune_device_stats();

CREATE OR REPLACE FUNCTION prune_hotspot_stats() RETURNS void AS $$
BEGIN
  EXECUTE format($f$delete from hotspot_stats where inserted_at < NOW() - INTERVAL '25 hours'$f$);
  RAISE NOTICE 'Pruned hotspot stats table';
END;
$$ LANGUAGE plpgsql;
select prune_hotspot_stats();