Hands on

Tracking a Raspberry Pi with WLAN and Golang, then Displaying the Results with HERE XYZ

By Nic Raboy | 06 December 2018

Over the past few weeks I’ve been exploring how to track device position using nothing more than the meta information of nearby wireless access points. For example, in my tutorial titled, Tracking Device Location with Golang, WLAN Access Points, and the HERE Positioning API I demonstrated how to use Golang and the HERE Positioning API to get a a position estimate of a Mac or Linux device. However, in this example it was very sandboxed and didn’t serve a huge use-case.

What if we wanted to continuously track location and then make sense of the data using a tool with a user interface?

In this tutorial we’re going to see how to continuously try to position a Raspberry Pi device and then map the data using HERE XYZ Studio so it can be further analyzed.

Before we start development, I want to share a further breakdown of what I accomplished and hope to share. Take the following picture:

raspberry-pi-portable

I have this Raspberry Pi 3 which runs Linux and is powered by the same battery pack I use to recharge my mobile phone. After installing the Golang application that we’re going to build, I started to roam around my neighborhood. Did I look like a crazy person carrying this thing around the neighborhood? I sure did, but I didn’t care.

After spending a good 15 minutes outside, I came back in and harvested the data from my Raspberry Pi. The Raspberry Pi had been collecting WLAN information from around the neighborhood and had been sending it to HERE for a position whenever it had an internet connection. The data I later harvested was the latitude and longitude information.

I then uploaded the latitude and longitude information to HERE XYZ Studio to make sense of it:

xyz-studio-raspberry-pi-data

Latitude and longitude data isn’t the most pleasant to look at, so a visualization tool is great. Looking at the plotted points of my neighborhood journey, I can see that my Internet of Things (IoT) experiment was successful. Given that the WLAN positioning is only an estimate, the coverage was still in the area I traveled.

Creating a Golang Application with Boilerplate Code

The first step of this project is to get some boilerplate Golang code in place. This code should include our data structures as well as any imports that will be used. Assuming you have Go installed and configured, create a main.go file somewhere in your $GOPATH path that includes the following:

package main

import (
    "bytes"
    "encoding/json"
    "fmt"
    "io/ioutil"
    "net/http"
    "net/url"
    "os/exec"
    "regexp"
    "runtime"
    "strconv"
    "strings"
    "sync"
    "time"
)

type HerePlatform struct {
    AppId   string `json:"app_id"`
    AppCode string `json:"app_code"`
}

type PositionResponse struct {
    Location struct {
        Latitude  float64 `json:"lat"`
        Longitude float64 `json:"lng"`
        Accuracy  int     `json:"accuracy"`
    } `json:"location"`
}

type WlanInfo struct {
    BSSID string `json:"mac"`
}

type AccessPoints struct {
    Wlan []WlanInfo `json:"wlan"`
}

type AccessPointData struct {
    mux  sync.Mutex
    data []AccessPoints
}

var waitGroup sync.WaitGroup
var platform HerePlatform
var accessPointData AccessPointData

func main() {
    fmt.Println("Starting the application...")
}

The HerePlatform data structure will hold our application id and application code values obtained from the HERE Developer Dashboard. These values will be used when making requests to the HERE Positioning API. The PositionResponse data structure is what is returned from the HERE Positioning API. It will be what we use when plotting our points in XYZ Studio. The goal of our scanner is to pick up the BSSID values of WLAN access points, hence the WlanInfo data structure. When we send the WLAN information, we need to format it like the AccessPoints data structure. Finally, since we will be scanning and tracking position in parallel, we need to create an AccessPointData data structure with a sync.Mutex in it. Since two threads will be accessing our scan data, we need to lock it while it is in use, hence the sync.Mutex.

With the foundation of our application in place, we can start adding logic.

Developing a WLAN Access Point Scanner with Golang

The second step in being successful with this tutorial is to implement some network scanning functionality with Golang. Since this tutorial is more of a next step in a series of tutorials, I strongly encourage you to read Tracking Device Location with Golang, WLAN Access Points, and the HERE Positioning API and Searching Location with the Command Line, WLAN Information, and the HERE Positioning API if you haven’t already.

Take the following WlanScanner function:

func WlanScanner() {
    defer waitGroup.Done()
    for {
        if runtime.GOOS == "linux" {
            iwlistCmd := exec.Command("iwlist", "wlan0", "scan")
            iwlistCmdOutput, err := iwlistCmd.Output()
            if err != nil {
                fmt.Println(err)
                return
            }
            lines := strings.Split(string(iwlistCmdOutput), "\n")
            var wlans []WlanInfo
            for _, line := range lines {
                columns := strings.Fields(line)
                if len(columns) == 5 {
                    match, _ := regexp.MatchString("^([0-9A-Fa-f]{2}[:-]){5}([0-9A-Fa-f]{2})$", columns[4])
                    if match == true {
                        wlan := WlanInfo{
                            BSSID: columns[4],
                        }
                        wlans = append(wlans, wlan)
                    }
                }
            }
            accessPointData.mux.Lock()
            accessPointData.data = append(accessPointData.data, AccessPoints{Wlan: wlans})
            accessPointData.mux.Unlock()
        } else {
            fmt.Println("An unsupported operating system was detected.")
            return
        }
        time.Sleep(2000 * time.Millisecond)
    }
}

The WlanScanner function will be called as part of a goroutine which is a type of parallel operation. When and if the goroutine ever finishes, it will tell the main thread that it has done so via the waitGroup.Done().

For as long as this function is running, it will check to make sure we are using Linux. If we’re using Mac or Windows, we’ll exit because the goal is to use a Raspberry Pi which uses Linux. If we’re using Linux, we can use the operating system level iwlist command to scan for networks. With the network information found, we can parse it for the BSSID values as demonstrated in the previous tutorials. If you’re a champion with regular expressions and parsing, you might try to collect the signal strength for more accurate position estimates.

When the WLAN information has been collected, our accessPointData variable is locked, data is appended to it, then it is unlocked so it can be accessed by other goroutines. To prevent scanning more than we need to, we can sleep the goroutine. I’ve chosen to sleep the goroutine for 2 seconds between operations, but you might sleep it for longer.

The main function can be updated to the following:

func main() {
    fmt.Println("Starting the application...")
    waitGroup.Add(1)
    go WlanScanner()
    waitGroup.Wait()
    fmt.Println("Done")
}

In the above function, we are saying that there is one item to wait on and the goroutine is started. When we’re no longer waiting, we can then close the application. This prevents the application from closing prematurely. At this point, the application will continuously scan for networks every two seconds.

Determine Position Estimates from WLAN Data with the HERE Positioning API

Now that we have scanning happening, we can make periodic requests to convert the scans into positions. However, before we make those periodic requests, let’s define how we’re going to use the HERE Positioning API with Golang.

Let’s create the following position function:

func (platform *HerePlatform) position(payload []byte) (PositionResponse, error) {
    endpoint, _ := url.Parse("https://pos.api.here.com/positioning/v1/locate")
    queryParams := endpoint.Query()
    queryParams.Set("app_id", platform.AppId)
    queryParams.Set("app_code", platform.AppCode)
    endpoint.RawQuery = queryParams.Encode()
    response, err := http.Post(endpoint.String(), "application/json", bytes.NewBuffer(payload))
    if err != nil {
        return PositionResponse{}, err
    } else {
        data, _ := ioutil.ReadAll(response.Body)
        var positionResponse PositionResponse
        json.Unmarshal(data, &positionResponse)
        return positionResponse, nil
    }
}

Using a WLAN payload and user defined application id and application code values, we can make a POST request to the API. The response will be a latitude, longitude, and accuracy radius in meters.

With an API function defined, we can create our goroutine which will periodically make requests:

func DevicePositioner() {
    defer waitGroup.Done()
    for {
        time.Sleep(5000 * time.Millisecond)
        accessPointData.mux.Lock()
        if len(accessPointData.data) > 0 {
            if len(accessPointData.data[0].Wlan) > 0 {
                _, err := http.Get("http://clients3.google.com/generate_204")
                if err == nil {
                    payload, _ := json.Marshal(accessPointData.data[0])
                    positionResponse, err := platform.position(payload)
                    if err == nil {
                        data, _ := json.Marshal(positionResponse)
                        fmt.Println(string(data))
                        accessPointData.data = accessPointData.data[1:]
                    }
                }
            } else {
                accessPointData.data = accessPointData.data[1:]
            }
        } else {
            fmt.Println("No WLAN data to zero in on!")
            break
        }
        accessPointData.mux.Unlock()
    }
}

Like with the other goroutine, this will also potentially run forever, assuming there is data to work with. We want to wait a little longer than the scan so that way there is data to work with. This is why we sleep at the start rather than the end.

In every iteration, we lock the slice that has our data to avoid deadlocks and corruption. If we have data and that data isn’t null, we need to make sure we have a connection to the internet before attempting to make a request to the API. If everything checks out, we can create a payload from our WLAN information. If our request is successful, we are going to remove the data from our queue. Remember, we’ll potentially have more scan data than requests for positioning, so we use it as a queue.

With the goroutine finished, we can update the main function to the following:

func main() {
    fmt.Println("Starting the application...")
    waitGroup.Add(1)
    go WlanScanner()
    waitGroup.Add(1)
    platform = HerePlatform{AppId: "APP-ID-HERE", AppCode: "APP-CODE-HERE"}
    go DevicePositioner()
    waitGroup.Wait()
    fmt.Println("Done")
}

We’ve just started our other goroutine and defined our HERE platform information. If you felt like I did a lot of handwaving in this tutorial thus far, it is because I have. I strongly encourage you to check out my previous tutorials in this series to get all the information around scanning and making requests to the API. As of now, the only difference is that we’re using goroutines to continuously run each of these operations.

After you’ve build a binary for your Raspberry Pi, you can run it with the following:

sudo nohup ./raspberry-pi-tracker > output.out &

The above command will run it in the background and send all output to an output.out file. The output will contain the data we need for XYZ Studio. If you’re having trouble cross-compiling the application for the Raspberry Pi, check out a previous tutorial I wrote titled, Cross Compiling Golang Applications for use on a Raspberry Pi.

Making Sense of Custom Latitude and Longitude Data with HERE XYZ Studio

If everything thus far has been successful with your Go application and Raspberry Pi, you should have several lines of data in the following format:

{"location":{"lat":10.713308,"lng":3.4509832,"accuracy":88}}

Unfortunately, this format is not a supported format by XYZ Studio. We need to be using CSV or preferably GeoJSON format. We could have designed this output in Golang, but that is a topic for a different day. Instead, you can make use of an online conversion utility or do it yourself.

When I did my conversions, I used two different online tools, none of which are officially supported by HERE. First, I took my output and ran it through a JSON to CSV tool by Eric Mill. I then changed the format of the CSV to the following:

lat,lng
10.713308,3.4509832

With the CSV data in hand, I ran it through another converter to make it GeoJSON format. The CSV to GeoJSON tool gave me what I needed to visualize.

In HERE XYZ Studio, I chose to create a new project and add a new dataset via a file upload. After uploading the GeoJSON dataset, points were immediately displayed on the map, giving me something to make sense of.

The beauty of XYZ Studio is that I can give my data to someone non-technical and they can analyze it very easily.

Conclusion

You just saw me walk through a simple project that I was experimenting with. I had written a Golang application, installed it to my Raspberry Pi, wandered my neighborhood to collect WLAN information, exchanged the WLAN information with coordinates, then plotted those coordinates on a map with XYZ Studio.

The goal was to demonstrate how to use the HERE Positioning API and how easily HERE XYZ Studio could be used. The concepts of this tutorial can be taken beyond Raspberry Pi and into other IoT applications. For example, what if fitness trackers that didn’t include GPS started to track your runs via WLAN information. I personally think that would be pretty awesome.