Hands On

Integrating Location Data into Pipedream Workflows

By Raymond Camden | 23 July 2020

Try HERE Maps

Create a free API key to build location-aware apps and services.

Get Started

Over the past few months, I've become somewhat enamored with a new service called Pipedream. Pipedream is yet another serverless solution with a bit of a twist. It makes it easy to build workflows by piecing together various actions like LEGO bricks. These actions are small pieces of programming logic that cover a wide variety of different services. You can connect these actions together along with the ability to write Node code for your own logic.

For example, maybe you want to read in a Google Sheet, do sentiment analysis on the text, and then email a summary to a product team? Or perhaps monitor a RSS feed for new items and send a SMS message with an alert? Pipedream has an incredibly powerful, easy to use website that makes it easy to setup these workflows, share them with others, and modify when you find the need.

img1

I've been playing with Pipedream and have built many different workflows for my own needs. This week I created something that I think could be really powerful for people who take a lot of pictures - automatic geographic sorting.

Before I go into the code for this, let me demonstrate how this works. In my Documents directory, I've got a folder called Photos:

img2

I can then upload an image (or use the integrated operating system client):

img3

In a few seconds, the image automatically disappears... and reappears in a new folder, GeoSortedPhotos/Louisiana/Lafayette:

img4

And if I were to go back in time and find a photo from my visit to Galaxy's Edge, I see it sorted in Florida/Orlando:

img5

How did I create this incredible work of programming wizardry? By connecting a few Pipedream actions with a bit of integration of the HERE Reverse Geocoding service. All in all the process took roughly thirty minutes. Let me walk you through the steps.

Every workflow begins with a trigger, basically a way to define how the workflow starts. Pipedream supports many different types of triggers with a simple "here is a URL to hit" HTTP trigger being the simplest. On the more powerful side, Pipedream supports what are known as "event sources", these are triggers that act as responses to various services. One of them is the Dropbox source. I began by selecting this source, logging in to my Dropbox account, and then specifying the folder to check.

img6

Note the link to edit the code and configuration. Every Pipedream action, trigger, or event source you work with is 100% editable, letting you make tweaks if you need something custom.

When I had this event source trigger selected, I could immediately begin testing. Pipedream provides a way to look at all invocations of the workflow and inspect the data at each step. So, with just this one part in place, I was able to upload a photo and see how the data looked.

img7

See that "media_info" block in there? If we expand it, and keep expanding, we get to the location data:

IMG8

That's what I need! For my next step, I added a Node.js step. This is a generic "write the code you need" step. (As an aside, I plan on contributing HERE steps to Pipedream to make it even easier to use HERE APIs in Pipedream!) All I needed in my step was a bit of code to hit the Reverse Geocode endpoint.

async (event, steps) => {
    const fetch = require('node-fetch');
    const key = process.env.HERE_API_KEY;

    if(steps.trigger.event['.tag'] !== 'file') $end('wrong type = ' + steps.trigger.event['.tag']);
    let loc = steps.trigger.event.media_info.metadata.location.latitude + ',' + 
            steps.trigger.event.media_info.metadata.location.longitude;

    let url = `https://revgeocode.search.hereapi.com/v1/revgeocode?apikey=${key}&at=${loc}`;

    let resp = await fetch(url);
    let data = await resp.json();
    return data.items[0];
}

For the most part this is pretty simple usage of the API but pay special attention to steps.trigger.event['.tag]. One of the things I discovered was that the event source triggered for any file change in the folder, including deletes. When a file was added, the value was file so if the value is anything but that, I use a special Pipedream utility available $end to stop the workflow from continuing. I then fire off the call to reverse geocode the longitude and latitude. When done, I return the result.

Now, my intent is to take the location and move my pictures to a folder named /STATE/CITY/. I need a bit more Node.js code for that and instead of just adding code here, I built another step. While Pipedream doesn't care how you organize your workflows, I like to each function does just one task. So I made another step where the sole purpose was to figure out the destination folder:

async (event, steps) => {
    let destPath = `/documents/geosortedphotos/${steps.reversegeocode.$return_value.address.state}/${steps.reversegeocode.$return_value.address.city}/`;
    let filename =  steps.trigger.event.path_lower.split('/').pop();
    destPath += filename;
    return destPath;
}

Notice how I have access to the previous step using the variable steps. All your steps and their outputs are available to later steps. I also have access to the first trigger and use the path_lower value to get the original filename. The end result is a new path based on the state and city.

The last step was simple. There's a Dropbox action for moving files. I added that and literally just supplied two values:

IMG9

It may be terribly obvious, but the 'white box' after the values is meant to give you an idea of what the input will look like. As you use a Pipedream workflow, it learns what kind of data flows through it and begins to intelligently offer help to you in a way that's almost creepy in its accuracy.

And that's it! My workflow is incredibly brittle, but that's my fault, not Pipedream's. For example, I noticed what I assume has to be a security setting on my work laptop that images I downloaded had the location data stripped. The same photo downloaded on my personal machine didn't have this issue. I could easily add support for noticing this and ending the workflow early. Also, it obviously expects an American state and city. It wouldn't be hard to make this use a country/state (or province)/city approach. I could also support cases where a photo couldn't be located and move it into another bucket. I could then use another Pipedream workflow to notice changes to that folder and email a human to do organization manually.

One of the coolest aspects of Pipedream though is how easy it is to share workflows with others. You can see the entirety of my workflow here - https://pipedream.com/@raymondcamden/geosortphotos-p_8rCPgn/. None of my credentials will be exposed and even better, in one click you can copy this to your Pipedream account and start playing it.

Pipedream is currently free, and of course we offer a generous free tier if you sign up on our portal.