The other day, I had an opportunity to build a project leveraging the power of streaming data manipulation in a fantastic platform called Flowthings (flowthings.io) and the amazing real-time visualization power we’ve all come to know and love at Initial State.
The goal was simple: I’ve got a couple smart devices that are sending data to an Internet gateway device which in turn is streaming via various protocols to Flowthings and now I need to make a few edits, perhaps do a little messaging and then see it in a pretty, effortless, real-time dashboard for sharing and analysis. The beauty of Flowthings in this scenario is two-fold: 1) protocol support – it doesn’t matter if data is coming in via HTTP, MQTT, or even RSS; and 2) real-time data manipulation/visual data pipe-line editing.
Starting Point: In this particular implementation I’ve got 3 devices streaming data to Flowthings currently: a SmartSensor, a SmartSwitch (wall outlet switch) and a Water Presence Detector. They’re sending data every second to a flow in Flowthings. So, the data is flowing right along. Now time to make something useful out of the stream.
Inside Flowthings web application, I can access the flow and see that I have all my data going to a child of that flow called default. Inside here, I can see the following data streaming in:
As you can see in the middle column, I have a stream of data which in Flowthings jargon is a flow of drops. Each drop represents a payload of data that has been sent. In the right-most column, you can see a little bit about the elements in the drop payload towards the bottom. This is a visual parsing of the JSON drop payload. Inside each individual drop, I have 3 devices producing multiple data points each. This is where I need the power of Flowthings data manipulation to pull the data points out into separate tracks, enact some business logic, and send them over to Initial State. Flowthings calls this functionality a track.
Creating a Flowthings Track
In the Tracks section, I can see that there are two tracks currently enabled for this flow. I created the Initial State Track to handle the data manipulation and forwarding along to Initial State. To create this track, I used Flowthings’ graph editor. This is a useful, visual way of manipulating the drop through this track. I wanted to be able to send different parts of the data to different Initial State buckets as well as do some fun visual things like insert emoji tokens in for certain data events. Here is a view of the final track.
The track seems to pretty intuitively explain itself, but just in case, I’ll walk you through what I’m doing here. First, the track is read left to right starting at the “source” box. The first thing I’m doing is splitting the source into three different tracks. While splitting seems like a very simple task, it provides a lot of data manipulation power in a streaming application. The “split” box is doing nothing more than making a three way copy of the stream to forward along to the next boxes where I’m actually applying some filtering logic.
A few notes here:
- I didn’t attempt to do any error handling here to keep the example simple
- I’m taking some of the data and, based on its value, I’m changing what I pass to Initial State to include some visually enticing emojis.
By looking at the JSON, we can see that all the properties are accessible through `drop.elems.[json-from-here]`.
So, now that I’ve split the drop into three different sub-tracks within my track and then filtered the drop information down from the full JSON payload to a JSON serialized string, I then pass that string to the next block which performs an HTTP request to Initial State. The configuration for this block looks something like this:
The final step is to send the response from this HTTP POST to a null terminating box. This ties off the loose ends for the track and indicates that we don’t really care about the response from the HTTP POST because we’re not doing error handling in this example and a successful response is an HTTP 204 which is empty.
The Smart Sensor sub-track has an additional split that allows me to pass the data in to two different Initial State buckets, one that provides all the sensor information as a summary and another that is sending that Sensor information into a bucket containing other environmental sensors that were being streamed from a different project. As you can see, there is a lot of power in the integration of both Flowthings and Initial State. Here is an example of the resulting Tiles view of the work performed above:
- When needing multi-protocol support like Websockets, HTTP, MQTT, RSS, and more to come, Flowthings provides a fantastic solution
- The transformations in the pipeline didn’t bottleneck at Flowthings, the data flowed very smoothly from device, to flowthings, through transformations, and on to Initial State.
- When needing to perform data manipulation, filtering, splitting, real-time calculations, etc. before sending data to Initial State, Flowthings provides a great tool.
- If you have a flow of data and you’d like to share it, doing so within Flowthings is super simple. Additionally, if you’ve got a great track streaming data to your own Initial State account and want to share that visualization, it’s also super easy! Conclusion: data and visualizations with Flowthings and Initial State can easily leverage collaborative mindshare.