kk
Gabriel Morell





ThingCosm

The overarching Thing-* project

Subprojects
Thingistry ThingEnvoy Thinghole
A namespaced registry
posted June 23, 2017, 2:17 a.m.
under thingistry · perma

TBD

Send the envoy
posted June 23, 2017, 2:17 a.m.
under thingenvoy · perma

While implementing a stream parser for JSON responses for each and every wireless microcontroller project I build seemed like a fantastic use of time, I figured that I could minify responses coming from large APIs and only pull out the pertinent information that the microcontroller would need to use. Thus was born the Envoy. The envoy sits between a microcontroller and a datasource and through a series of user configurable rules will pull out the relevant data.

The initial purpose for this was polling the old hubway API that would return all the stations in the system and their statuses which was in the tens of kilobytes and would exhaust my heap. The initial version worked well and there was value in making it generic.

On top of being a minifier, the original unfiltered payload is cached in memory. This is so different devices are able to poll the same Envoy instance at the same time and have their own unique filtering rules against the original payload while lowering the call count to the original service. Users are able to configure a TTL on their filter rules to uniquely control when the microcontrollers get fresh data.

ThingHole Intro
posted June 23, 2017, 2:17 a.m.
under thinghole · perma

ThingHole is the center of the ThingCosm and the first tool that was written. It was written with multiple instances of any service running in parallel from the start., thinghole has its base services written in twisted with a Django front-end for management, configuration, and data visualization.

The twisted services are:

  • Horizon which ingests data,
  • Stream which deals with passing the data along to the front-end
  • Manifold which specifies triggers to pass the data onto other services.

[arch]

The horizon

The horizon is a service that collects arbitrary data and inserts it into the database as well as passing it along to the other services such that they don't need to access the database. For speed reasons, the horizon service connects to postgres and loads all the auth tokens into memory and continues to update the in-memory cache every thirty seconds. On data received the horizon inserts the data into postgres with [code] as well as passes the data along to the various stream and manifold workers.

The stream

The stream service receives data from the horizon services and forwards them to connected web socket clients for charting and tabulation. It makes sure that it passes on the values to the correct connected client. The data is also converted into the various formats for each charting and tabulation feature.

The manifold

The manifold service passes on data to other services based on user described arbitrary rules, either to a crossbar topic, another thinghole reciever, or arbitrary URLs with programatic injection of additional data and headers

A cosmic Introduction
posted June 23, 2017, 2:16 a.m.
under thingcosm · perma

The ThingCosm is series of loosely connected tools for sending and receiving data to and from services. While similar solutions exist, I wanted a series of tools that were open source, standalone, and not beholden to any particular vendor. In the past I had been burned by proprietary tools disappearing or being bought out. My goal was to have the only person who could cause me trouble be either my hosting providers or myself.

From this motivation, I initially built three tools, The Hole an arbitrary data storage and graphing tool. The Envoy, an api minimizer for microcontrollers, and the Registry a place to store key value pairs within a namespace