It’s just been declared in Quebec that “snow sports instructors” are essential workers (?), so it looks like I am going to be busy from Sunday onwards, although we have rain forecast for Xmas so who knows.

Something has happened with the Ottawa data resulting in this thrashing around to get things working again:

https://github.com/swfiua/blume/commit/5b1b5f81bd34dab87cb4c65cee7ecf1e9574ceb9

I can’t leave the magic like that…. I want to keep the magic simple, also want the magic to figure out stuff and do the right thing.

What am I trying to do with the magic.Spell part of blume?

The thing that turns data into blume tables?

I am increasingly needing to write something to generate “meta data”. Clues, if you like to help the blume programs interpret what is there.

I have a whole folder on my laptop of various folders of data from different sources, data from sensors on various raspberry pi’s I have had running at various points.

Most of the data is just csv files, but they are all messed up in different ways. The good news is it is mostly from code I wrote, so I know some of the ways it is broken.

The data from the sensehats is a good example.

The karmapi code just reads the sensors, records the values as a record to a file. It sounds simple, what could go wrong?

I have different folders for each year/month/day, with a bunch of files: weather, compass, gyro and accel.

The problem is that when the pi’s wake up, they don’t know where they are or what time it is.

And my code just tries to append to the files for the day. And sometimes it all goes wrong and the append ends up over-writing data and the header might be missing.

I also use a thing called cron to schedule a new task each day and combine it with the monitoring code exiting at midnight, it all sort of works – and for the pi’s that have stayed in one place for a while, the whole thing has worked pretty well.

… back to blume and the ocixx example.

I hadn’t run the code for a week or so, so I can’t pin when, but they changed from yyyy-mm-dd to yy-mm-dd for the dates in one of the data feeds – I am sure there is a story to why it changed.

Suffice to say it gave matplotlib’s magic date handling for axis labels some indigestion.

Dates always cause trouble eventually.

The observations are all good in the karmapi data, it is just that there is some uncertainty in the actual time and place of the event.

Dates and locations, the very things that define an event are the very thing we have difficulty with.

There is a clue in the timestamp that might help with time and place. There’s also weather data and data from surrounding times that might help adjust the window.

Some data comes with pictures too, if the pi has a camera, with consistent timestamps.

The fixes here were just a minimum to figure out what was up and then get it working so I can see the latest Ottawa plots again.

There are so many little changes like this data throws at us.

Maybe the meta data just gives us information about expected distribution of values?

I think what I am looking for is something that allows me to tune things, but somehow make those tunings available more generally – the way to do that might be to go up a level and re-generate the meta data, given your new knowledge.

And the probability that the year includes the century too?

Update

Just came across this old note that git wasn’t managing for me.

No actual magic yet. But things are warming up and it won’t be long until I have more time on my hands to think of reasons not to write code.

I have been skiing with what I call piski.

It is raspberry pi with a sense hat and a fairly low resolution, fish-eye camera.

It records all the data from the sense hat, currently ten times a second, as well as taking a picture once a minute.

And finally..

Here’s the docs from the blume.magic module.

Matplotlib plus async

mainloop()

queues of images

clients set up their own queues

draw from Q

Going round in ever decreasing circles.

The idea is there is some sort of calculation going on that generates grids of numbers.

You want to see what is going on without grinding the calculations to a halt.

At the same time, the more you explore the data with plots (thanks matplotlib) the more questions appear.

Often the calculation has a number of parameters, or a selection of data streams on which it could be run.

So my code sprouts command line arguments and then the simple U/I might allow me to control more of the parameters.

This module aims to provide a small number of components to help asynchronous data exploration and graphical monitoring of data processing pipelines.

It also aims to help with the process of maintenance of data streams over time. See metagit.py for more ideas there.

Many magic classes started as objects in other short scripts, see examples for some of those.

Internals?

I have gone with Tk because it is usually around and I don’t need much here.

But I will be thinking about raspberry pi’s with sense hats and a mini joy stick too.

All the user interface and display handling are ultimately done by the EventLoop object, so a place to start when thinking beyond the current.

For in put the focus is very much on keyboard, so things should work well with anything that can create a stream of key characters.

I am also using David Beazeley’s curio module to help me use python3.6+ async features.

I am starting off here with some pieces from my project karmapi.

So there will be Pig Farms, Piglets and widgets all mixed up for a while.

Update

I am planning to re-work how objects pass between Balls.

Balls await put() and get() messages and not care where they come from.

The plan is for the GeeFarm or Shepherd to allow you to examine queue statistics and add and delete edges to the graph, with an option whereby it randomly connects things together until it works or melts or something.

Make it a bit easier to view and navigate the various graphs.

Leaning to a bunch of different Controller objects that you can call up for different tasks such as viewing queues, starting and stopping processes.

Other objects, waiting on input.

In short, there is a working prototype.

It is time to untangle the various objects and mold it into something smaller and more powerful.

That we can then use in the farm module to build a functioning whole.

Let the balls bounce around randomly with the magic roundabout routing the photons.

class blume.magic.Axe(delegate, carpet)

A matplotlib axis that has some extra methods

The idea is to hand these out to anyone looking for an axis.

By wrapping the axis object we can carry around some meta data that might come in useful.

But mostly I want an Axis that I can show and hide from the figure and change its layout.

I would like to just go with show and hide, but suspect I might need draw too.

get()

Return a fresh axis

hide()

Hide the axes

please_draw()

Try to force a draw of the axes

position(target)

Set position to that of target

projection(name)

Set the projection

Not sure if this is possible.

show()

Show the axes

class blume.magic.Carpet

Current status: history just added, wormholes opened.

FIXME: figure out lifecycle of an Axe

generate_mosaic creates and adds to self.axes

Need to be able to know:

1. Axe has been handed out

2. Axe has been shown

3. Axe still in history

4. geometry – so we can spot Axe replacing another in same spot.

Cases:

handed out, still in history == keep handed out, not in history,

Deletion:

1. not in history

2. has been handed out

3. not in current image: ie self.showing

async add_row()

add a row to the mosaic

keypress(event)

Take keypress events put them out there

async less()

Show fewer pictures

async more()

Show more pictures

async poll()

Gui Loop

async save()

Save current image

toggle_expand(names=None)

Toggle making each axis fill its space

toggle_expand2()

ask figure to expand the axes to fill the space

class blume.magic.RoundAbout

Pass self around.

Just a collection of random queues that everyone shares.

For maximal information sharing, I’m curious what happens when objects with the roundabout do something like:

await self.put(self)

The magic roundabout just looks after the queues.

select(name=None, create=True)

pick a q

create: if True, create if missing – actually seems to create regardless, why not?

status()

Show some stats

class blume.magic.Shepherd

Watches things nobody else is watching

This currently sets up all the message handling and runs everything.

To route messages between balls it is seems to be necessary to have access to the graph of relationships.

In the current setup this is GeeFarm, but this leaves everything to the Shepherd.

The Shepherd in turn does it’s work by setting up relays for each edge in the graph, watching all the roundabout queues and passing messages along.

There is one relay per edge in the graph and it just connects the output and input queues of objects along that edge.

There is a whole other chunk of code dealing with passing keyboard events along a path of objects, and all the related key binding fun.

Again, the roundabout holds the data and every Ball has a Roundabout.

It might be good to have just one RoundAbout, managed by the Shepherd.

Dynamic graph?

doc_firstline(value)

Return first line of doc

async down()

Move focus to next node

async edit_current()

open code for current in idle

FIXME: what to do with no idle, eg in pyscript land?

async helper()

Task to run if you want help on the carpet

One day it will be easy to start and stop this.

async interact()

Go into interactive mode

hmm…. worm can time

async next_ball()

Move focus to next

path management.

async previous_ball()

Move focus to previous

async quit()

Cancel all the tasks

async run()

run the flock

Decide what to run.

Manage the resulting set of roundabouts.

Pass messages along.

set_flock(flock)

Supply the flock to be watched

async show_help(name='keys')

Show what keys do what

async start()

Start things going

FIXME: make it simple

async status()

Show current status of object graph

async toggle_run(sheep=None)

Toggle run status of sheep

async up()

Move up path

class blume.magic.Spell(cache=None)

A magic spell, or cast if you like, if it works

For help processing lists of dictionaries.

Or csv files, where we have to turn strings into values.

The game is guessing the types of the columns in a csv file.

Use a magic.Spell.

spell(data)

Apply casts to data

Would like this to be dynamic, updating the casts as we go

class blume.magic.Table

Magic table.

A list of dictionaries, and ways to explore them?

async blume.magic.canine(ball)

A sheep dog, something to control when it pauses and sleeps

runner for node.run and more

This was an an attempt to factor out some boiler plate from run methods.

so run has turned into “do one iteration of what you do”

and canine here is managing pausing and sleep

Now we could loop round doing timeouts on queues and then firing off runs.

With a bit more work when building things … TheMagicRoundAbout time?

Update: trying to accommodate balls where run is just a function.

Try to accommodate coroutines and couroutine functions.

blume.magic.fig2data(fig=None, background='grey')

Convert a Matplotlib figure to a PIL image.

fig: a matplotlib figure return: PIL image

There has to be an easier way to do this.

FIXME – turning matplotlib figures into PIL or numpy