In a future post I’ll write down the reasons why, and probably release the development diaries. But for now, the source code for the server is available here: https://github.com/ggaughan/ThinkSQL

As I say in the README:

The source code is linear per se, but while writing it, it was an organic thing, generating great interwoven trees in the computer memory and in my head that were many-layered and that were modified and traversed, and impacted on a dynamic multi-versioned data store alongside many other threads, causing and needing deep psychological flow. The code comments are released as-is, and are often streams-of-consciousness.

(Instead, how about holding your arms apart to show them the actual size of the space and reduce the gap between your hands as they near the obstacle? That way they get immediate feedback and can see ahead of time how far they have left.)

pipe UuvYtuMe3hGDsmRgPm7D0g

Tony Hirst, a prolific Yahoo! Pipes user, had the idea to translate the pipe definitions into code so that they could be run on your own computer, in case the Yahoo! Pipes server was unavailable. This sounded like an interesting challenge so I developed pipe2py. The pipe2py package can compile a Yahoo! Pipe into pure Python source code, or it can interpret the pipe on-the-fly. It supports embedded pipes too. (Not all of the Yahoo! Pipes modules are available yet, but they’re gradually being added: if you find the need for one that’s missing please let me know, or better still provide me with the code for the module.)

The design for the compiled pipes was based on David Beazley’s work on building Python generators into pipelines, together with ideas from SQL query compilers and XProc pipelines. Each Yahoo! Pipes module is implemented as a Python generator which iterates over items provided by an input module and processes them to yield output results. Once these generators are connected together, iterating over the final one will initiate a cascading call to all earlier generators for them to iterate over their inputs and, in turn, yield their output. There are several benefits to this architecture:

1. the compiled pipeline closely matches the original Yahoo! pipeline
2. adding new modules is easy because they are loosely coupled
3. each item is typically passed through the whole pipeline one at a time, so:
   1. memory usage is kept to a minimum
   2. no module is waiting on an earlier module to finish processing the whole data set
4. by adding queues between the modules they could easily be made to run in parallel, each on a different CPU, to give great scalability

Here’s an example pipe2py session which converts the pipe shown above into Python and then runs it locally:

$ python compile.py -p UuvYtuMe3hGDsmRgPm7D0g

$ python pipe_UuvYtuMe3hGDsmRgPm7D0g.py

Name (default=Lancaster) Neill

{u'title': u'Bob Neill',

...

u'TotalAllowancesClaimedIncTravel': u'157332'}

Since pipe2py can compile pipes into Python modules, it seemed a good idea to try to run them in Google’s cloud via App Engine. So now there’s pipes-engine, which uses pipe2py to run your Yahoo! Pipes on Google’s servers.

You’ll need to log on with your Google account, and then you can take the Id of your Yahoo! Pipe (you can find it in the url when editing a pipe) and add it to the list. pipes-engine will then compile it and store the Python version of it. Clicking the pipe Id will run it on the App Engine. If you change the pipe in Yahoo, you can reload it in pipes-engine to re-compile the latest version (although I hope to automate this step in future).

There’s currently an App Engine timeout of 30 seconds, but Google have said that they are working on increasing that soon.

There were some tricky bits to developing this, like storing the generated Python source in the datastore and then importing it dynamically back from the datastore, and doing so recursively for any embedded pipe imports. Some Python PEP 302 magic helped here.

The pipes-engine.appspot.com service is a proof of concept and needs some more work, not least to provide the output in formats other than json, but I think it proves it’s feasible. Let me know what you think.

To hear why, first listen to some of “It’s a Sin” from 1987:

Now listen to the theme tune of that mini-series, “If Tomorrow Comes”, composed by Nick Bicât:

That was broadcast in 1986, the year before “It’s a Sin” was released.

I’m no musician, but they sound surprisingly the same to me.