“Welcome to InfoLink, the best way to organise your life.”

         █████ █   █ █████  ███  █     █████ █   █ █   ▄█ 
           █   ██  █ █     █▀ ▀█ █       █   ██  █ █ ▄█▀ 
           █   █▀█▄█ ████  █   █ █       █   █▀█▄█ ███  
           █   █  ██ █     █▄ ▄█ █       █   █  ██ █ ▀█▄  
         █████ █   █ █      ███  █████ █████ █   █ █   ▀█

I recently found one of my first shareware programs, InfoLink, tucked away in a BBS (bulletin board system) archive.

InfoLink was a personal information manager (PIM) – they were all the rage in the early 1990s. It took the hyperlinking ideas that were just being employed by the nascent world-wide-web and let the user build and edit interlinked pages of information. I suppose it would now be called a personal wiki. It distinguished between links for people, places, things and events and had a calendar to show the events. It could call out to external programs to view and edit files or send faxes.

I designed it in 1993 while I was travelling around Europe with work. I wrote it in 1994 using Turbo Pascal, which I’d won in an AI programming competition in a Computer Shopper show in London when I was 18. The indexing in InfoLink was my first implementation of a B-tree.

It still runs (via DOS emulators in Linux for me now), and looks pretty useful I think, even two decades on.

Here’s a download of the zipped INFOLINK software. The program executable is 70K.

Here’s a shot of the main index page:

And editing it shows the link tags:

See the breadcrumbs in the Route panel, and the lovely drop-shadow!:

I have the source code somewhere on a floppy disk, and the compiler on several 5¼” floppy disks. I think it would be a challenge to build it again.

Best Thing I Never Had

Beyoncé’s “Best Thing I Never Had” is a song about her saying good riddance to her boyfriend. It has some cleverly engineered but contradictory sounding lyrics:

  1. You turned out to be the best thing I never had
  2. I will always be the best thing you never had
  3. I will never be the best thing you never had

Sentences 1 and 2 use exactly the same phrase “the best thing never had” to mean directly opposing things for You and I. While sentences 2 and 3 seem to flatly contradict each other, but mean the same thing. How come?

Let’s analyse the sentences and see how they work. To make it simpler, I’ll refer to the singer as B (for Beyoncé) and him as C.

Starting with sentences 1 and 2, the trick lies with the word “best” and how its scope can be stretched in two ways, making the phrase ambiguous.

Sentence 2 (“I will always be the best thing you never had”) has the simpler interpretation. B (Beyoncé) states that she is the best thing. The word “best” is tightly bound to the word “thing” – it has a very restricted scope. So “best” here can mean its usual ‘highest valued’. And then the “you never had” is an extra bit of information that states that C (he) has never had B. The upshot is that B is the best thing. Here it is a bit more formally:

[B is a thing ∧
 ∀x[[x is a thing ∧ ¬x = B] → B has higher value than x] ∧
 ¬C had B
]

In sentence 1 (“You turned out to be the best thing I never had”), the not-having is being done by B. The scope of the word “best” can include the “never” and then its negation affects the interpretation of “best”. So “best” here can mean the ‘best to not have’. The best thing to not have, from her point of view, is the lowest-valued thing. So, loosely speaking, C is the worst thing. This is the key to the seemingly misworded song title.

[C is a thing ∧
 ∀x[[x is a thing ∧ ¬B had x ∧ ¬x = C] → C has lower value than x]
]

Sentence 3 (“I will never be the best thing you never had”) is interpreted in the same way as sentence 1 (with the parties exchanged) and then negated by the initial “never”, and not as simply a negation of sentence 2 despite the syntactic similarity. C is doing the not-having and from C’s point of view, the best thing to not have is the lowest-valued thing. So here “the best thing you never had” says that B is the lowest-value thing that C has never had. The “I will never be” inverts this to mean B is the highest-value thing that C has never had (and adds that this state will be maintained as long as B is around). So, again roughly speaking, B is the best thing.

¬[B is a thing ∧
  ∀x[[x is a thing ∧ ¬C had x ∧ ¬x = B] → B has lower value than x]
 ]

Of course, our dual interpretation of these ambiguous sentences is made in the context of the rest of the lyrics. In isolation, we could interpret them in other ways and have one of the following:

  • a) Beyoncé as the worst thing wallowing in self-deprecation singing about her missed opportunity to have the best thing (flip our interpretations above)
  • b) Beyoncé and him both being each other’s best thing and both missing out (only use one interpretation, based on our original view of sentence 2)
  • c) Beyoncé and him both being each other’s worst thing and both lucking out (only use one interpretation, based on our original view of sentence 1)

Using the open world assumption, these ‘best to not have things’ will never do. Neither party can know the value of all the things they’ve never had, so the superlatives are moot.

 

ThinkSQL ODBC Driver Source

The ThinkSQL ODBC driver source code is now available here.

The driver is pure Delphi but the specification is C-based and very detailed. Memory handling and complex state transitions between client and server made this a large undertaking.

The Java JDBC specification was closely based on ODBC, but garbage collection made the JDBC driver much easier to implement.

It’s worth mentioning that the Python DB API specification is tiny in comparison to the ODBC documents but led to a much simpler, smaller and more elegant driver codebase. The Python driver can do all the things that the ODBC driver can do, but with less than a tenth of the code. It is more readable and maintainable and less fragile, and easier to use. And it actually works on multiple platforms. The surprising thing was how much simpler Python made the low-level network communications (mostly thanks to the struct module), especially considering Java was originally designed for low-level systems. Compare this Python method with the Java and Delphi ones:

The Python version

def getpUCHAR_SWORD(self):
   if self.bufferPtr+_sizeof_short>self.bufferLen:
       if self.bufferPtr==self.bufferLen:
           self.read()
       else:
           return _fail

   s=self.buffer.read(_sizeof_short)
   self.bufferPtr+=_sizeof_short
   si=struct.unpack('<H', s)[0]
   self.bufferPtr+=si
   return self.buffer.read(si)

The Java version

public String getpUCHAR_SWORD() { 
  if (bufferPtr+Global.sizeof_short>bufferLen) {
    if (bufferPtr==bufferLen) {
      Read();
    }
    else { //the buffer is not quite empty
      return Global.failString; 
    }
  }
  short usi=0;
  for (int siz=Global.sizeof_short-1; siz>=0; siz--) {
    usi<<=Global.sizeof_byte;
    short b=(short)buffer[bufferPtr+siz];
    if (b<0) {b=(short)(b+256);}
    usi = (short)(usi | b); //i.e. reverse order
  } 
  bufferPtr=bufferPtr+Global.sizeof_short;
  if (bufferPtr+usi>bufferLen) {
    if (bufferPtr==bufferLen) {
      Read();
    }
    else { //the buffer is not quite empty
      return Global.failString; 
    }
  }
  bufferPtr=bufferPtr+usi;
  return new String(buffer,bufferPtr-usi,(int)usi-1);
}

The Delphi (C-like) version

function TMarshalBuffer.getpUCHAR_SWORD(
                                var puc:pUCHAR;
                                allocated:SWORD;
                                var sw:SWORD):integer;
{RETURNS: ok,
          fail = probably means data is left in buffer, 
                 but not enough 
                 else errors as from Read
 Assumes:
   puc has 'allocated' space allocated by caller,
   unless -1 => allocate here (but caller must free!)
 Note:
   returns buffer data up to the allocated length
   (including 1 character for a null terminator)
}
const routine=':getpUCHAR_SWORD';
var actual:SWORD;
begin
  if bufferPtr+sizeof(sw)>bufferLen then
  begin
    if bufferPtr=bufferLen then
    begin //the buffer is empty
      result:=Read;
      if result<>ok then exit; 
    end
    else
    begin //the buffer is not quite empty
      result:=fail; //buffer overflow
      exit;
    end;
  end;

  move(buffer[bufferPtr],sw,sizeof(sw));
  bufferPtr:=bufferPtr+sizeof(sw);

  if bufferPtr+sw>bufferLen then
  begin
    if bufferPtr=bufferLen then
    begin //the buffer is empty
      result:=Read;
      if result<>ok then exit; 
    end
    else
    begin //the buffer is not quite empty
      result:=fail; //buffer overflow
      exit;
    end;
  end;

  if allocated=DYNAMIC_ALLOCATION then
  begin
    {Now allocate the space for the buffer data}
    getMem(puc,sw+sizeof(nullterm)); 
    allocated:=sw+sizeof(nullterm);
  end;

  if (allocated-sizeof(nullterm))<sw then
    actual:=(allocated-sizeof(nullterm)) 
  else
    actual:=sw;
  move(buffer[bufferPtr],puc^,actual); 
  move(nullterm,(puc+actual)^,sizeof(nullterm));
  bufferPtr:=bufferPtr+sw;
  result:=ok;
end; {getpUCHAR_SWORD}

The best a man can get?

For some reason I don’t really understand, I shave my face almost every day. I use a manual razor, currently a Wilkinson Sword Quattro-Iced-Titanium-Strontium-something or other. I did try to stick with a perfectly acceptable Gillette three-blade razor but was forced to upgrade to a four-blader by some clever, nationally coordinated, stocking systems. Whatever’s next? I’m now trying to avoid upgrading to a manual razor with a battery in, and an extra indicator to tell me to buy more blades.

For many years I put up with buying a new pack of blades every couple of months or so, as a single blade used to last me between 1 and 3 weeks (over £9 for a pack of 4 blades, which cost under £0.05 each to manufacture). But I could never stomach the adverts. The ones where men are hugging babies and doing man things, like driving cars and playing golf all set to a crescendoing power ballad. Especially the father’s day specials. So here’s my response to the bad adverts – a top tip that Gillette don’t want you to know that will save you hundreds of pounds: When you’ve finished shaving, dry the razor head on a towel.

That’s it: after every shave, make sure there’s no water left on the metal blades by dabbing them on a towel and your blades will last many, many times longer. I’ve been doing this for a few years, and I now only use about 3 blades a year. Do it – they’ll have less money to spend on their insulting adverts. Or grow a beard.

Think SQL

Several years ago, I spent a lot of time developing a relational database management system – ThinkSQL.

In a future post I’ll write down the reasons why, and probably release the development diaries. But for now, the source code for the server is available here: https://github.com/ggaughan/ThinkSQL

As I say in the README:

The source code is linear per se, but while writing it, it was an organic thing, generating great interwoven trees in the computer memory and in my head that were many-layered and that were modified and traversed, and impacted on a dynamic multi-versioned data store alongside many other threads, causing and needing deep psychological flow. The code comments are released as-is, and are often streams-of-consciousness.

 

Run a pipe2py-generated pipe on Google App Engine

Once you’ve used pipe2py to convert your Yahoo! Pipe into a local Python module, you can then host it as your own Google App Engine application. You just need a bit of boilerplate to handle the parameters and output formatting. I’ve created a sample project containing this boilerplate here: http://github.com/ggaughan/testpipe. You’ll need to include a copy of pipe2py, so it can find the modules it needs, and replace the demo pipeline references with your own. See the README for details.

Don’t flap when reversing

Top tip: when helping someone to reverse their vehicle into a small space – don’t stand behind the car and beckon them with gusto until you think they’re about to hit something and then show them your palm like some kind of traffic police.

(Instead, how about holding your arms apart to show them the actual size of the space and reduce the gap between your hands as they near the obstacle? That way they get immediate feedback and can see ahead of time how far they have left.)

Running Yahoo! Pipes on Google App Engine

Yahoo! Pipes is an excellent tool for processing data. It provides a visual way to aggregate, manipulate, and mashup content from around the web. It’s very much like plumbing with data and is a great metaphor. I’m convinced that this approach is just the beginning, and look forward to connecting systems using pipes in a three-dimensional virtual environment with tactile and audio feedback… soon.

pipe UuvYtuMe3hGDsmRgPm7D0g

Tony Hirst, a prolific Yahoo! Pipes user, had the idea to translate the pipe definitions into code so that they could be run on your own computer, in case the Yahoo! Pipes server was unavailable. This sounded like an interesting challenge so I developed pipe2py. The pipe2py package can compile a Yahoo! Pipe into pure Python source code, or it can interpret the pipe on-the-fly. It supports embedded pipes too. (Not all of the Yahoo! Pipes modules are available yet, but they’re gradually being added: if you find the need for one that’s missing please let me know, or better still provide me with the code for the module.)

The design for the compiled pipes was based on David Beazley’s work on building Python generators into pipelines, together with ideas from SQL query compilers and XProc pipelines. Each Yahoo! Pipes module is implemented as a Python generator which iterates over items provided by an input module and processes them to yield output results. Once these generators are connected together, iterating over the final one will initiate a cascading call to all earlier generators for them to iterate over their inputs and, in turn, yield their output. There are several benefits to this architecture:

  1. the compiled pipeline closely matches the original Yahoo! pipeline
  2. adding new modules is easy because they are loosely coupled
  3. each item is typically passed through the whole pipeline one at a time, so:
    1. memory usage is kept to a minimum
    2. no module is waiting on an earlier module to finish processing the whole data set
  4. by adding queues between the modules they could easily be made to run in parallel, each on a different CPU, to give great scalability

Here’s an example pipe2py session which converts the pipe shown above into Python and then runs it locally:

$ python compile.py -p UuvYtuMe3hGDsmRgPm7D0g
$ python pipe_UuvYtuMe3hGDsmRgPm7D0g.py
Name (default=Lancaster) Neill
{u'title': u'Bob Neill',
...
u'TotalAllowancesClaimedIncTravel': u'157332'}

Since pipe2py can compile pipes into Python modules, it seemed a good idea to try to run them in Google’s cloud via App Engine. So now there’s pipes-engine, which uses pipe2py to run your Yahoo! Pipes on Google’s servers.

pipe2py running Yahoo! Pipes on Google App Engine

You’ll need to log on with your Google account, and then you can take the Id of your Yahoo! Pipe (you can find it in the url when editing a pipe) and add it to the list. pipes-engine will then compile it and store the Python version of it. Clicking the pipe Id will run it on the App Engine. If you change the pipe in Yahoo, you can reload it in pipes-engine to re-compile the latest version (although I hope to automate this step in future).

There’s currently an App Engine timeout of 30 seconds, but Google have said that they are working on increasing that soon.

There were some tricky bits to developing this, like storing the generated Python source in the datastore and then importing it dynamically back from the datastore, and doing so recursively for any embedded pipe imports. Some Python PEP 302 magic helped here.

The pipes-engine.appspot.com service is a proof of concept and needs some more work, not least to provide the output in formats other than json, but I think it proves it’s feasible. Let me know what you think.

It’s a Sin

The internet has enabled me to research something I noticed over 20 years ago. I remember watching an ITV mini-series in the 1980s that had a dramatic and memorable theme tune. Soon after, Pet Shop Boys released their massive hit “It’s a Sin” and I was amazed.

To hear why, first listen to some of “It’s a Sin” from 1987:

Now listen to the theme tune of that mini-series, “If Tomorrow Comes”, composed by Nick Bicât:

That was broadcast in 1986, the year before “It’s a Sin” was released.

I’m no musician, but they sound surprisingly the same to me.