Luke's Thought Dump

How I Deal with Lots of Data

2012-01-02T18:00:00.000+11:00

Just for some context, by "lots of data", I mean a couple hundred million rows in a table joined to other tables of similar size. By no means is it a lot compared to what other people deal with, but it's certainly enough to warrant some forethought instead of just diving right in.

Late last year, I was tasked with a few one-off reports that required me to summarise data that was stored over a couple hundred million rows in a database. Knowing that it was going to be a long process to retrieve the data for these reports, I had to come up with a game plan to do this as quickly and as efficiently as possible. In the process of doing this, as always when handling relatively large quantities of data, a few things were learned. None of these lessons/ideas are new or original, but I wanted to write them down somewhere, and here feels like as good a place as any.

Extracting the data that I needed (which, in one instance, was a subset of a couple of ~300 million row tables) into a local database meant I was able to modify the data (like cleaning up dirty, inconsistent spellings of suburbs, states and countries) and modify the schema (like adding new indexes which, because of the odd nature of the reports, the production databases didn't have or ever need previously).

Assuming that you don't require something like schema changes, that local database doesn't even need to be a relational database. CSV files work perfectly fine a lot of the time. For a couple of reports I wrote a handful of scripts, the first of which was to pull the data out of the MySQL database, perform some simple operations on the data and output it into a CSV file. The other scripts that needed to operate on the same data could then easily (thanks to Text::CSV_XS) read the CSV data, which was a lot quicker than reading it from a relational database.

Why do CSVs lend themselves nicely to this kind of task? Because with reports like these, in my experience, you very rarely perform complicated operations on the data after extracting it from the original data source(s); you just want to suck the data up, summarise the data, output the summary, and then output the nitty gritty details on subsequent pages or into a separate file.

An obvious advantage to storing the data like this is the speed in which you can retrieve and process the data. That improvement made a huge difference for me because I like to run my scripts very often throughout the development process, no matter how small the change.

Of course, depending on the size of the data (in bytes), extracting the data into a local database of some sort may not always be possible.

The last big win I had was not using object-relational mappers (Class::DBI in this case). They're great a lot of the time and save on code and development time, but when dealing with millions of rows, they just add bloat and everything runs much slower than it should.

That's all I can think of now, a few months later.

Crappy IRC and Unicode

2011-06-28T13:44:00.005+10:00

I use MacIrssi as my IRC client at work and at home. It's mostly great. By mostly, I mean, I wish when people sent smart-ass messages on IRC filled with unicode characters, I could actually appreciate how much of a smart-ass they are being. Instead, all I see is this:

< ganeshanator> gonna go to the \u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588
    and \u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588
    getting those \u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588
    and the \u2588\u2588\u2588\u2588\u2588
    on \u2588\u2588\u2588\u2588\u2588

I always forget how to convert it, so I wrote this, and now I never have to remember again:

#!/usr/bin/env perl

use warnings;
use strict;

use Encode;

sub unicode_plz { encode( 'UTF-8', pack( 'U', hex shift ) ) }

( my $message = shift ) =~ s{\\u([a-fA-F0-9]+)}{unicode_plz($1)}ge;
print "$message\n";

And, BAM!

Crappy was a harsh word. MacIrssi is pretty great, except for this.

DateTime Woes

2011-03-31T08:32:00.004+11:00

This morning, I hate DateTime for this.

$ perl -MDateTime -wle'$a = DateTime->today( time_zone => "Australia/Melbourne" ); $b = DateTime->today; $b->set_time_zone("Australia/Melbourne"); print $a->iso8601; print $b->iso8601'
2011-03-31T00:00:00
2011-03-30T11:00:00

I understand why it happens, but it's still an annoying, pissy, little bug to find. sigh...

Update to an Old Post

2011-02-03T12:48:00.003+11:00

A past post about tweeting from the command line with the Net::Twitter Perl module is one of my most visited entries. Anytime I check my traffic, it is always the second most visited entry for the day/week/whatever. However, it has a problem. The code does not work anymore for a simple, yet annoying, reason.

Last year, Twitter turned off its basic authentication support and now requires application developers to use OAuth. More detail about how that affected Net::Twitter can be read here. This kinda sucks because it's not longer as simple as providing a username/password combination. You need consumer keys/secrets and access tokens/secrets and that makes things a tad messier.

The original code I posted uses basic authentication and, obviously, will not work anymore. However, if anyone is interested, the code for that small side-project, Twitsh.pm, is on github and I recently updated it to use OAuth so that it works again, however the configuration file requires you to add your own consumer/access keys/tokens/secrets. Enjoy.

Goal for the Year: IPv6

2011-01-23T20:10:00.003+11:00

First blog post for the year! Yay? And I'm only setting myself one goal for the year: Learn everything about IPv6 and set it up at home.

And by everything, I mean everything; from IPv6 packet headers to the implementation complexities to the business issues to the security issues to the political issues.

Why? Because, potentially, in the next decade, migrating from IPv4 to IPv6 will become important, and considering that it's a topic that I know next-to-nothing about, other than what I've read in passing on IRC, it's a knowledge gap that needs filling.

It's certainly possible that this will all take me less than a month, assuming enough free time. When that time comes, perhaps I'll pick a non-geek goal to spend my time on this year.

Host Discovery via SPF Records?

2010-12-01T22:48:00.010+11:00

So... my once-a-month posts OCD issue went unsatisfied for the past 2 months. Oh well. Back to business...

Sender Policy Framework (SPF) records are often used as one method of combatting spam; specifically, spam that wants to look like it has been sent by your company. The general idea is that your SPF records specify which hosts and IP addresses are allowed to send email using your domain(s) in the envelope 'from' address. Any other host, which is not contained within the SPF policies, that attempts to send mail from your domain will fail the SPF check and, hopefully, get picked up by some kind of spam detection software further down the track. It is highly recommended that you setup SPF records for any domains that you own.

How are SPF records requested? Via DNS. They are usually contained within the TXT record, however they can also, occasionally, be found in the SPF record aswell. The format is very easy to understand. An typical example to look at would be optus.com.au:

IN TXT "v=spf1 mx/24 include:opt01._spf.optin2.com.au ip4:180.92.216.0/21 include:rightnowtech.com include:rnmk.com include:custhelp.com ~all"

Do the hosts (and the range of IP addresses) in the policies belong to optus.com.au? Maybe. Maybe not. We cannot discern that just by looking at the host names, nor can we, within any real degree of certainty, determine it programmatically. However, the hosts' SPF records have been trusted to send mail on behalf of optus.com.au, so they remain hosts of interest for penetration testers and may not pop up in other DNS requests (A, MX, AFXR, etc..)

What is a common example of a host who would be in a domain's SPF policies but not actually part of the company who owns the policy? Any mail filtering company who filters outbound mail.

Just a thought that has been floating around in my head...

End of the Month...

2010-09-30T13:48:00.008+10:00

Well, it's the end of the month, which seems to have become the day that I write a new blog entry (I feel somewhat empty if there's a month missing from the sidebar over there), but I don't have anything very interesting to post, so I'll post something a little less interesting: A github project I use for testing libraries, Mazer.

Mazer is a rewrite of an old assignment I did years ago in C++. Originally it just had to automatically solve a maze step-by-step, as if simulating a human walking through the maze. I often write this same program when I start learning a new programming language (which I did for Haskell and Common Lisp), or, as was the case this time around, to play around with libraries. It was the first bit of code I wrote using Moose, Test::Class (the tests actually pass!) and, most recently, Curses::UI.

There's nothing special about it. It has basically become my sandbox for Perl libraries. Perhaps others would find it useful, though.

The State of the PDF::API2 Graphics Object

2010-08-31T21:10:00.004+10:00

Often when I write scripts with PDF::API2, I modify the state of my graphics object a lot. Sometimes to change the current fill color, or line thickness, or perhaps to set line dashing. Usually, once I have finished drawing the object, I want to set the state back to what it was before, and then continue on with the rest of the script. Rather than having to either re-set everything on the graphics object or use a new graphics object, there are two methods that will help you out.

The save and restore methods will, surprisingly, save and restore the state of your graphics object. What makes it even more handy, is that you can save multiple states and have a stack of states. Some example code:

#!/usr/bin/env perl

use warnings;
use strict;

use PDF::API2;

my $pdf = new PDF::API2;

my $page = $pdf->page;
$page->mediabox( 100, 60 );
my $gfx = $page->gfx;

# Make it red.
$gfx->strokecolor('red');
# Save the first state
$gfx->save;

$gfx->move( 10, 10 );
$gfx->line( 90, 10 );
$gfx->stroke;

# Make it blue, a bit thicker and add some dashing.
$gfx->strokecolor('blue');
$gfx->linedash( 1, 5 );
$gfx->linewidth(5);
$gfx->move( 10, 20 );
$gfx->line( 90, 20 );
$gfx->stroke;
# Save the second state
$gfx->save;

# Set some new parameters.
$gfx->strokecolor('yellow');
$gfx->linedash();
$gfx->linewidth(3);
$gfx->move( 10, 30 );
$gfx->line( 90, 30 );
$gfx->stroke;

# Revert back to the 'blue' state.
$gfx->restore;
$gfx->move( 10, 40 );
$gfx->line( 90, 40 );
$gfx->stroke;

# Revert back to the 'red' state.
$gfx->restore;
$gfx->move( 10, 50 );
$gfx->line( 90, 50 );
$gfx->stroke;

$pdf->saveas("$0.pdf");

Run the code and you should get the following:

Handy, I think.

PDF Library for Laying out Widgets... or something...

2010-07-27T12:28:00.005+10:00

At work I have pretty much become the developer that writes the code to produce reports, whether the output be in PDF, CSV, XLS or whatever format is required. When I started this job, the first task I was assigned was to create a particular report (mentioned in a previous PDF::API2 post), and even now, a year and a half later, I am re-writing a handful of reports for our client-facing web console. Most of the reports I develop, new ones in particular, are outputted as PDF documents.

My process for developing PDF reports is simple:

Figure out what data is going where.
Measure up the coordinates.
Do the magic that gets the data and outputs it in pretty graphs.

After writing code for dozens of reports, step 2 has become a lot faster than the first time I did it, but it's still a manual job and it still takes time, and since the process of measuring the coordinates is always the same for me, I want to automate it. I want to be able to hand a bunch of objects (or widgets) to a function or class and have it automatically figure out how and where to lay them out - in a grid or column layout, for example - given certain metadata, e.g. widths, heights, padding, font. I also want the ability later to modify a specific object's metadata if a particular graph or table feels like it would look more appealing with a little extra padding on the left, for example.

I noticed there is a PDF::APIx::Layout module, however it hasn't been updated since 2005.

So far I only know what I want it to do, and I have a basic idea of how the backend, PDF::API2 code might look. But I still haven't completely decided how I want the API to look to the developer; procedural, OO, declarative or some fusion of them all? I may even end up trying them all out for myself to see which is easier to develop with.

I will post more updates in the future, regarding the API, and any code will probably end up on GitHub, but that is basically the idea that has been floating around in my head for a few months.

Light Mojolicious::Lite Experience

2010-06-30T22:35:00.010+10:00

I've been playing with Mojolicious::Lite for the past couple of weeks. Why? Well, I had started a small project which records public data from a MMORPG that I frequently play (don't ask which one... it is no longer worth playing or even giving a free mention to) and needed a pretty front-end that myself and some friends could use to view the data.

Choices, Choices, Choices

The initial code I had written for the project was written in Python, which was two simple scripts; one that ran in a cron job once per day to retrieve the data as XML and one that analysed the data and outputted an ugly, little summary. I had intended to store some of the raw data in a database and wanted a web front-end to display it all and allow my non-technical friends to view and/or export the data as Excel spreadsheets.

Since this was a hobby project and I like to learn new libraries/frameworks for hobby projects, I had two decisions to make:

Which library for database access?
Which web-framework?

I was itching to use DBIx::Class for something, especially after using Class::DBI at work and hearing about the wonderful things DBIx::Class has that Class::DBI doesn't. But that's a Perl library, so I rewrote the Python script that ran in a cron job in Perl, so I could later hook DBIx::Class into it.

Deciding on the web-framework was next. A couple of years ago at university I had used Sinatra in a Ruby project and loved it and since then I have had a thing for web micro-frameworks. Python has Flask, which I always had in the back of my mind, and I like the way it uses decorators for the routing. Perl, luckily, has Mojolicious::Lite. I realise now that Dancer exists, however I found Mojo first (sorry, Dancer devs).

What I Like

The usual good things about web micro-frameworks: Rapid development.

The mechanism that Mojolicious::Lite uses for routing is very nice, feels very Perl-ish and allows for very compact code.

What I Don't Like

Errors in templates can result in cryptic error messages. For example:

Scalar found where operator expected at (eval 488) line 25, near "} $_M"
        (Missing operator before  $_M?)

And:

Template error in "rank.html.ep": syntax error at template line 2, near ";
$_M "
syntax error at (eval 488) line 25, near "} $_M "
syntax error at (eval 488) line 29, near "} }"
1: <% my $server_table = { %>
2:      <% my ($ranks, $server, $align) = @_; %>
3:
4:      <div class="rank_list <%= $align %>">

What caused the errors? The following two lines of code were:

<% my $server_table = { %>
...
<% } %>

When they should have been:

<% my $server_table = {%>
...
<%} %>

So a couple of spaces in the template produced those error messages. Not terribly obvious, unfortunately.

In the end...

A cryptic error message from a template error is far out-weighed by the things I like, and HTML::Mason's error messages have caused me much bigger headaches. Most of the time, any personal projects that I have only ever, if at all, need a super basic web interface, and I like to get my web applications up and functional as quickly as possible. Mojolicious::Lite let me do exactly that for this project and I intend on using it again in future projects.

PDF::API2 is Underrated

2010-05-31T12:34:00.015+10:00

I feel like there has not been enough talk about PDF::API2 in the Perl community. Searching for code examples and tutorials does not yield many results, however it does lead to this tutorial, which is the only useful one that I have been able to find. It is worth reading. Oh, and it's written by my boss ;) Go and read it if you plan on using PDF::API2 in the future.

At work, for the past year, we have used PDF::API2 to create this nice looking report and many others like it. Since then, I have learnt a number of lessons and picked up a few tips for using this powerful library, so here are some of them. This is by no means a tutorial, or even an introduction to PDF::API2, so if you are looking for something like that, refer to the tutorial I referenced in the previous paragraph. This is more of a coredump of the part of my brain that knows PDF, so to speak.

The coordinate system is that of a Cartesian plane

Personally, every toolkit I have used in the past (which, admittedly, is not a long list) has placed the origin (0, 0) in the top-left corner. I assume, without actually researching, that this was because it was easier for developers to think about placing the top-left corner of a widget/window around the screen. We (mostly) read articles and books and posters from the top-left to the bottom-right.

However, coordinates in PDF::API2 work differently; the origin resides at the bottom-left of the document, just like the first quadrant of a Cartesian plane. Once you think of it this way, it becomes a lot easier to visualise what you are doing and the placement of your content.

Work from the bottom-left

This is linked to the previous point. If you are factoring out specific data or widgets into separate classes or functions, specify the position of the data by the bottom-left coordinate, rather than the top-left. This saves a lot of recalculating when you have to use various text/graphics object functions.

I prefer to refer to the coordinates as 'bottom' and 'left', which clears up any confusion that using 'x' and 'y' might introduce. I recommend doing the same.

Occasionally you may need to work from the top of your widget, e.g. for tabular data, in which case you can simply use the bottom + height to get the top-most Y-coordinate.

Useful CPAN documentation

PDF::API2 documentation is shocking. It's horrible. It's almost non-existant. However, PDF::API2::Content contains 95% of the functions that you probably want. Specifically, it contains all of the member functions for the text and graphics objects, which are the objects you will use most of the time.

Useful function that you might skim past

$text->advancewidth('foo bar', \%options)

Given the state of your text object, i.e. font name and size, this will return the width, in points, required to display the text 'foo bar' on the page. Great for figuring out how much space you need to display content like names of people. If you do not want to use the current state of your text object, the second parameter, with text object settings, can be used instead. I have never needed to use the second parameter so far.

That's it for now. In the future I would like to write some more about this library - preferably with more structure and a nice example rather than random ramblings - but that will have to wait until I can spare the time.

Broken Sound in OS X

2010-04-30T07:43:00.004+10:00

I use a MacBook with OS X 10.5.8 at work. Every now and then, at work, while streaming radio off the internet through iTunes, running Parallels, an IRC client, a Twitter client, an MSN client, a dozen terminal windows/tabs, Chrome with a dozen tabs, Firefox with a handful of tabs open, 30+ Stickies floating around and having not restarted my laptop for 2 weeks, my sound, all of a sudden, goes distorted and choppy (technical term!), which is very annoying when you are streaming radio to four other developers through a surround sound system.

At first I was just restarting my laptop. But that is a little annoying when you have that many programs, websites/tabs open and have to spend 15 minutes remembering what exactly it was you were doing when the sound decided to go rogue.

I noticed that when this occurs, the coreaudiod process starts to use a lot more CPU than usual, i.e. it becomes one of the top 4 CPU-hungry applications according to top. After killing the coreaudiod process, a new coreaudiod process was spawned and my sound returned to normal. Success!

New Perl Hate

2010-03-23T12:25:00.003+11:00

You don't work with a language every day without building up a "things I hate in language X" list. Well, I have a few hates of Perl, but they've all been fairly well covered in other blogs and cover the usual topics (pre-Moose OO, getting bitten by scalar/list context, etc), however yesterday I came across a new one. Here is a one-liner demonstrating the problem:

$ perl -MData::Dumper -le'@a=qw/a b c/; @foobar = map { $_ } @a[0..7]; print Dumper \@a'
$VAR1 = [
          'a',
          'b',
          'c',
          undef,
          undef,
          undef,
          undef,
          undef
        ];

Fairly obviously, I have created a list with 3 elements in it, tried to slice it with 8 values and assign that to a different list, however when printing the contents of the original list, it has now been padded out to 8 values with undefs. Ok, I thought perhaps doing a slice on a list had that kind of destructive behavior but I had just never noticed it before and had never heard about it previously. So I tried another one-liner to perhaps prove that theory:

$ perl -MData::Dumper -le'@a=qw/a b c/; @foobar = @a[0..7]; print Dumper \@a'
$VAR1 = [
          'a',
          'b',
          'c'
        ];

Nope. Bug? Or expected behavior for some reason that I cannot figure out? This was on Perl 5.8.9

Diving into Reverse Engineering

2010-03-17T06:55:00.013+11:00

Last year I read a blog entry (linked below) about a programmer who edited a component of Skype to fix a bug which was causing problems for many users. The blog entry then showed a very simple example of how to edit programs using OllyDbg without the source code. Even though I always knew the basic idea, I had never seen, or even searched for, any clear examples because I had no immediate need to know the details.

There is a MMORPG that I play on the weekends and recently there was discussion on our guild's forum about client tweaks. In the back of my mind, I knew I could probably apply a little reverse engineering to remove annoying art that clutters up the screen, change font colors to be easier on the eyes, and on top of that, maybe figure out how to fix a few bugs which we have been pushing the developers to fix for months, with no results. The problem was that I had no idea how to reverse engineer a program.

Of course, knowledge of how programs work and assembly language was required. Luckily the former was taught fairly well during my course at university, and although we were taught assembly language for the Motorola 68k processor, it never really clicked for me, or I never had an "ah hah!" moment with it and thus never did anything practical with that knowledge.

But now I did! So for the past ten days I have almost gone blind reading articles about reverse engineering and looking at assembly code.

Despite the moral questions, it has been a very interesting exercise and I'm happy that I've found a practical exercise for reverse engineering and assembly language that interests me. So far I've managed to fix one, semi-annoying bug and modified the layout to remove various things that get in the way of seeing what I'm trying to do/click on..

Below is a link to the initial blog entry that piqued my interest and some of the more useful resources I came across over the past ten days:

Perl Profilers - Devel::NYTProf

2010-02-10T12:40:00.004+11:00

Keeping up the spirit of a blog post from Leo Lapworth, I am advocating the use of Devel::NYTProf and doing my bit to get its name out there. On a number of occasions it has saved me a lot of time searching for bottlenecks in my Perl code and also pointed out parts of code that seemed unsuspicious.

The docs on the CPAN page are very clear in its usage, so I won't bore anyone with the details here.

So as far as profiling Perl code goes, Devel::NYTProf is definitely the winner!

Programming Goals for Common Lisp

2010-02-02T20:55:00.004+11:00

First post of the year... Somehow I was too busy for the whole of January to post anything. Anyway, moving on...

I've been yearning to use cl-opengl and Vecto for SOMETHING but can never seem to come up with any ideas. The only useful thing I have used cl-opengl for is a cellular automata simulator, but it's all very basic and minimal. So instead of trying to come up with a new idea, I'm going to re-use an old idea and extend cl-mines; my small, currently CLI-based, Minesweeper game.

I intend to use Vecto to generate each of the images for the mines and uncovered tiles. Not the most intense usage for Vecto, but it's better than nothing for now.

And I, obviously, intend to use cl-opengl for displaying and running the whole game.

Another, secondary, goal is to use more macros, if appropriate. Macros are a feature of Common Lisp I am yet to use in a complex way and is a skill I think I'm missing out on with Lisp.

So that's what I am going to be slowly working on in my spare time between everything else! That should keep me busy for a little while, I hope.

Experimenting with Befunge

2009-12-10T19:11:00.005+11:00

A few years ago I had a look into my first esoteric programming language, Brainfuck. Why? Because an IRC channel I frequented required users to message a bot with Brainfuck code, of, at most, a certain number of bytes depending on how long your nickname was, to output your IRC nickname to get voice status so you could actually chat in the channel! I ended up writing an interpreter for it in Perl in ~130 characters too! Not bad considering I was seventeen years old at the time.

So for the few days this week that I was home sick, I looked into Befunge; an esoteric, stack language. Why? Because I could ;)

Easily the most interesting part of the language, for me at least, is the four operations that change the direction of the program counter, allowing programs to read in all different directions. For example, conventional programming languages start from the top and read left-to-right. In a Befunge program you can change the direction to read operations in the left, right, upward or downward directions. This can be used to create some very interesting programs.

Anyway, here's my first, and, so far, only, Befunge program:

v          v                 ,<
                        >:"z"`|
>"!raB ooF">:#@!_:"a"1-`|     >84*-v
           ^           ,<          <

It simply capitalises some text. Nothing special.

The things you do when you're sick at home...

Profiling Common Lisp Code

2009-12-08T11:53:00.003+11:00

I just finished reading (and applying to my own projects) an article on profiling and optimising Common Lisp code using SBCL's profiler. The article is here. The whole series that the article belongs to is good reading for any novice-to-intermediate Lisp programmer actually, but this one stands out more for me.

Manually installing ASDF packages for SBCL

2009-11-19T21:00:00.020+11:00

When I started getting back into Lisp earlier this year, after a bit of reading I quickly found out that ASDF was the more popular method for installing third-party packages. I also quickly found a problem though: simply typing (asdf-install:install 'package-name) doesn't always work, because some packages, while they have ASDF system files written, are not uploaded to - or are not up to date on - the remote site (cliki.net) and thus fail when you try to install it. So when you're out of options and decide to download or checkout the source for the package, how the hell do you install it?

In your home directory you should have a .sbcl directory. Inside this directory you should have two more directories: site and systems. To manually install a package:

Unzip/untar/checkout the source code into its own directory inside the site directory. For example, on my machine cl-ppcre is unpackaged into .sbcl/site/cl-ppcre-2.0.1/
Create a symlink from the ASDF system file to the systems directory. For example, .sbcl/site/cl-ppcre-2.0.1/cl-ppcre.asd is symlinked to .sbcl/systems/cl-ppcre.asd

That should be it. Now doing a (require 'cl-ppcre) in SLIME locates the ASDF system file in .sbcl/systems/ and loads the library as expected.

Note: If your SBCL doesn't look in systems for packages, you can check the asdf:*central-registry* variable for the correct directory.

Another note: If you prefer to install packages for the entire system rather than just your specific user, ASDF system symlinks live in /usr/local/lib/sbcl/site-systems (assuming you installed SBCL in /usr/local).

Last note, I promise: The obvious shortcoming of manually installing libraries is it won't automatically retrieve dependencies for you :(

Twitter shell tools

2009-10-21T12:36:00.002+11:00

I've posted my current shell tools for interacting with Twitter here. Still a work in progress, obviously, but it's progressing none-the-less!

URL Shortening Services

2009-10-15T12:44:00.006+11:00

I'm currently brainstorming some ideas on determining who is linking to a domain that I own via shortened URLS (e.g. tr.im, is.gd, ow.ly and similar). Without approaching the owners of the site to purchase that data ;) Basically I'm trying to answer the question "who is linking to my site from Twitter" since Twitter clients all use URL shortening services.

If the URL shortening service sends the shortened URL as a referrer in the HTTP headers, and the web browser should do exactly that, then we can use Twitter's search interface to search on those shortened URLs when they pop up in our access logs. The obvious shortcoming is if nobody clicks on a particular shortened URL, it will never show up in the access logs and therefor we won't know about it.

More brainstorming is needed. And probably some reading.

SBCL and Command-line Common Lisp Apps

2009-09-03T13:27:00.012+10:00

Ok, first things first: read Xach's steps to creating small Common Lisp projects. Without it, I was unable to find anything as straight-forward as that to explain a process for creating Lisp projects. There is a lot of very useful stuff in there.

Second thing: I am not a Lisp expert so this may not be The One True Way to do things.

Finished reading? Good. Ok. Xach's article is not aimed at creating utilities to be run from the command line, but in full Lisp environments, like SLIME. Xach's guide gives you all the information you need as far as laying out your Lisp code and setting up ASDF system files are concerned. The last bits and pieces you need are concerned with the shell environment. So, you will need:

Your code (obviously)
An ASDF system file (from Xach's article)
A Makefile to compile your code to .fasl and a .core (for SBCL)
A shell script to run your .core file

The contents of the Makefile for a project named "my-project":

SBCL = /usr/local/bin/sbcl

all:
    $(SBCL) --eval "(require 'asdf)" --eval " \
        (progn \
         (asdf:oos 'asdf:load-op 'my-project) \
         (save-lisp-and-die \"my-project.core\"))"

clean:
    rm -rf *.fasl
    rm -rf *.core

This will compile all of the packages you defined in your ASDF system file into .fasl bytecode files and then save a .core file. Now all you need is a shell script to run the .core:

#!/bin/sh

SBCL=/usr/local/bin/sbcl

$SBCL --core my-project.core --noinform \
    --eval "(progn (my-project:main) (quit))"

And that'll do it! This assumes that your code has an exported defun, 'main', as the entry point for your program. Now run the shell script and your application will run.

Edit: Xach mentioned in the comments that you can use the :executable argument to save-lisp-and-die to create an executable, thus removing the need for the wrapper script above.

This is the build process I use for my little cl-mines game, so you can check out the setup there if there is anything that was unclear.

cl-mines - A Minesweeper Clone

2009-08-26T22:23:00.002+10:00

I uploaded my Minesweeper clone, cl-mines to GitHub a few days ago. It's just a simple, command-line Minesweeper game. It doesn't really have a proper build process yet.

I've only tested it on SBCL, but there isn't any SBCL-specific code that I'm aware of and it is, by far, not the most complicated piece of Lisp code out there, so it should work with other Common Lisp implementations.

Releasing Some Code to GitHub

2009-08-21T12:18:00.003+10:00

I'm going to be releasing a bunch of code to my GitHub over the next few weeks, after I clean up some of it and fix a few design issues and small bugs. This should include:

Minesweeper clone
Command-line Twitter tools

I may also release my Cellular Automata simlator aswell. However, at the moment, it has performance issues due to the Common Lisp LTK libraries. I have also started designing a Tetris clone (to be written in either Python+PyGame or Perl+SDL). Any updates to that will be pushed to GitHub aswell.

Cute Perl Gem to Get the Minimum/Maximum Value

2009-08-02T21:41:00.005+10:00

Saw this little nugget on #perl@irc.perl.org the other night. It determines the minimum of two values:

[$b, $a]->[$a <= $b]

It takes advantage of the fact that Perl doesn't have a boolean return type for true or false, so the comparison operators return 1 or 0 for true and false, respectively, which are then used by this code to index the array ref.

To get the maximum of the two values, just flip the operator to >=

Net::Twitter::Lite 0.05

2009-07-16T21:44:00.004+10:00

The latest version of Net::Twitter::Lite has pretty much made my tweet script redundant! Or at least removed the need for half of the code: http://gist.github.com/143950 Enjoy!

Following a Topic on Twitter with Net::Twitter

2009-07-14T22:16:00.003+10:00

Last week I was keeping up to date with Triple J's Hottest 100 of All Time for 2009 and the hashtag to follow was #hottest100, so that's what I typed into search.twitter.com and kept refreshing every few minutes to see the latest thoughts as the countdown progressed. However, it very quickly became annoying to constantly refresh the search page to view the latest tweets. So I threw together this quick Perl script to display Growl notifications for the latest updates:

#!/usr/bin/env perl

use warnings;
use strict;

use Config::ApacheFormat;
use Encode qw/encode/;
use HTML::Entities;
use Mac::Growl;
use Net::Twitter;

use constant {
    TWEETRC             => "$ENV{HOME}/.tweetrc",
    MAX_RESULTS_DISPLAY => 5,
    POLL_FREQUENCY      => 30
};

die "usage: tweet-follow \n" unless @ARGV > 0;

Mac::Growl::RegisterNotifications(
    'tweet-follow',
    ['tweet'],
    ['tweet']
);

my $num_results_display = MAX_RESULTS_DISPLAY;
my $poll_frequency      = POLL_FREQUENCY;

if ( -e TWEETRC ) { 
    my $config = new Config::ApacheFormat;
    $config->read(TWEETRC);
    my $block = $config->block('follow');
    $num_results_display = $block->get('max_display_results');
    $poll_frequency      = $block->get('poll_frequency');
}

my ($query) = @ARGV;
my $twitter = Net::Twitter->new( traits => [qw/API::Search/] );

my $since_id = 0;
my $max_id   = 0;
while (1) {
    my $response = $twitter->search(
        {   q   => $query,
            rpp => $num_results_display,
            $since_id ? ( since_id => $since_id ) : ()
        }   
    );  

    unless ( defined $response ) { 
        printf "error while following: %s\n",
            $twitter->get_error->{error};
        exit 1;
    }   

    for my $tweet ( @{ $response->{results} } ) { 
        next unless $tweet->{id} > $since_id;

        $max_id = $tweet->{id} if $tweet->{id} > $max_id;

        Mac::Growl::PostNotification(
            'tweet-follow',
            'tweet',
            decode_entities( $tweet->{from_user} ),
            encode( 'UTF-8', decode_entities( $tweet->{text} ) ) 
        );  
    }   

    $since_id = $max_id;
    sleep $poll_frequency;
}

Usage is simple:

tweet-follow hottest100

Add something like this to your ~/.tweetrc to tweak some basic settings:

<follow>
    max_display_results 7
    poll_frequency 30
</follow>

The options are pretty self-explanatory:

max_display_results controls how many tweets to post Growl notifications for.
poll_frequency configures how often, in seconds, to poll for, and display, new tweets.

If only I could get Unicode characters to display properly in the Growl boxes. At the moment they just display as question marks for me.

I made a design decision to only display the latest N tweets, rather than displaying the next N tweets since the last poll, otherwise if you are displaying 5 tweets every 30 seconds for a topic that receives 50 new tweets every 10 seconds (think #iranelection or celebrity deaths), the script will get far behind very quickly.

Unicode and Perl

2009-07-08T20:08:00.003+10:00

Tonight I was having problems with different character sets while hacking together some code to display and colorize Twitter search results on the command line (will post code another time) and ended up finally, after all these years with Perl, reading up about Perl and unicode (see perlunicode, perluniintro, perlunifaq and perlunitut).

The Encode module makes dealing with different character encodings trivial. I wish I had known about it before. I remember using the bytes pragma a couple of years ago because it was the easiest and quickest solution and I was (am?) lazy, but it is recommended to not use it.

Project Euler problem 28

2009-07-01T19:01:00.005+10:00

DISCLAIMER: This post contains a solution to Project Euler problem 28. Go away if you don't want to know. DISCLAIMER 2: I am not the most mathematically minded person, so this may/will seem slow for pro mathematicians.

See here for the problem description.

So I got around to solving problem 28 last night (14 more until I get to level 2!) and, like most problems, I quickly found out the reason why this problem is on Project Euler: there's a subtle and beautiful pattern hidden in the middle of what seems like a random math problem.

The first pattern is that as the pyramid builds, the top right corners are always perfect squares of 1, 3, 5, 7 etc...

43 44 45 46 47 48 49
42 21 22 23 24 25 26
41 20  7  8  9 10 27
40 19  6  1  2 11 28
39 18  5  4  3 12 29
38 17 16 15 14 13 30
37 36 35 34 33 32 31

So we start looping from 1 to 1001 by 2s.

The other subtle pattern is that the gap between the corners also increments by 1, 3, 5, 7 etc... with each new layer of the pyramid. Therefor if we are at the top right corner which is n², the top left corner is n² - n + 1, the bottom left corner is n² - 2n + 2, and the bottom right corner is n² - 3n + 3, which follows an easy summation pattern which we can represent with a simple Lisp function:

(defun sum-corners (n)
  (loop for x from 0 to 3
     sum (+ (- (* n n)
               (* x n))
            x)))

Note: That can be simplified further into one equation, rather than a loop, if you like.

And a simple loop, described earlier, to tie everything together:

(defun euler-28 ()
  (1+ (loop for n from 3 to 1001 by 2
         sum (sum-corners n))))

Done. A simple problem disguised as something a little more difficult than it really is and has a couple of simple but cool patterns.

Vim-style Line Numbers in Emacs

2009-06-20T20:12:00.004+10:00

Being a heavy Vim user throughout my university degree and now throughout work, I get very used to having the line numbers down the side (with "set numbers" in a .vimrc) as I hack code. To get something like this in Emacs, there is linum-mode.

You can toggle linum-mode for any Emacs buffer with the command M-x linum-mode. Or if you want to set it globally for every file you edit, add this to your .emacs file:

(require 'linum)
(global-linum-mode 1)

linum-mode should come pre-installed with Emacs 22 and beyond. It is also bundled with Aquamacs 1.7 (and maybe earlier...)

Getting a Perl REPL in Emacs

2009-06-14T19:48:00.004+10:00

Although Perl doesn't really have a REPL, you can use Perl's debug mode to obtain something as close to a REPL as you can get without installing extra software with the command perl -de1

So when in Emacs (or XEmacs or Aquamacs or whatever), using the following Emacs commands in sequence will open Perl's debug mode inside a horizontally split window:

C-x 2
C-x o
M-x shell
perl -de1

And away you go.

When to use defparameter and defvar

2009-06-11T07:26:00.005+10:00

Because I always forget:

< gigamonkey> Say you were running a genetic algorithm and you
              said (defparameter *populations* nil)
< gigamonkey> Then you run it for a few days and *populations*
              is full of thousands of populations you've
              generated. Then you change some code in the file
              and reload it. Ooops. You've just wiped out your
              *populations*.
< gigamonkey> (defvar *populations* nil) saves you from that.
< gigamonkey> Conversely if you have
              (defvar *mutation-rate* .01) and then you decide
              to tweak things and change the .01 to .001 and
              reload the file, you'll be confused when nothing
              changes.
< gigamonkey> In that case you should have used defparamater.
< gigamonkey> spelled right, of course.

Tweeting from the Command Line with Perl and Net::Twitter

2009-06-06T14:53:00.012+10:00

The Net::Twitter module implements the entire Twitter API. This gives us the ability to create Perl scripts, small and large, to interact with Twitter.

Although I use Twitter a bit, I'm not that heavy a user that I need a massive application to tweet from. Instead I usually just use the web interface. But sometimes even that's too much if the tweet is short. Here's some code to tweet from the command line:

#!/usr/bin/env perl

use warnings;
use strict;
use Config::ApacheFormat;
use Net::Twitter;
use Term::ANSIColor;

use constant TWEETRC => "$ENV{HOME}/.tweetrc";

die "usage: tweet <message>\n" unless @ARGV == 1;

die "tweet: Unable to locate ~/.tweetrc file\n"
    unless -e TWEETRC;

my $tweet = shift;

if ( length $tweet > 140 ) {
    print "error: tweet is longer than 140 characters\n";
    print color 'green';
    print substr( $tweet, 0, 140 );
    print color 'bold red';
    print substr( $tweet, 140 );
    print color 'reset';
    print "\n";
    exit 1;
}

my $config = new Config::ApacheFormat;
$config->read(TWEETRC);
$config->autoload_support(1);

my $username = $config->username;
my $password = $config->password;

my $twitter = Net::Twitter->new(
    {   username => $username,
        password => $password
    }
);

my $result = $twitter->update( { status => $tweet } );

unless ( defined $result ) {
    printf "error updating status: %s\n",
        $twitter->get_error->{error};
    exit 1;
}

To use it, create a .tweetrc file in your home directory. Add something like this:

username foo
password bar

Replace 'foo' and 'bar' with your Twitter username and password, respectively. Make sure you set permissions correctly so other users can't read your .tweetrc.

And that's it. I place the script in my ~/bin directory, which is in my $PATH, under the name tweet with executable privileges and then run it like this:

tweet "I am a BANANA!"

Originally I didn't want to have to have to put quotes around the message, but shell meta-characters (like single quotes as an apostrophe) were getting in the way, so this was the safest alternative short of reading from standard input.

UPDATE: The code in this post has a problem which I have posted about here.

Checking Perl Syntax Within Vim

2009-06-03T20:29:00.008+10:00

Sick of switching back and forth between your shell and Vim buffer fixing syntax errors and adding forgotten "use" statements? Running perl -c is a handy command to run on your Perl source files to check for syntax errors. Here's a little key mapping to add to your .vimrc file:

map ,pc <Esc>:! perl -c %<CR>

Now type ,pc (mnemonic: perl -c) in command mode to check the syntax of your Perl source file without having to switch windows. It is a little annoying to have to do it multiple times (if there are many errors) since Vim swaps between the file you're editing and the shell to show the output, but it's definitely a step above manually swapping windows :)

Yet Another Blog sigh

2009-06-03T20:10:00.000+10:00

Ok. I've had many blogs before. On those blogs I've promised to blog regularly and failed miserably. This time I make no such promise to anyone (not even myself), that way I won't feel bad when I inevitably run out of things to write about. But what am I going to write about? Well, I named the blog "Luke's Thought Dump" for that exact reason: a place to dump thoughts. Any little tricks I pick up (for Vim, Emacs, Perl, Python, Lisp, Linux, OS X or whatever), that I want to remember for some time in the future, will end up here, and I've already got a handful in mind that I've picked up since starting my new job back in February. So that's it. I'll make a small post this weekend when I have a bit of time :)