Host Discovery via SPF Records?

So... my once-a-month posts OCD issue went unsatisfied for the past 2 months. Oh well. Back to business...

Sender Policy Framework (SPF) records are often used as one method of combatting spam; specifically, spam that wants to look like it has been sent by your company. The general idea is that your SPF records specify which hosts and IP addresses are allowed to send email using your domain(s) in the envelope 'from' address. Any other host, which is not contained within the SPF policies, that attempts to send mail from your domain will fail the SPF check and, hopefully, get picked up by some kind of spam detection software further down the track. It is highly recommended that you setup SPF records for any domains that you own.

How are SPF records requested? Via DNS. They are usually contained within the TXT record, however they can also, occasionally, be found in the SPF record aswell. The format is very easy to understand. An typical example to look at would be optus.com.au:

IN TXT "v=spf1 mx/24 include:opt01._spf.optin2.com.au ip4:180.92.216.0/21 include:rightnowtech.com include:rnmk.com include:custhelp.com ~all"

Do the hosts (and the range of IP addresses) in the policies belong to optus.com.au? Maybe. Maybe not. We cannot discern that just by looking at the host names, nor can we, within any real degree of certainty, determine it programmatically. However, the hosts' SPF records have been trusted to send mail on behalf of optus.com.au, so they remain hosts of interest for penetration testers and may not pop up in other DNS requests (A, MX, AFXR, etc..)

What is a common example of a host who would be in a domain's SPF policies but not actually part of the company who owns the policy? Any mail filtering company who filters outbound mail.

Just a thought that has been floating around in my head...

End of the Month...

Well, it's the end of the month, which seems to have become the day that I write a new blog entry (I feel somewhat empty if there's a month missing from the sidebar over there), but I don't have anything very interesting to post, so I'll post something a little less interesting: A github project I use for testing libraries, Mazer.

Mazer is a rewrite of an old assignment I did years ago in C++. Originally it just had to automatically solve a maze step-by-step, as if simulating a human walking through the maze. I often write this same program when I start learning a new programming language (which I did for Haskell and Common Lisp), or, as was the case this time around, to play around with libraries. It was the first bit of code I wrote using Moose, Test::Class (the tests actually pass!) and, most recently, Curses::UI.

There's nothing special about it. It has basically become my sandbox for Perl libraries. Perhaps others would find it useful, though.

The State of the PDF::API2 Graphics Object

Often when I write scripts with PDF::API2, I modify the state of my graphics object a lot. Sometimes to change the current fill color, or line thickness, or perhaps to set line dashing. Usually, once I have finished drawing the object, I want to set the state back to what it was before, and then continue on with the rest of the script. Rather than having to either re-set everything on the graphics object or use a new graphics object, there are two methods that will help you out.

The save and restore methods will, surprisingly, save and restore the state of your graphics object. What makes it even more handy, is that you can save multiple states and have a stack of states. Some example code:

#!/usr/bin/env perl

use warnings;
use strict;

use PDF::API2;

my $pdf = new PDF::API2;

my $page = $pdf->page;
$page->mediabox( 100, 60 );
my $gfx = $page->gfx;

# Make it red.
$gfx->strokecolor('red');
# Save the first state
$gfx->save;

$gfx->move( 10, 10 );
$gfx->line( 90, 10 );
$gfx->stroke;

# Make it blue, a bit thicker and add some dashing.
$gfx->strokecolor('blue');
$gfx->linedash( 1, 5 );
$gfx->linewidth(5);
$gfx->move( 10, 20 );
$gfx->line( 90, 20 );
$gfx->stroke;
# Save the second state
$gfx->save;

# Set some new parameters.
$gfx->strokecolor('yellow');
$gfx->linedash();
$gfx->linewidth(3);
$gfx->move( 10, 30 );
$gfx->line( 90, 30 );
$gfx->stroke;

# Revert back to the 'blue' state.
$gfx->restore;
$gfx->move( 10, 40 );
$gfx->line( 90, 40 );
$gfx->stroke;

# Revert back to the 'red' state.
$gfx->restore;
$gfx->move( 10, 50 );
$gfx->line( 90, 50 );
$gfx->stroke;

$pdf->saveas("$0.pdf");

Run the code and you should get the following:

Handy, I think.

PDF Library for Laying out Widgets... or something...

At work I have pretty much become the developer that writes the code to produce reports, whether the output be in PDF, CSV, XLS or whatever format is required. When I started this job, the first task I was assigned was to create a particular report (mentioned in a previous PDF::API2 post), and even now, a year and a half later, I am re-writing a handful of reports for our client-facing web console. Most of the reports I develop, new ones in particular, are outputted as PDF documents.

My process for developing PDF reports is simple:

1. Figure out what data is going where.
2. Measure up the coordinates.
3. Do the magic that gets the data and outputs it in pretty graphs.

After writing code for dozens of reports, step 2 has become a lot faster than the first time I did it, but it's still a manual job and it still takes time, and since the process of measuring the coordinates is always the same for me, I want to automate it. I want to be able to hand a bunch of objects (or widgets) to a function or class and have it automatically figure out how and where to lay them out - in a grid or column layout, for example - given certain metadata, e.g. widths, heights, padding, font. I also want the ability later to modify a specific object's metadata if a particular graph or table feels like it would look more appealing with a little extra padding on the left, for example.

I noticed there is a PDF::APIx::Layout module, however it hasn't been updated since 2005.

So far I only know what I want it to do, and I have a basic idea of how the backend, PDF::API2 code might look. But I still haven't completely decided how I want the API to look to the developer; procedural, OO, declarative or some fusion of them all? I may even end up trying them all out for myself to see which is easier to develop with.

I will post more updates in the future, regarding the API, and any code will probably end up on GitHub, but that is basically the idea that has been floating around in my head for a few months.

Light Mojolicious::Lite Experience

I've been playing with Mojolicious::Lite for the past couple of weeks. Why? Well, I had started a small project which records public data from a MMORPG that I frequently play (don't ask which one... it is no longer worth playing or even giving a free mention to) and needed a pretty front-end that myself and some friends could use to view the data.

Choices, Choices, Choices

The initial code I had written for the project was written in Python, which was two simple scripts; one that ran in a cron job once per day to retrieve the data as XML and one that analysed the data and outputted an ugly, little summary. I had intended to store some of the raw data in a database and wanted a web front-end to display it all and allow my non-technical friends to view and/or export the data as Excel spreadsheets.

Since this was a hobby project and I like to learn new libraries/frameworks for hobby projects, I had two decisions to make:

• Which library for database access?
• Which web-framework?

I was itching to use DBIx::Class for something, especially after using Class::DBI at work and hearing about the wonderful things DBIx::Class has that Class::DBI doesn't. But that's a Perl library, so I rewrote the Python script that ran in a cron job in Perl, so I could later hook DBIx::Class into it.

Deciding on the web-framework was next. A couple of years ago at university I had used Sinatra in a Ruby project and loved it and since then I have had a thing for web micro-frameworks. Python has Flask, which I always had in the back of my mind, and I like the way it uses decorators for the routing. Perl, luckily, has Mojolicious::Lite. I realise now that Dancer exists, however I found Mojo first (sorry, Dancer devs).

What I Like

The usual good things about web micro-frameworks: Rapid development.

The mechanism that Mojolicious::Lite uses for routing is very nice, feels very Perl-ish and allows for very compact code.

What I Don't Like

Errors in templates can result in cryptic error messages. For example:

Scalar found where operator expected at (eval 488) line 25, near "} $_M"
        (Missing operator before  $_M?)

And:

Template error in "rank.html.ep": syntax error at template line 2, near ";
$_M "
syntax error at (eval 488) line 25, near "} $_M "
syntax error at (eval 488) line 29, near "} }"
1: <% my $server_table = { %>
2:      <% my ($ranks, $server, $align) = @_; %>
3:
4:      <div class="rank_list <%= $align %>">

What caused the errors? The following two lines of code were:

<% my $server_table = { %>
...
<% } %>

When they should have been:

<% my $server_table = {%>
...
<%} %>

So a couple of spaces in the template produced those error messages. Not terribly obvious, unfortunately.

In the end...

A cryptic error message from a template error is far out-weighed by the things I like, and HTML::Mason's error messages have caused me much bigger headaches. Most of the time, any personal projects that I have only ever, if at all, need a super basic web interface, and I like to get my web applications up and functional as quickly as possible. Mojolicious::Lite let me do exactly that for this project and I intend on using it again in future projects.

PDF::API2 is Underrated

I feel like there has not been enough talk about PDF::API2 in the Perl community. Searching for code examples and tutorials does not yield many results, however it does lead to this tutorial, which is the only useful one that I have been able to find. It is worth reading. Oh, and it's written by my boss ;) Go and read it if you plan on using PDF::API2 in the future.

At work, for the past year, we have used PDF::API2 to create this nice looking report and many others like it. Since then, I have learnt a number of lessons and picked up a few tips for using this powerful library, so here are some of them. This is by no means a tutorial, or even an introduction to PDF::API2, so if you are looking for something like that, refer to the tutorial I referenced in the previous paragraph. This is more of a coredump of the part of my brain that knows PDF, so to speak.

The coordinate system is that of a Cartesian plane

Personally, every toolkit I have used in the past (which, admittedly, is not a long list) has placed the origin (0, 0) in the top-left corner. I assume, without actually researching, that this was because it was easier for developers to think about placing the top-left corner of a widget/window around the screen. We (mostly) read articles and books and posters from the top-left to the bottom-right.

However, coordinates in PDF::API2 work differently; the origin resides at the bottom-left of the document, just like the first quadrant of a Cartesian plane. Once you think of it this way, it becomes a lot easier to visualise what you are doing and the placement of your content.

Work from the bottom-left

This is linked to the previous point. If you are factoring out specific data or widgets into separate classes or functions, specify the position of the data by the bottom-left coordinate, rather than the top-left. This saves a lot of recalculating when you have to use various text/graphics object functions.

I prefer to refer to the coordinates as 'bottom' and 'left', which clears up any confusion that using 'x' and 'y' might introduce. I recommend doing the same.

Occasionally you may need to work from the top of your widget, e.g. for tabular data, in which case you can simply use the bottom + height to get the top-most Y-coordinate.

Useful CPAN documentation

PDF::API2 documentation is shocking. It's horrible. It's almost non-existant. However, PDF::API2::Content contains 95% of the functions that you probably want. Specifically, it contains all of the member functions for the text and graphics objects, which are the objects you will use most of the time.

Useful function that you might skim past

$text->advancewidth('foo bar', \%options)

Given the state of your text object, i.e. font name and size, this will return the width, in points, required to display the text 'foo bar' on the page. Great for figuring out how much space you need to display content like names of people. If you do not want to use the current state of your text object, the second parameter, with text object settings, can be used instead. I have never needed to use the second parameter so far.

That's it for now. In the future I would like to write some more about this library - preferably with more structure and a nice example rather than random ramblings - but that will have to wait until I can spare the time.

Broken Sound in OS X

I use a MacBook with OS X 10.5.8 at work. Every now and then, at work, while streaming radio off the internet through iTunes, running Parallels, an IRC client, a Twitter client, an MSN client, a dozen terminal windows/tabs, Chrome with a dozen tabs, Firefox with a handful of tabs open, 30+ Stickies floating around and having not restarted my laptop for 2 weeks, my sound, all of a sudden, goes distorted and choppy (technical term!), which is very annoying when you are streaming radio to four other developers through a surround sound system.

At first I was just restarting my laptop. But that is a little annoying when you have that many programs, websites/tabs open and have to spend 15 minutes remembering what exactly it was you were doing when the sound decided to go rogue.

I noticed that when this occurs, the coreaudiod process starts to use a lot more CPU than usual, i.e. it becomes one of the top 4 CPU-hungry applications according to top. After killing the coreaudiod process, a new coreaudiod process was spawned and my sound returned to normal. Success!

New Perl Hate

You don't work with a language every day without building up a "things I hate in language X" list. Well, I have a few hates of Perl, but they've all been fairly well covered in other blogs and cover the usual topics (pre-Moose OO, getting bitten by scalar/list context, etc), however yesterday I came across a new one. Here is a one-liner demonstrating the problem:

$ perl -MData::Dumper -le'@a=qw/a b c/; @foobar = map { $_ } @a[0..7]; print Dumper \@a'
$VAR1 = [
          'a',
          'b',
          'c',
          undef,
          undef,
          undef,
          undef,
          undef
        ];

Fairly obviously, I have created a list with 3 elements in it, tried to slice it with 8 values and assign that to a different list, however when printing the contents of the original list, it has now been padded out to 8 values with undefs. Ok, I thought perhaps doing a slice on a list had that kind of destructive behavior but I had just never noticed it before and had never heard about it previously. So I tried another one-liner to perhaps prove that theory:

$ perl -MData::Dumper -le'@a=qw/a b c/; @foobar = @a[0..7]; print Dumper \@a'
$VAR1 = [
          'a',
          'b',
          'c'
        ];

Nope. Bug? Or expected behavior for some reason that I cannot figure out? This was on Perl 5.8.9

Diving into Reverse Engineering

Last year I read a blog entry (linked below) about a programmer who edited a component of Skype to fix a bug which was causing problems for many users. The blog entry then showed a very simple example of how to edit programs using OllyDbg without the source code. Even though I always knew the basic idea, I had never seen, or even searched for, any clear examples because I had no immediate need to know the details.

There is a MMORPG that I play on the weekends and recently there was discussion on our guild's forum about client tweaks. In the back of my mind, I knew I could probably apply a little reverse engineering to remove annoying art that clutters up the screen, change font colors to be easier on the eyes, and on top of that, maybe figure out how to fix a few bugs which we have been pushing the developers to fix for months, with no results. The problem was that I had no idea how to reverse engineer a program.

Of course, knowledge of how programs work and assembly language was required. Luckily the former was taught fairly well during my course at university, and although we were taught assembly language for the Motorola 68k processor, it never really clicked for me, or I never had an "ah hah!" moment with it and thus never did anything practical with that knowledge.

But now I did! So for the past ten days I have almost gone blind reading articles about reverse engineering and looking at assembly code.

Despite the moral questions, it has been a very interesting exercise and I'm happy that I've found a practical exercise for reverse engineering and assembly language that interests me. So far I've managed to fix one, semi-annoying bug and modified the layout to remove various things that get in the way of seeing what I'm trying to do/click on..

Below is a link to the initial blog entry that piqued my interest and some of the more useful resources I came across over the past ten days:

• Patching a Program Without Source Code: How to be like the Skype Hacker for Newbies
• Under the Hood: February 1998
• Under the Hood: June 1998
• x86 Win32 Reverse Engineering Cheat Sheet [PDF]

Perl Profilers - Devel::NYTProf

Keeping up the spirit of a blog post from Leo Lapworth, I am advocating the use of Devel::NYTProf and doing my bit to get its name out there. On a number of occasions it has saved me a lot of time searching for bottlenecks in my Perl code and also pointed out parts of code that seemed unsuspicious.

The docs on the CPAN page are very clear in its usage, so I won't bore anyone with the details here.

So as far as profiling Perl code goes, Devel::NYTProf is definitely the winner!

Programming Goals for Common Lisp

First post of the year... Somehow I was too busy for the whole of January to post anything. Anyway, moving on...

I've been yearning to use cl-opengl and Vecto for SOMETHING but can never seem to come up with any ideas. The only useful thing I have used cl-opengl for is a cellular automata simulator, but it's all very basic and minimal. So instead of trying to come up with a new idea, I'm going to re-use an old idea and extend cl-mines; my small, currently CLI-based, Minesweeper game.

I intend to use Vecto to generate each of the images for the mines and uncovered tiles. Not the most intense usage for Vecto, but it's better than nothing for now.

And I, obviously, intend to use cl-opengl for displaying and running the whole game.

Another, secondary, goal is to use more macros, if appropriate. Macros are a feature of Common Lisp I am yet to use in a complex way and is a skill I think I'm missing out on with Lisp.

So that's what I am going to be slowly working on in my spare time between everything else! That should keep me busy for a little while, I hope.