Perl for DevOps: IO::All

A stupidly common task to perform is file and directory IO. In the Perl world, the IO::All module has wrapped up nearly every common IO task I can think of into a very expressive interface. With the many options available to perform the same task, it can fit into scripts in many different ways.

For example - as a sort of "hello world" of file IO - if I wanted to read the contents of a file, do some basic processing and then output to a new file, here is a very simple solution:

use IO::All;

my $contents < io("foo.txt");
$contents =~ s{foo}{bar}g;
$contents > io("bar.txt");

Or, if you're not a fan of operator overloading, that's cool too! Here's the same script, with a more explicit usage:

use IO::All;

my $contents = io("foo.txt")->slurp;
$contents =~ s{foo}{bar}g;
io("bar.txt")->print($contents);

And there are a bunch more options to do similar things in the documentation.

What about reading a file backwards? This is sometimes useful to look for the last instance of an event in a log file:

use v5.10;
use IO::All;

my $io = io("/var/log/maillog");
$io->backwards;
while (my $line = $io->getline) {
  if ($line =~ m{ dsn = 4\.5\.0 }xms) {
    say "last success: $line";
    last;
  }
}

What About Directories?

Perhaps we wanted to traverse /var/log recursively and list out anything that's gzip compressed:

use v5.10;
use IO::All;

my @files = io('/var/log')->deep->all_files;
foreach my $file (grep { $_->ext eq 'gz' } @files) {
  say $file->name;
}

Something I've had to do on more than one occasion - when bringing up and initialising a new VM - is create a directory structure and all of the parent directories with it:

use IO::All;

foreach my $a ('0' .. '9', 'a' .. 'f') {
  foreach my $b ('0' .. '9', 'a' .. 'f')
    io->dir("/var/my-application/tmp/$a/$b")->mkpath;
  }
}

So What?

The tendency is to just use bash scripts for a lot of these tasks. But bash scripts become unwieldy when the scope of a tiny script creeps, and it now needs to compress files, encrypt data, maybe upload stuff to S3, logging everything it does along the way to a centralised location, perhaps logging all errors to a Slack channel, or maybe just sending a notification to a Slack channel when the job is done. Perl is more than ready to handle those tasks.

I'll tackle some of these modules and tasks in more detail in future posts.

Worthy Mention: Path::Tiny

Although more can be accomplished with IO::All, the Path::Tiny module is also worth knowing about. There have been certain times where I've needed more specific control that IO::All doesn't provide. In those cases, Path::Tiny usually does what I want, so it's a handy backup tool and worth knowing about.

Between these two modules, pretty much all filesystem IO needs should be taken care of.

So Much More

I'd encourage anyone to look through the docs. There are tons of examples for all kinds of tasks that I haven't touched on at all in this post, even as far as being able to send emails via a plugin.

Unless - or until - you need to use the low-level IO functions for fine-grained control and/or better performance for a critical piece of functionality, the IO::All module (and Path::Tiny as its companion) should be more than enough almost all of the time.