Perl for DevOps: Mojo::UserAgent

There's no way anyone "doing devops" can work without needing to interact with products/systems remotely, and these days pretty much everything offers an API and that API is more than likely available over HTTP. The Mojolicious distribution provides a ton of useful modules for doing almost anything HTTP-related; both client- and server-side.

This post is going to mostly focus on writing client code with Mojo::UserAgent, and some utility modules that Mojolicious ships with. I will also touch on using Mojolicious::Lite for writing web interfaces and services, but deployment will be deferred until the next post, which will focus on Plack/PSGI, the various Plack servers available, and some of the web frameworks that support PSGI.

Mojo::UserAgent vs LWP::UserAgent

For the longest time, LWP::UserAgent has been the de-facto standard HTTP client library that the Perl community uses. But I've written before about why I prefer Mojo::UserAgent over LWP::UserAgent; it has both blocking and non-blocking interfaces, so you can write both kinds of applications and only need to know one API, should you get into writing non-blocking code in your devops journey.

But there's more.

Prototype with mojo

The mojo command is provided by the Mojolicious package, and is a great tool to query web services and pages. I wouldn't call it a curl or wget replacement (that's not its purpose), but it's a great tool for extracting data that is embedded in a response, for use in shell scripts, and for rapid prototyping before converting to Mojo::UserAgent in a Perl script.

As an example, if you wanted to use the MetaCPAN API to get the latest version of a package:

---

$ mojo get http://fastapi.metacpan.org/v1/package/Mojolicious
{
   "version" : "7.74",
   "module_name" : "Mojolicious",
   "dist_version" : "7.74",
   "file" : "S/SR/SRI/Mojolicious-7.74.tar.gz",
   "distribution" : "Mojolicious",
   "author" : "SRI"
}

---

This shows the JSON output of the API request. Mojolicious comes bundled with its own JSON decoding/encoding module, Mojo::JSON, which can be used directly in any application you want - perhaps as a replacement for the other JSON modules, if you so desired - but it's also integrated into Mojo::UserAgent, for decoding and easily extracting data from JSON responses.

The version is what I'm after. We can easily grab that with another parameter, utilising some simple notation.

---

$ mojo get http://fastapi.metacpan.org/v1/package/Mojolicious /version
7.74

---

But what if there is no nice JSON API, and we have to extract the same data from a web page? Well, we can do that too:

---

$ mojo get https://metacpan.org/pod/Mojolicious 'span[itemprop="softwareVersion"]' text
7.74

---

This fetches the Mojolicious documentation on MetaCPAN and looks for a span tag with an itemprop attribute value of softwareVersion and displays the text in the tag. In this case, we're pretty lucky that the MetaCPAN page gives us a friendly way to locate this data, but more complex queries can be used for less mojo-friendly websites.

The beauty of the mojo tool is that once you've prototyped how to extract the information that you want, you can either leave it in a bash script, or you can port the code to use the Mojo::UserAgent module and use it as part of a larger application.

---

#!/usr/bin/env perl

use v5.10;
use warnings;
use strict;

use Mojo::UserAgent;

my $ua = Mojo::UserAgent->new;
my $tx = $ua->get('http://fastapi.metacpan.org/v1/package/Mojolicious');

if ($tx->res->is_success) {
 say $tx->res->json->{version};
}

---

This is just a part of what the Mojolicious distribution has to offer; there's also an event loop and a promises implementation for writing non-blocking client and server code, and a whole lot more. Mojolicious wants to be a self-contained installation with as few external dependencies as possible, which makes it stable, and resilient to issues in the greater CPAN package ecosystem. Check out the other packages it provides.

Next in the series (whenever I get to it) I'll go through Mojolicious::Lite and Plack/PSGI, for when the time comes to write and deploy web sites and services.

Perl for DevOps: IO::All

A stupidly common task to perform is file and directory IO. In the Perl world, the IO::All module has wrapped up nearly every common IO task I can think of into a very expressive interface. With the many options available to perform the same task, it can fit into scripts in many different ways.

For example - as a sort of "hello world" of file IO - if I wanted to read the contents of a file, do some basic processing and then output to a new file, here is a very simple solution:

use IO::All;

my $contents < io("foo.txt");
$contents =~ s{foo}{bar}g;
$contents > io("bar.txt");

Or, if you're not a fan of operator overloading, that's cool too! Here's the same script, with a more explicit usage:

use IO::All;

my $contents = io("foo.txt")->slurp;
$contents =~ s{foo}{bar}g;
io("bar.txt")->print($contents);

And there are a bunch more options to do similar things in the documentation.

What about reading a file backwards? This is sometimes useful to look for the last instance of an event in a log file:

use v5.10;
use IO::All;

my $io = io("/var/log/maillog");
$io->backwards;
while (my $line = $io->getline) {
  if ($line =~ m{ dsn = 4\.5\.0 }xms) {
    say "last success: $line";
    last;
  }
}

What About Directories?

Perhaps we wanted to traverse /var/log recursively and list out anything that's gzip compressed:

use v5.10;
use IO::All;

my @files = io('/var/log')->deep->all_files;
foreach my $file (grep { $_->ext eq 'gz' } @files) {
  say $file->name;
}

Something I've had to do on more than one occasion - when bringing up and initialising a new VM - is create a directory structure and all of the parent directories with it:

use IO::All;

foreach my $a ('0' .. '9', 'a' .. 'f') {
  foreach my $b ('0' .. '9', 'a' .. 'f')
    io->dir("/var/my-application/tmp/$a/$b")->mkpath;
  }
}

So What?

The tendency is to just use bash scripts for a lot of these tasks. But bash scripts become unwieldy when the scope of a tiny script creeps, and it now needs to compress files, encrypt data, maybe upload stuff to S3, logging everything it does along the way to a centralised location, perhaps logging all errors to a Slack channel, or maybe just sending a notification to a Slack channel when the job is done. Perl is more than ready to handle those tasks.

I'll tackle some of these modules and tasks in more detail in future posts.

Worthy Mention: Path::Tiny

Although more can be accomplished with IO::All, the Path::Tiny module is also worth knowing about. There have been certain times where I've needed more specific control that IO::All doesn't provide. In those cases, Path::Tiny usually does what I want, so it's a handy backup tool and worth knowing about.

Between these two modules, pretty much all filesystem IO needs should be taken care of.

So Much More

I'd encourage anyone to look through the docs. There are tons of examples for all kinds of tasks that I haven't touched on at all in this post, even as far as being able to send emails via a plugin.

Unless - or until - you need to use the low-level IO functions for fine-grained control and/or better performance for a critical piece of functionality, the IO::All module (and Path::Tiny as its companion) should be more than enough almost all of the time.

Perl for DevOps: perlbrew and carton

I'm sick of seeing the same, old, and very dated articles/books/whatever relating to Perl and systems administration. There are a ton of Perl modules and tools available to make life easy for developers, testers and operations staff in a DevOps environment, but unless you're already deep in the Perl world, many remain fairly hidden from the public eye and hard to come by.

I'm hoping that the next few posts will show off what's currently on offer in the Perl world. Whether it's a brand new startup launching a product from scratch, or an established organisation with an already mature product, I'm not trying to convince anyone to change their main product's stack, but for all of the glue required to support the application in a production environment, Perl is an excellent choice.

Perl's "There's More That One Way To Do It" attitude has inspired a variety of modules with expressive APIs and tools that make working with Perl in a production environment easy.

But before getting into specific modules and tools, the first thing to discuss, even if it's less exciting, is the management of multiple perl versions and the management of CPAN dependencies.

What I'm going to discuss here isn't new material; an article from 2016, A Perl toolchain for building micro-services at scale, summed up a great set of relevant tools for using Perl which can be extended from building microservices to doing almost any other development work with Perl. This first post will focus on the two tools that I think are the most important.

Perlbrew

Unfortunately, most Linux distributions still ship with perl 5.8, despite reaching end-of-life years ago. This often leads to people sticking with perl 5.8 and installing modules from CPAN to the system-level perl, sometimes even using their distribution's package manager instead of a CPAN client to do it. This is a terrible idea. Often, depending on which OS and distribution you're running, the system-level perl is used for internal tools, and breaking the system-level perl starts to break other important things.

This is where perlbrew is a no-brainer.

Perlbrew is just like pyenv for Python, or rbenv for Ruby; it's a tool for managing and using various perl versions without interfering with the system-level perl.

The added bonus of running a more recent version of perl out of perlbrew is the availability of some modules which require perl 5.10 or later, having left behind 5.8 long ago, e.g. Mojolicious.

Alternatively, plenv is another tool for managing Perl versions, although it's not a tool I have a lot of experience with.

Carton

There are a few options for managing Perl dependencies. I'm only going to describe Carton, but there are also distribution- or OS-specific options aswell that cover all or some of the functionality of Carton, e.g. Red Hat Software Collections.

While not strictly necessary, carton - comparable to using a combination of virtualenv + pip for Python or Bundler for Ruby - is an excellent tool to manage dependencies.

The cpanfile (used by carton) provides the ability to specify the direct dependencies of the script or system and, along with the generated cpanfile.snapshot file which contains the full dependency tree, can be checked into a source control system along with the code it supports. The carton utility then provides the ability to use this cpanfile to create a local repository containing only the modules and versions specified in the snapshot.

Multiple cpanfiles may be used to track the dependencies of multiple different systems or subsystems.

An example setup might be to only use a carton bundle for critical customer-facing services, as you would want that environment to be as static as possible and not be prone to failure just because someone updated a dependency for a utility script. Or perhaps use one carton bundle for critical production stuff, and another one for the less critical stuff. Or perhaps a more granular setup, depending on the situation.

The caveat with carton is that for any dependency on third party libraries (e.g. IO::Socket::SSL requiring openssl, or EV requiring libev), the third party library will not be bundled into your carton repository.

Kinda Boring But Important

I feel like this was a pretty boring introduction to using Perl as a language for your devops needs, but it's an important topic that - unfortunately, in my own personal experience - can be a real pain in the ass to deal with if it's not considered early on in the piece.

Perlbrew and Carton are powerful tools, are both worth knowing and, when used in tandem, they allow any development to be as isolated as possible, so as to interfere with as little as possible on a system.