Performance: Pick Better Libraries/Frameworks

I both love and hate CPAN; it has a module for every need, but often, for one reason or another, those modules aren't written by someone who cares about performance.

For example, Crypt::Blowfish is a XS module, which is great, but it's meant to be used, for example, via Crypt::CBC, which is a pure-Perl library, and it runs terribly slow if you encrypt a lot of data (and we do). On the flipside, Crypt::Mode::CBC is a XS module and performs brilliantly (as does the rest of CryptX). Swapping from Crypt::CBC/Crypt::Blowfish to Crypt::Mode::CBC/Crypt::Cipher::Blowfish, we noticed a 7x performance gain in our system.

In the Perl world, when utilising microservices is the right tool for the job, Plack/PSGI is the winning platform for writing RESTful webservices.

An old service that I was maintaining used HTTP::Server::Simple::CGI, which was originally used for the sake of simplicity and getting something up and running with what seemed like a lightweight framework. It was also able to plug in a Net::Server subclass to leverage Net::Server's features, which was desirable from an operational point-of-view given we'd used Net::Server successfully for other purposes. However, HTTP::Server::Simple is terribly written from a performance point-of-view.

Simply porting the service over to Plack/PSGI and serving it out of Starman resulted in a 6x performance gain.

Popular web frameworks, like Mojolicious, support PSGI out of the box, if you'd prefer the abstraction they give you. However, in my experience, even a "Hello World" will run significantly slower. But, depending on the complexity of your service, the performance hit may be worth it.

Performance: Compress Your Payloads

... unless the CPU is the source of your latency.

Compressing payloads requires a little more CPU, but it can save on network IO, i.e. bandwidth, which can also have cost-saving benefits if your bandwidth costs are high.

The beauty is, if you're serving content out of a HTTP server, the popular ones can handle this for you with a small amount of configuration.

See:

• HTTP compression
• nginx compression

Performance: Use Persistent Connections

Small Changes First

The first few points of this series are going to be around moderately small changes for the greatest gain.

It's easy to recommend the complete overhaul and re-engineering of a system using all the latest and greatest technologies and platforms available, but for many organisations, especially if you're having to maintain and slowly migrate a monolith to something more granular, that kind of change takes many months (and longer) to achieve once you factor in costs (both in time and dollars), training, testing, competing business priorities, etc...

As I run out of quick wins, I'll get into more architectural changes because they're pretty much inevitable as a system grows.

Be Persistent

If a single process is making multiple network calls to the same service, it makes sense that maintaining a persistent connection instead of creating a new connection for each individual call will lead to a performance gain; it reduces the number of connect() syscalls, the number of TLS handshakes and any sort of authentication/authorisation that happens once a connection is established.

See:

• HTTP persistent connections
• SMTP pipelining
• Connection pooling

The question then becomes: can the endpoint you're connecting to (if you control it) handle the number of persistent connections you'll have open in the worst case?

Performance: Profile Your Damn Code First

I started jotting down notes for this post without much of an idea of how I was going to present them; I was just working on a bunch of performance issues and investigating potential future improvements and noticed a number of recurring ideas across different systems, and I thought they were worth sharing with the world.

Originally I wanted to post all of these ideas in one large post, but with the number of things to discuss, it felt like I was never gonna get the damn thing finished, so I decided to split it up instead.

This isn't for the developer who is already well-seasoned in tackling performance issues; it's for the developer who may need to do it soon, but doesn't know exactly where to start.

Although the points in this series will be on the Perl-ish side, the ideas should easily transfer.

Latency

Mostly what I'm going to be talking about in this series relates to latency.

With the various infrastructure providers around today, if a system is running slow, it's pretty easy to add more machines to the mix to pick up the slack. But all you've done is increased the capacity of the system. The latency - the time it takes to serve a single request - still sucks, no matter how many more machines you add.

Until you address the latency issues, you won't be utilising all of the resources of the individual machines you're paying for and your cost-to-serve (which your CFO cares about) will be unnecessarily high. That's bad.

Stretching the hardware and infrastructure you've already got by managing and minimising latency lowers cost-to-serve and increases capacity and throughput.

Use a Profiler

I feel like this goes without saying, but evidently, I've found that having a vested interest in this kind of work makes me the exception to the rule, so: profile your code first. If you have a slow application, there's no point guessing which bit is the slow bit.

In the past, I've seen slow page-load times and memory spikes, and everyone was sure what the cause was going to be, only to hook up NYTProf and quickly see from the flame graph that a Class::DBI relationship inflating timestamps into DateTime objects was the cause.

Don't guess! Profile the damn thing!