I started jotting down notes for this post without much of an idea of how I was going to present them; I was just working on a bunch of performance issues and investigating potential future improvements and noticed a number of recurring ideas across different systems, and I thought they were worth sharing with the world.
Originally I wanted to post all of these ideas in one large post, but with the number of things to discuss, it felt like I was never gonna get the damn thing finished, so I decided to split it up instead.
This isn't for the developer who is already well-seasoned in tackling performance issues; it's for the developer who may need to do it soon, but doesn't know exactly where to start.
Although the points in this series will be on the Perl-ish side, the ideas should easily transfer.
Latency
Mostly what I'm going to be talking about in this series relates to latency.
With the various infrastructure providers around today, if a system is running slow, it's pretty easy to add more machines to the mix to pick up the slack. But all you've done is increased the capacity of the system. The latency - the time it takes to serve a single request - still sucks, no matter how many more machines you add.
Until you address the latency issues, you won't be utilising all of the resources of the individual machines you're paying for and your cost-to-serve (which your CFO cares about) will be unnecessarily high. That's bad.
Stretching the hardware and infrastructure you've already got by managing and minimising latency lowers cost-to-serve and increases capacity and throughput.
Use a Profiler
I feel like this goes without saying, but evidently, I've found that having a vested interest in this kind of work makes me the exception to the rule, so: profile your code first. If you have a slow application, there's no point guessing which bit is the slow bit.
In the past, I've seen slow page-load times and memory spikes, and everyone was sure what the cause was going to be, only to hook up NYTProf and quickly see from the flame graph that a Class::DBI relationship inflating timestamps into DateTime objects was the cause.
Don't guess! Profile the damn thing!
