This is more of an essay than a blog post, but this subject comes up time and again, and since I tripped across this interesting blog post by Pedro Timóteo about why he has decided not to be a sysadmin any more, I thought now’s as good a time as any to comment on what I think is a significant industry trend in production engineering work.
Pedro here has discovered the essential truth of most sysadmin or DBA jobs: if you’re any good, you will soon be bored and under-appreciated. (That’s a different blog post of his, also worth reading, linked there.)
That’s because the best way to be excited and appreciated in a production engineering role is to let things fail, and then come in riding your white horse to fix things up when everyone notices. There’s only one problem — this is also distasteful to those among us who believe in doing a good job by ensuring things never fail.
Another good way to stay busy is to do everything manually all the time — clone databases, create users, etc. This is also distasteful to those of us who have seen and put to work the huge savings available by automating routine production engineering tasks such as daily verifications.
So what do we do instead? We tune, automate, streamline — in Pedro’s words “some software upgrades here, some tuning there, some cron entries here, some scripting there, some changes to the network, and so on.” Next thing you know, “most of your job is done.”, you’re not so busy any more and there’s plenty of time, which your employers are happy to fill with
dumb, repetitive, non-sysadmin (and therefore non-scriptable) tasks — which, since you have free time, you probably can’t refuse, or at least feel you can’t. Any raise or promotion will certainly not go to you, but to your “hard-working” co-workers, who are always so “busy” and have so much “work” that they stay at work every day after 6, that they can never do a task “right now”, but only in a week’s time, and that, even their own results are much inferior to yours, it’s you who’re not “dependable”, “dedicated”, or “competent”.
This is the quintessential problem of the best production engineers. It is the impetus bringing the finest of them to come work at Pythian, and driving a large part of our growth.
It’s also a huge piece of the dynamic for most of our competitors’ profit models as MSPs, and the primary differentiator of Pythian vis-a-vis the lot. And finally, it is the underlying factor in why we believe here at Pythian that we are the vanguard of a broad IT shift away from using solely in-house resources for the production engineering functions of database and systems administration. Let me explain.
Our service model allows our customers to subscribe to our services based on a “co-op” model, essentially a retainership expressed in hours per month. We have customers subscribing to the tune of 16 hours per month, our minimum, up to 400 hours per month, our largest routine contract, and indeed up to 700 hours per month, our largest month for a single customer. And everywhere in between; we have tons of customers in the 80 to 250 hours per month range (70 active customers in total as I write). Customers can change their allocation with 30 days’ notice at any time.
We do all the automation, tuning, and streamlining work expected of any production engineer, but there’s a catch — when we automate or tune something and our workload decreases, we don’t need to get bored or have our customers fill up our days with make-work projects. Our customers can literally downsize their contracts in step with our success at tackling the workload. Some major success stories have us handling shops that used to have a full-time DBA with as few as 40 hours per month, one year in.
In the typical MSP model, where companies charge a flat monthly rate in exchange for a checklist of items being done, the fact that good engineers can quickly and dramatically reduce the amount of work that needs doing is in essence their profit model. They’re just like us in that their major cost is their personnel: they are essentially a human services company. However, since they negotiate the monthly rate up-front for a set list of service items, whenever they automate, tune, or streamline, they keep the benefits for themselves, charging the same rate indefinitely or for the term of the contract.
Even worse, when their customers invest in faster hardware or more storage, eliminating the need for tuning and for some configuration and storage management work, they still keep the benefits for themselves. And when the RDBMS or OS vendor releases a major upgrade that further eliminates “busy work”, which has been a major feature of every such release (remember how we used to need to “defragment tables” and how long that used to take?) they still keep those benefits for themselves, even though it’s their customers paying the licensing maintenance costs for those upgrades.
At Pythian, all of those savings are given back to the customer and all we get is our modest mark-up on our rate. For the typical MSP, the fact that it becomes obvious how little the vendor is spending to deliver on the checklist causes huge friction between the service provider and the customer. Meanwhile, at Pythian, since the customer gets to keep those savings for themselves, we get to retain the customers. For instance, we have two active customers that have been active customers continuously since 1999 — 100 monthly renewals and counting! And eight customers that have been customers since 2001 or longer. This is because our model allows them to make working with us a long-term strategy.
So why do I say I believe we’re at the beginning of a broad IT shift away from using solely in-house1 resources for production engineering work?
- The best DBAs would rather work at Pythian. Much less boredom. Working directly in their field in a company specialized in their area of expertise. Serving multiple businesses and multiple platforms simultaneously. Working with the best and brightest in the field. Being able to blog and prepare presentation abstracts, attend and present at conferences, all on company time. The best DBAs would also rather work at Pythian than at a typical flat-rate MSP provider, for the simple reason that our model allows fun and challenging work to be routed to us, whereas our competitors’ DBAs are stuck delivering the same checklist and that’s it — who would want to do that!?
- We’re saving our customers money while improving the quality of service to their production operations.
These two factors point in the same direction and reinforce each other. The first means that Pythian is a great place to find the leading technical resources (witness our participation in IOUG Collaborate 07 in April and in the MySQL Conference and Expo two weeks ago). The second factor, coupled with efficient markets, means that companies will be taking advantage of the cost and quality advantages of outsourcing some or all of their DBA and sysadmin operations to Pythian or companies much like it.
So you know quite a bit more about Pythian and how our service works now. If you think Pythian can make a contribution to your IT operations, either by outsourcing a DBA or SA opening, or by blending us in, please contact us and let us explain how we might be of service to you.
1. I say “solely in-house” here because one of the successful ways Pythian is put to work by our customers is by blending us in with their in-house resources to tap the synergies while retaining the advantages of the in-house personnel within a larger team. A good example of this is at the University of Pennsylvania, where they have four in-house DBAs blended seamlessly with Pythian.