I give a lot of thought of what makes a DBA awesome as remote DBA, or a consultant. Many DBAs are very proficient technically and are very successful as in-house DBAs, but I can’t imagine them working with customers. I spent some time thinking what would make someone awesome as a consultant, based on my experience as a in-house DBA working with consultants we hired, on my experience as remote DBA for Pythian and on my observation of the amazing guys working for Pythian Consulting.
Here’s the secret: You need to give the customer a warm fuzzy feeling.
Usually this means actually solving the problem the customer has, perhaps even efficiently and professionally. But there is more to it – the customer really need to be able to trust you and feel comfortable working with you. As a consultant, the default for the customer is not to trust you. You are an outsider, you have slightly different goals than the organization that hired you and very different risk profile. You need to prove yourself over and over.
Here’s what I consider key parts of being effective at giving customers warm fuzzy feelings. Obviously all this applies in addition to being a very proficient technical problem solver:
- Be nice, friendly and polite. It sounds trivial, but often in-house DBAs are seriously snarky and evil to their users, because they see them as nags and not customers. The customer will love working with someone who says “yes sure” with a smile when you ask to fix a trigger instead of someone who starts complaining about how busy and unappreciated they are. In a previous position we had an SA consultant. Everyone was always commenting on how friendly and helpful he is compared to the confrontational in-house admins. He is now my role model for customer relations.
- Be very easy to reach. Replying to pages, phone calls and emails within minutes definitely gives the customer a good reason to trust you – they know you are always there to help them in an emergency in a timely manner. Even hearing “I can’t work on this right now, but I’ll check this in 3 hours” is more reassuring than wondering if someone got your page and is working on the problem. This is probably my weakest point – I feel that if I can’t make a technical contribution, I should just say nothing until I can. It always turns out to be a huge mistake.
- Be calm. Nothing erodes trust like hearing your consultant panic. I have a slight tendency to panic, especially under time pressure. That’s one of the reasons I’m glad I work as a remote DBA – my customers can’t see me hyper ventilating. Even if it means putting my phone on mute for a moment, I always make sure I sound calm, collected and professional when talking to the customer. Its amazing how panicking even once can cause you to be perceived as less capable forever.
- Be confident. Be very confident about things you know for certain, be confident about things that you are almost certain about but are still a good idea to do and can’t cause much harm in the situation. I used to say things like “You may consider trying X.”, but I noticed that customers feel much better when I say “Do X”. Confidence inspires trust and trust is the basis for warm fuzzy feelings.
- Be very careful about admitting lack of knowledge. On one hand, sounding confident about something you know nothing about and getting caught erodes trust. On the other hand, if you are fairly certain you can acquire all necessary knowledge in few minutes from Google, Metalink or asking a colleague, I’d say something like “I’m somewhat familiar with this issue, let me check it out and see if I can help”. The customer will get his problem solved, you’ll gain expertise you may otherwise miss. Again, its a very fine line, so use with extreme caution.
- If you are out of your depth, find someone who can help the customer. Pythian makes it very easy to escalate issue to colleagues, but this applies to any consultant. The best consultants I’ve worked with referred me to their colleagues when they thought they are not the best person to handle this.
- Communicate very clearly. This means:
- Match the detail level and technical language level to the person you are talk to.
- Don’t make assumptions. If something was even close to being unclear, review your understanding with the customer. If you are missing information, ask for it (We are against any guess, remember?). And don’t assume that the customer will know what you are talking about without a lot of background information from you.
- Be clear. Let the customer know in unambiguous language information like: How bad is the situation, what you are planning to do, what you did to fix the issue and what do you need from them.
- Don’t blame the customer for any problems. It took your 12 hours to upgrade to the latest EMC software and the customer is wondering why it took you so long? Don’t even think about blaming the customer’s sysadmins for being incompetent. Even if its true. Especially if its true. Management will (nearly) always trust the in-house employees more and this is guaranteed to strain relations.
- Don’t make mistakes. Yes, I know this sounds silly, but nothing causes more mistrust than a consultant dropping the wrong schema, shutting down the wrong database or deleting very critical archive logs. This is easier said than done, but you can build systems to help you reduce your mistake rates, such as read-only connections in Toad and different background color for terminal connections to production. My latest trick – having a personal page in the Pythian wiki called “lessons learned the hard way” with all the mistakes I’ve made and my ideas for avoiding them. I try to review the page once a week to make sure I don’t forget, because making the same mistake twice is not acceptable to me.
Any other tips for DBAs-for-rent? I’m always looking for ways to improve.
28 Comments. Leave new
Hi Chen,
Nice summary! Soft interpersonal skills are quite important to be a top consultant.
Cheers,
Ben
> Be very careful about admitting lack of knowledge
This is a typical property for a consultant ;-)
I have been blamed by my sale representatives in the past for admitting my lack of knowledge of a specific product.
Even if you know nothing about Oracle in-Memory Database, *you are* the Oracle Expert and the customer pays for an expert…
However as a customer, I hate those external employees who never admit their lack of knowledge ! I prefer the one who always ask to make sure he is not dropping the wrong schema :-)
Great Post.
However, I have to disagree with #5.”Be very careful about admitting lack of knowledge.”
Admitting lack of knowledge is a greater virtue than not admitting. Even experts have gaps of knowledge. The key is probably in how it is communicated.
If I think about number 4 and number 5 then what I think Chen is saying is that you cannot ruin the confidence level by admitting you don’t know something.
It doesn’t mean that you automatically come back with direct lie that you, of course, know everything and that particular situation. However, you do need to assure that you have access to the required knowledge and skills (whether it’s your colleagues, friends, manuals or your sharp brain). Nothing will ruin your image faster than committing to something and not delivering.
Before you can make such claims, you have to have certain trust earned already.
Obviously, the most important rule (which was omitted for obviousness) is to never ever lie to a customer. About anything. Don’t lie about what you know, don’t lie about hours worked, don’t lie about changes made on customer system.
Point #5 is for DBAs who tend to say “I don’t know X” when they mean “I feel somewhat uncomfortable doing X because I don’t have much experience with it” and what they should say to the customer is “I’d love to help with X, but give me few minutes to refresh my memory about the issue”.
If you never work outside your comfort zone, you will never learn and improve.
Good consultants typically have enough experience and knowledge to work with systems they are not 100% familiar with.
On my first week on the job I configured dataguard for a customer. I never did that before, but I had years of experience with somewhat similar technologies (streams) and I got a colleague to double-check so I won’t miss anything. It worked and the customer was happy.
As long as you solve their problems, the customer will be happy. And they will be happier if you sound confident that you can do it, so they won’t have to worry.
All of the above does not apply to restoring databases where real experience and expertise is very important to avoid causing expensive damage and data loss.
Now, I have worked as. DBA for 10 years and never got chance to do a recovery !!
Maybe, it speaks to the stable systems I built :-)
On a serious note, that is always my fear. Now what will u suggest in my strange case?
@Jcnars: that’s an excuse. You gotta practice test recovery regardless.
Alex,
Sometimes improving the client process or representing the risk factor to the client is considered the other way- What do you say about it ? Remote consultant YES man.
————
One more thing based on my experience is that Put client ‘s benefit on top of your company ‘s benefit and your benefit, In long run it will be paied back …..
@sh: If I may add a little bit of flavor… You company priorities must match customers priorities in order for service business to be sustainable and prosper.
You priorities are priorities of your company and they in turn should align with customer priorities and targets. I.e. you should never have to choose and when that happens, some actions should be taken.
Not necessarily. Your company biz path could be different from the client. That sometime make it challenge to be completely fair to the client.
If we are 100% fair with the client and treat it as the way we like to be treated, we find a final recepie.
@Jcnars:
I suggest you come work for Pythian. You’ll run several recoveries each month, guaranteed!
Seriously, though:
1. Create a test environment and practice recoveries. Invent the most horrible scenarios you can imagine and try to recover from them.
Remember that you need to test your backups frequently (otherwise they may not work!), the best test is to recover from the backup to a spare server.
2. Do you know another DBA who has more experience? If you do, definitely call him on an emergency. If you don’t, you should network better.
My first recovery was after 6 weeks as a DBA, I was glad I could call an experienced friend for advice.
Similarly, I was happy to help a friend who had to recover a system where some data files were added after the most recent backup.
I agree with Gwen on point 5.
It would leave a terrible and devastating impression on client, when you in response of an issue/request, you say to client,”oops, I don’t know this sorry.”
This would leave client wondering why in the world he hired you at the first place. The clients who dish out their databases for remote administration are certainly very intelligent and experienced people who understand that a DBA cannot know everything, but they expect their hired hand to be confident and pro-active and helpful. They expect not to hear from you ‘No.’ They expect from the hired DBA to hear. ‘Yes, please give me a little time and I will get back to you in 15 minutes.’
At Pythian, any DBA can safely give ETA of 15 minutes, even if he never encountered that issue before. We just search our wealthy repository for any past occurrence, or ask from any other DBA who is online, or page the backup DBA, or escalate it even further, or engage other teams. The whole army of DBAs is there.
A Disclaimer:
I am not a marketing guy at all. Just yet another DBA at this heaven for Databases, which we know as Pythian.
@Fahd : I understand the strategy of not telling you have never done this before and look in your note and call your colleagues.
But are you really worth your fee if you are a 15-minutes expert in, let’s say, Oracle OLAP ?
On the other hand, if you say : “No, I cannot do this, but we have another specialist in house that will take care of you shortly”, then I think you desserve more respect than being a 15-minutes-specialist
The “Before I do this, I’ll check with our in-house specialist for any new alerts in this field” gambit.
The key point is execution. Are you in a firm with the broad experience base to call upon. Or are you ringing someone who will run a Google/MyOracleSupport search while out of sight of the client. Being in the former category is the big selling point of a larger consultancy over a smaller one.
Nice Summary.
I’ve found the above traits missing from alot from dba that come and try to consult on an issue. It seems that consultant DBA’s work on a takeover rather than a partnership approach.
@Fahd saying i dont know this but i have someone that has experience and is willing to help actually puts a big warm and fuzzy for the client. Clients like to change scope midstream due to technology or requirement change. it is important for the consultant to adapt and provide with respect and solution rather than chug away and try to duke it out on his own just not to say ” I dont know” . This not only can cause irreparable harm to the consultants reputation but also the consulting companies reputation. I just recently had a similar experience with a consultant and consulting company. After the experience we were recommended the consultant a couple of times and we flatly said no to the consultant and consulting company. Asking for Help is not a bad thing and companies know that no consultants know it all but its the consultants that claim to know it all and them screw things up cause pains for the hundred’s of other good consultants out there.
@Laurent
Fahd’s job is very different than your old job. He is not a consultant, but rather a remote DBA. If he gets the job done efficiently and the customer doesn’t have to wake up at 2am, the customer definitely got full money worth. That’s the famous Pythian “intermediate DBA” – people who can get the job done without issues have different costs and expectations than top experts.
Pythian customers normally get better value than your former customers, because they don’t pay for time spent playing chess with the in-house DBAs. They only pay for actual hours worked on the problem. Which is why I believe that eventually Pythian will drive the old “contract based” model out of business – why pay for full time DBA when you don’t need one?
But I’d argue that even in your old job, when your employer sold you as top expert when you just learned a new topic, the customer also got their money worth. I’m sure you did a top notch work for them and they knew it.
Ay-ay-ay… you are getting close to piss some folks off! :) You don’t imply that all in-house DBAs are spending most the day playing games? I mean some do but not those who read our blog in the end! ;-)
While I agree, I’d also argue that there can’t be one-size fits all even when it comes to services. At the right time, in-house DBA can be the right solution (and often blended with services vendor as many customers do you know).
Oh, and as Fahd said – I don’t work for marketing either :)
Boy, this thread is going way more interesting than I thought.
Thanks for the advise Gwen. We did all the RMAN scenarios and documented it, only we are waiting for the real crash :-)
(There, I might’ve jinxed myself just b4 Thanksgiving)
We are yet to simulate a crash in our Streams POC lab where we are testing migration from 9i to 11g with zero, yes 0, downtime (maybe minutes). It will be interesting to observe how much the Streams configuration is screwed up when we do a restore. Yeah, you can call me a sadist.
@jc: you seem to be sane calling minutes outage as zero down-time. Are you working for marketing? Best marketing folks are calling for sub-zero response time. You know. ;-)
@alex, i deserved the brick. In my enthusiasm, I slipped.
let me clarify, IT IS 0 downtime at the db level. When the 2 dbs are in sync, you just shut the 9i db down. boom. it’s a new day, in a new world.
@jc nars
Its bidirectional streams? Because those are even more interesting after restore :)
Streams after restore sounds like a good blog topic actually. Maybe I’ll get to it eventually or maybe one of my colleagues will get to it first.
It’s not a bidirectional. Just a simple cross-platform (solaris-linux), cross-version (9i-11g) env with some rules to avoid replication of UDTs.
You owe me a coffee, for your next blog idea :-)
Database recovery is one thing that DBAs cannot fail at. Practice is key to keep skills up so if the unfortunate disaster ever comes up, you are prepared for it.
Chen,
Very nice article. Enjoyed every bit of it. And the comments are like extra cheese on the Pizza !
Cheers !
@chen thanks :-) of course you get the point, if you tell the customer you do not know OLAP, even if a 15-minutes briefing is enough for you to resolve his issue, and even if you are very good at problem-solving, he may use this “I don’t know” statement to annoy your sales representative to get a lower rate
@gwen Very interesting and enlightening. While I agree with almost every point – either in spirit or in text, I have a perspective different from the intended audience of this piece since I am one of those “in-house DBAs”, and I have been one for a while. Your article invigorated me to write a similar blog for being a better inhouse DBA. I will complete it some day. In the meantime, some specific observations on your piece.
(1) There is a perception for the inhouse DBA being snappy and nagging (particularly for being under-appreciated) is probably true in general; but there is a quite reasonable basis to the perception; not the fact. The reward profiles of the inhouse DBA and the consultant DBA are as different as day and night. The more time the consultant DBA spends on the jobs, the more the financial benefits are. The benefits of most (if not all) inhouse DBAs I know are pretty much static. The same goes for retribution profile. The appreciation for an inhouse DBAs is the incentive. Not getting one is a sign of retribution by management and/or users.
(2) The relationship factor is very important one. The inhouse DBA is pretty much married to the customers. Like any relationships, where there are as likely to be bad hair days as fruitcake hours, the strains do take a toll. On the other hand, the consultant comes in without the baggage. That is not the trait of the consultant as a person, but the attribute of a fresh entrant. The similar effect can be on a fresh new inhouse DBA as well.
(3) The trust factor, I have felt, is just the reverse. The consultant is usually trusted more, compared to the inhouse DBA. A whole lot of complex interpersonal factors play a role – from prior baggage to other commitments.
(4) I think your point on “admit lack of knowledge” is right on the money. This is what makes a consultant shine. When an inhouse DBA makes an assetion, s/he is held up to that statement even days later. Careers depend on it. Consultant DBA is not usually subject to such scrutiny primarily due to the transient nature of the person’s involvement.
(5) The consultant DBA is usually specialized, brought in for a specialized task. The inhouse DBAs are supposed to work on a whole bunch of stuff – from performance tuning, to backup recovery, to fighting for space, even database security. This may lead to dilution of skills.
(6) Finally, as a team lead and mentor, I want to mention here what I see as a bright star in being an inhouse DBA. If I could choose just one word, it would be “passion”. The inhouse DBA should be passionate about the database – almost like a parent. Some may stretch it to be possessiveness – not desirable; but if balanced with the passion for nurturing and caring, that is exactly what we look for in an inhouse DBA. That is *not* something I look for when I hire a consultant DBA. It’s knowledge first and foremost.
Anyway, I just wanted to present the perspective from the other side of the fence. While the objectives may see the same, there are vast differences in the details. At the end of the day, soft interpersonal skills are important to any job – inhouse DBA, consultant, CEO or janitor. But it’s understanding the factors that affect that soft skill is what differentiates leaders from laggards.
With best regards,
Arup Nanda
Longtime inhouse DBA and proud of it.