Pythian’s Oracle Apps DBA team recently upgraded a client’s E-Business Suite system to version 12.1.3, bringing them into compliance with Oracle’s baseline support requirements for Release 12.1 nearly one year ahead of deadline. We’d like to tell you a bit about this project — not to toot our own horn (though that’s nice too, we are kinda proud), but because it provides an ideal illustration of the power of the Pythian service delivery model, particularly as it applies to large enterprise-class projects.
By the numbers
The numbers don’t tell the whole story, of course, but they set the stage. This massive patching exercise entailed:
- 377 patches applied to the apps and database servers
- over 110 client requests resolved as a result of the upgrade
- 6 months of planning and testing
- 3 end-to-end test upgrade runs
- more than 20 pages of step-by-step instructions tailored to our client’s environment in an internally-generated, collaboratively composed upgrade guide
- a team of 7 DBAs involved in patch analysis, planning, and testing
- 25 hours, 30 minutes of downtime for all production patching activity
- 10 hours of post-upgrade online activities
- 6 team members executing the production upgrade, with the 7th on standby for emergencies
- 0 actual emergencies
- 0 new issues reported by client during post-upgrade, pre-release testing
- 0 show-stopping issues reported by business users in the first business days after the upgrade
The usual way
If you’ve worked with Oracle Applications for any length of time, then you know how these kinds of projects can go. I’ve devoted my share of long weekends to E-Business Suite upgrades, and I’d like to share a few quotes from discussions with my teammates that pointedly illustrate the Apps DBA perspective:
It is not rare the go-live process of these projects takes 1-2 days of downtime, considering all the various components. I was involved in 5-6 upgrades of at least this scale before, and each time it was a nightmare for the DBAs working on it, in some cases I’ve even worked up to 40 hours in 2 days. It’s extremely hard to stay concentrated and also probability of making mistakes, because of tiredness, becomes very high. I know 40 hours in 2 days is the absolute limit for myself, after that I just “drop dead” for ~12-14 hours and it takes about a week to get back on track (the very important 1st week after the upgrade).
I remember my last upgrade project… 40 hours non-stop with some 1-2 h nap breaks, tons of red bull, and a shocked wife’s face seeing a zombie arrived back home instead of a husband.
So, there you have it. Start your upgrade on Friday night, and pray you don’t have any major issues on Monday, while your DBAs are all shuffling around mumbling, “Braaaaaiiinnnnsss…” And it’s not just the DBAs; chances are, the people most heavily involved in testing and vetting the upgrade are your “power users”, the very people you need to be fresh and ready to assist their fellow employees during the early days after an upgrade.
The Pythian way
Sounds fun, right? So much fun, in fact, that you might be delaying your own upgrade in search of “just the right time” to execute the project to minimize impact to your
planned holiday schedule business. To contrast, consider this comment from one of the same teammates quoted above, after our recent upgrade weekend:
[client]’s R12.1.3 upgrade was the 1st large upgrade for me here at Pythian and the experience was completely different, the ability to pass work around world is a huge benefit, I worked ~8 hours during the go-live, was not tired, could concentrate and did not make any mistakes.
Briefly (omitting the months of testing and planning — you’re welcome), here’s how this particular production upgrade unfolded:
- Pythian resources in North America started pre-upgrade backups and the first round of patches shortly after client’s close of business on Friday.
- Once long-running patches were rolling, we handed off to Pythian resources in India for monitoring and execution of the next steps.
- Team members in Europe continued with the implementation plan during their daytime hours, sharing tasks with team members in North America as waking hours began to overlap.
- Client stakeholders were kept apprised of status as milestones were reached, via direct communication with their Pythian team lead.
- The upgrade completed error-free on Saturday evening, early enough that our client felt comfortable waiving a scheduled Sunday morning status call, leaving the system open for testing by business users.
Net result: An on-schedule, error-free upgrade, a happy client, and well-rested DBAs ready to tackle new challenges the following week.
The power of the Pythian model
As I mentioned earlier, we’re proud of what we accomplished with this upgrade, but the real story here is the framework that enabled our success. Pythian’s service delivery model is ideal for these sorts of complex upgrades and migrations. We are built to provide:
- World-class talent: We have highly-skilled DBAs with a wealth of experience on a variety of platforms and architectures, who can quickly grasp the nuances of your systems and get to work on an approach that fits your needs.
- World-wide talent: Every step of the way, the person working on your upgrade is starting with a “full tank of gas,” and can hand off to fresh resources as required.
- Careful, point-by-point planning, informed by years of collective experience with similar projects, and driven by the need to have any team member be able to take up any part of the plan during implementation.
- Tight communications with the client throughout planning and execution. We are not a simple “work comes over the wall, and we toss it back completed” organization. No matter how good we are, no project of this magnitude can succeed without clear communication with the people who know the business processes, functional requirements, and external technical dependencies of the systems that we’re upgrading.
If you have an Oracle Applications upgrade project looming on the horizon and you’d like access to top-flight, globe-spanning talent to help you execute that upgrade, please reach out to Pythian, and we’ll see how we can help you out.
(Oh yeah, and Apps DBAs: If you’re interested in working on epic projects and still feeling like a human being on Monday morning: We’re hiring. C’mon over!)