There is a question that has quietly changed in enterprise procurement, and most SaaS vendors have not noticed. Five years ago, the RFP asked: "Do you have a disaster recovery plan?" You attached a PDF, ticked the box, and moved on. Today the mature buyers ask three different questions: When did you last test it? What RPO and RTO did you actually measure - not target, measure? And can we see the retrospective from that test?
A document answers the first question. Only a rehearsal answers the other two. I learned that in the manual era of disaster recovery, and everything the cloud has automated since has made the lesson more true, not less.
What Building DR by Hand Taught Me
Years back, I designed the disaster recovery for a cargo management system at one of the leading airlines in the US. The airline was not yet comfortable putting production in the cloud - a common posture then. So I proposed something deliberate: keep the primary where their confidence lived, and build the DR site on AWS. The strategic pitch was simple. Let disaster recovery be your first, low-risk step into cloud; when the confidence comes, the primary can follow. We will return to how that bet aged.
What I remember most is not the architecture. It is the months of process craft that surrounded it. I drafted the DR process library by hand - a master DR process, an activation process, a governance process, a release process for changes made during a DR event, the test process, the post-DR report format. Each went through peer review, CTO review, optimization, correction. Then came the human construction: DR teams and an emergency response team, built role by role, with documented ownership and deliberately redundant responsibilities - if the designated person was unreachable, a named alternate carried the duty. Numbered processes. Work instructions that answered "who does what, and when" precisely, because the people executing them had never lived through a disaster. We knew every brick and every layer, because we had laid them ourselves. That was the strength of the manual era.
The drill itself was fully manual and fully rehearsed. We simulated the disaster non-disruptively by disabling the network connectivity between primary and DR. The NOC detected and reported the outage. The helpdesk notified the customer's point of contact - who knew it was a drill. The disaster was formally declared. The DR bridge was activated. The teams executed the activation processes step by numbered step, and the application came up in the DR site on AWS. We verified the database against the committed RPO, ran application sanity tests, and closed with the post-DR report. The customer certified the test. And then I did what any consultant should do with something that works: I replicated that process framework across many other customers.
The US and European buyers, famously stringent about process, did not just accept those DR artefacts in pre-sales meetings. The evidence was what built their confidence. Operational maturity was closing deals for me before I had words for it.
— Sethunath U N, Chief Consultant & Advisor, Mavis DxThe Cloud Automated the Failover. Did It Automate the Test?
The epilogue to that airline story matters. Years after I moved on, I learned the airline had gone fully live in the cloud. The bridge I had pitched - DR first, primary when ready - carried them all the way across. And in the same period, the hyperscalers industrialised what we once built by hand. AWS Elastic Disaster Recovery continuously replicates at block level and launches drill instances without touching production. OCI Full Stack DR orchestrates the entire failover - compute, database, load balancers, DNS - as a prevalidated plan executed with one action. GCP Backup and DR brings the same discipline to backup-based recovery. Failover has become, in large part, push-button.
So the old test - the weekend war effort to prove the standby site boots - is largely obsolete. The platform proves that continuously. If your DR testing methodology still looks like mine did in that airline engagement, you are spending your rehearsal budget validating what the cloud provider already guarantees, and validating nothing that they do not.
Because here is the uncomfortable symmetry: automation moved the risk, it did not remove it. The platform proves your virtual machines launch in the recovery region. It does not prove that your payment gateway certificate is valid there. It does not prove your partner's firewall allowlist includes the new IP range, that your batch jobs resume in the right sequence, that your third-party integrations - the GDS link, the pharmacy interface, the channel manager - reconnect and reconcile. It does not prove your data is consistent, only that it is present. Every one of those is where real recoveries actually fail, and not one of them appears on the platform's green dashboard.
What the Methodology Must Become
The shift, in one line: from testing whether systems come back to rehearsing whether the business continues. That changes the cadence, the scope, and the evidence a rehearsal produces.
Because drill instances no longer disrupt production, there is no excuse for testing once a year. Quarterly non-disruptive drills validate the automation itself - replication lag, launch templates, orchestration plans - and catch configuration drift while it is still small. The annual event survives, but its job changes: a full failover with the business on the recovered stack, running real transactions, measuring real numbers.
A measured RPO and RTO with a date attached - the exact artefact enterprise procurement now asks for.
The rehearsal now interrogates everything the platform cannot see: end-to-end business transactions on the recovered environment, data consistency checks rather than data presence checks, third-party integration reconnection, DNS cutover and certificate validity, and the sequencing of dependent services. In regulated industries, add evidence capture - a pharma buyer will want the rehearsal itself documented to GxP standards.
A dependency map that reflects reality, refreshed every rehearsal - because architecture drifts faster than documents.
AI now belongs in the test twice. As a tool: agents can execute runbooks, watch telemetry during failover, and flag anomalies faster than a human bridge call ever did - rehearse with them, so you learn where they can be trusted. And as a subject: if your product now carries AI features, your model endpoints, vector stores, and third-party LLM API dependencies are production services. Almost nobody has them in DR scope yet. Your buyers will ask before your architects do.
A DR scope that includes the AI tier - and a documented answer for what happens when the model provider, not your cloud, is the one that is down.
So Are the Humans Still Essential? A Flood Answered That
It is tempting to conclude that if OCI can fail over a full stack with one click, the operations veteran is a cost line waiting to be optimised. An incident from my home state settled that question for me, and I want to be careful here: I had no role in this recovery. I only learned of it as it unfolded, and studied what surfaced afterwards.
During the 2018 Kerala floods, a major bank's primary data centre in Kerala went under water. On paper, they were prepared - a DR data centre stood ready in Bangalore, and the switchover was invoked. Yet the recovery took weeks. Not because the technology failed. Because the physical failover did not align with process knowledge, coordination, and application recovery. The site came up; the business did not follow it at anywhere near the same speed. Everything my airline engagement had spent months building by hand - the numbered ownership, the rehearsed sequences, the trained alternates - was exactly what was missing when the water actually rose.
That is the perspective I hold on the human question, and automation has sharpened it rather than softened it. Three things still belong to people. The declaration: no cloud service decides when a degradation becomes a disaster. In our airline drill, declaration was a formal, rehearsed step with a named owner - because the decision to fail over carries real cost in either direction, and someone accountable must make it under pressure. The coordination: a recovery is not a technical sequence, it is an organisational one - NOC, helpdesk, customer communication, release governance during the event - and the Kerala flood showed what weeks of misalignment look like when that muscle was never exercised. The paradox of reliability: the better your automated DR works, the less practiced your people become at recovery - until the day the automation itself is what failed. Aviation learned this with autopilot long ago; pilots still train in simulators for precisely that day. Your rehearsal is no longer only a systems test. It is how your people stay match-fit for the night the button does not work.
What Changes in the Buyer Conversation
- A DR plan PDF, last edited when the architecture was two versions younger
- Target RPO/RTO quoted from the contract, never measured
- Annual infrastructure test that proves servers boot
- DR knowledge concentrated in two veterans nobody can spare
- Quarterly automated drills plus an annual full business failover
- Measured RPO/RTO with dates, trends, and a retrospective on file
- Rehearsed declaration, coordination and communication - not just recovery
- AI tier in scope; agent-assisted runbooks rehearsed alongside the humans
Notice what the right-hand column really is: a sales asset. I saw it in the manual era, when process evidence won over the most demanding US and European buyers, and it is truer now. When procurement asks the three questions, the vendor who slides across a dated retrospective with measured numbers has just ended the conversation - and probably the competition. Operational maturity does not merely prevent losses. In enterprise deals, it closes them.
The Rehearsal Is the Point
In Kathakali, the performance is only the visible edge of the rehearsal - years of training stand behind a single night on stage. Disaster recovery has become the same discipline. The cloud has given every ISV a magnificent stage: replication, orchestration, one-click failover. What it cannot give you is a company that knows its part when the lights go out. A flooded data centre in Kerala proved the stage alone is not enough. A certified drill for a US airline proved what rehearsal makes possible.
So take your DR plan out tonight and ask it the buyer's three questions. When was it last rehearsed? What did you measure? What did you learn? If the answers are a shrug, a target, and a silence - you do not have a DR capability. You have a document.
When the night comes, the platform will play its part flawlessly. Will your people know theirs?
Could your DR evidence survive procurement's three questions?
Our Operational Maturity advisory builds the rehearsal programme end to end - drill cadence on AWS, OCI or GCP, declaration and governance processes, Day-0/1/2 runbooks, and the measured RPO/RTO evidence pack that turns your resilience into a deal-closing asset. Thirty minutes will tell you how far your current test methodology is from what buyers now expect.
Outcome-based · No-commit · Monthly engagement
You continue only when you see real value. That is our promise.
Start the Conversation →