Executive Summary
Fast Thinking runs a multi-tenant business intelligence and operational reporting platform for more than 50 enterprise customers across Europe, supporting around 500 daily active users. The service combines customer analytics data, uploaded content, APIs and scheduled processing in AWS, and needed a recovery model that protected that data, kept the business within its contractual downtime threshold and could be demonstrated in practice, at a cost proportionate to the value of the service it protected.
Epitechnic designed a cross-account Backup and Restore capability built around independently protected data, a dormant recovery environment rebuilt from infrastructure as code, and a quarterly testing programme that gives the recovery target a measured, repeatable result.
The service can now be restored in two to five hours against a 24-hour commitment, verified across four recorded recovery exercises with a 100% pass rate, at a 92% lower monthly cost than the next fastest option evaluated.
Business Challenge
Fast Thinking's production estate runs in AWS Europe (Ireland) and includes an Amazon Aurora PostgreSQL database of approximately 500GB, more than 600GB of Amazon S3 content and user uploads, 28 AWS Lambda functions, Amazon EC2 Auto Scaling Groups and Amazon API Gateway, separated across a five-account AWS organisation covering management, production, disaster recovery, security and log-archive responsibilities.
The service is classified as Business Important, with an agreed 24-hour target to restore the full platform, a 24-hour Recovery Point Objective for Aurora data, a 15-minute Recovery Point Objective for user uploads and static content, and a 48-hour Maximum Tolerable Downtime, beyond which contractual exposure and customer-retention risk increase materially. Fast Thinking's daily reporting cycle meant a daily Aurora recovery point was acceptable, since reporting data can be recovered in the following cycle, but customer uploads needed near-real-time protection given their immediate value to customers. The commercial constraint was just as real as the technical one: the service did not justify the cost of continuously operating a second production platform.
Success Criteria
- A recovery capability that meets the agreed 24-hour restoration target with genuine headroom
- Near-real-time protection for customer uploads, alongside a daily recovery point for reporting data
- A recovery cost proportionate to the value and economics of the service it protects
- A documented, repeatable recovery runbook that avoids decisions made under pressure during an incident
- Recovery performance proven through regular, repeatable testing
Epitechnic Approach
Epitechnic evaluated four recovery patterns, active-active, Warm Standby, Pilot Light and cross-account Backup and Restore, against Fast Thinking's agreed recovery targets and the cost profile of the SaaS service. Multi-region active-active gave recovery capability far beyond the 24-hour target, disproportionate to the service's economics. Warm Standby would reduce recovery time further but required roughly twelve times the monthly spend of the option ultimately selected. Pilot Light offered limited additional value against the documented 24-hour target.
Cross-account Backup and Restore met the target with actual recovery performance of two to five hours, at the lowest monthly cost of the four options. Multi-AZ Aurora availability already protects the service against routine database failure; the cross-account recovery capability is reserved for a more serious production event.
Solution
Protecting the data before rebuilding the platform
Fast Thinking needed a recovery source that stayed intact even if the production account itself was compromised or misconfigured. Epitechnic used AWS Backup to create automated Aurora snapshots and copy them into a dedicated disaster-recovery account, with the backup vault locked so a copied recovery point cannot be removed through operator error or a production-account compromise. Aurora recovery points are created daily at 02:00 UTC, retained for 30 days and moved to cold storage after seven days. Amazon S3 versioning and cross-region replication continuously protect content and deployment artefacts, with delete markers replicated and encryption maintained throughout.
This gives the recovery team a known, separated source for both data and application packages, independent of whatever state the production account is in when an incident happens.
A recovery environment that stays dormant until needed
Running a duplicate production estate continuously would have cost far more than the recovery target justified. Epitechnic designed the recovery environment to stay dormant until it is actually required, with the service recreated from version-controlled CloudFormation stacks in the disaster-recovery account: network components (VPCs, subnets, NAT gateways and security groups) in five to seven minutes, Aurora restored from the latest cross-account snapshot in two to two and a half hours, the 28 Lambda functions deployed from replicated S3 packages in ten to fifteen minutes, the API layer recreated in about five minutes, and monitoring enabled in about three minutes. Amazon Route 53 health-check failover completes the transition once the rebuilt service is ready.
The core platform can be rebuilt in approximately two and a half to three hours, with full validation, including application checks and DNS cutover, taking four to five hours. Documenting this sequence as a runbook keeps recovery decisions made in advance, before an incident begins.
Proving recovery performance through regular testing
A recovery design is only as good as its last test. Fast Thinking and Epitechnic run quarterly recovery exercises, alternating full end-to-end tests with focused snapshot-restore drills, so the team can improve the steps that consume the most time without waiting for a real outage. Four recorded exercises, an initial full test, two snapshot drills and an annual full test, all met the 24-hour target, and the most recent annual test completed 17 minutes faster than the first. Across the four exercises, measured recovery times ranged from 5.3 to 12.2 times faster than the contractual target.
Over an 18-month period, 183 backup jobs ran with a 99.5% success rate, and more than 1.2 million S3 objects replicated with a 99.98% success rate, giving the recovery design a track record beyond the quarterly test cycle.
Outcomes
Recovery performance: The service can be restored in two to five hours against a 24-hour commitment, preserving substantial recovery headroom.
Test record: Four recorded recovery exercises, spanning full end-to-end tests and snapshot drills, all met the 24-hour target, a 100% pass rate.
Cost efficiency: The selected design costs 92% less per month than Warm Standby, the next fastest option evaluated.
Operational reliability: 183 backup jobs completed at a 99.5% success rate and over 1.2 million S3 objects replicated at a 99.98% success rate over 18 months.
Governance: A backup failure now reaches Epitechnic's on-call engineer within 15 minutes through CloudWatch and SNS, and the Fast Thinking CTO approves changes to recovery targets and takes part in the annual full test.
Why It Worked
The design succeeded because Epitechnic matched the recovery pattern to what Fast Thinking's business actually needed. A daily reporting cycle meant a daily Aurora recovery point was genuinely sufficient, freeing the budget to protect customer uploads at the much tighter 15-minute standard they actually required.
Testing is what turned the design from an architecture diagram into a proven capability. Running the same exercise every quarter, and holding the annual full test to the same standard as day one, gave Fast Thinking a measured, improving recovery time.
Key Takeaways
Challenge: A multi-tenant SaaS platform needed a recovery capability that met a 24-hour restoration target and protected customer data, without the cost of a continuously running duplicate production environment.
Approach: Epitechnic evaluated four recovery patterns against Fast Thinking's targets and cost profile, then built a cross-account Backup and Restore capability with independently protected data, an infrastructure-as-code rebuild process and a quarterly testing programme.
Outcomes: Recovery in two to five hours against a 24-hour target, a 100% pass rate across four recorded exercises, and a 92% lower monthly cost than the next fastest option.
Lessons: Matching the recovery point objective to how the data is actually used, daily for reporting, near-real-time for uploads, is what let a proportionate design meet a strict recovery target without disproportionate cost.