Cutting production defects by 80% with automated release gates
Building the test automation and release pipeline that let Raven’s mobile teams ship weekly with far fewer bugs reaching customers.
- Role
- Engineering Manager
- Company
- RavenPay
- Stage
- Shipped
- Team
- 2 engineers on the automation, working with the engineers on each product
- Platform
- iOS, Android, Android POS
- Pillars proven
- Quality and delivery automation · Mobile architecture · Cross-team leadership
TL;DR
- Problem
- Manual regression testing took 3 days per release, so we shipped every 2 weeks, and defects still reached production in money flows.
- What I did
- Designed the test strategy and CI/CD pipeline, introduced automated release gates and staged rollouts, and changed how the team defines “done”.
- Result
- Production defects fell 80% per release, regression time dropped from 3 days to 40 minutes, and releases went from bi-weekly to weekly.
Context
Raven’s apps move real money, so every bug is expensive: a failed payment, a support ticket, sometimes a reversal. When I took on mobile engineering, releases were manually tested by each product’s engineers and shipped to stores.
The team was growing faster than manual processes could keep up.
The problem
For users
- Bugs in money flows, hotfixes that arrived days later, and an app rating of 3.9.
For the team
- Releases were stressful, testing was a bottleneck at the end of every sprint, and engineers waited days for feedback on their changes.
For Raven
- Slow releases delayed every product bet, and each production defect added support and reconciliation costs.
The insight that changed the direction: The problem wasn’t effort. Quality was being checked at the end instead of built in, and the riskiest paths had no automated protection.
My role
I owned the quality and delivery strategy for mobile and worked with the engineers on each product to put it in place.
- 1
Measured the baseline
Defects per release, escape rate, regression time and lead time from merge to store.
- 2
Designed the test pyramid
End-to-end tests on money-critical journeys (login, send, receive, bills, cards).
- 3
Built the CI/CD pipeline
With GitHub Actions and Codemagic: checks on every pull request, nightly end-to-end runs, automated store builds and staged rollouts.
- 4
Introduced release gates
No release unless critical-path tests pass and crash-free sessions stay at or above 99.5% during the staged rollout.
- 5
Changed team habits
Tests became part of the definition of done.
Key decisions and trade-offs
| Decision | Why | What we gave up |
|---|---|---|
| Smoke tests on every pull request; full end-to-end on real devices nightly | Fast feedback without slowing every merge | Some bugs found the next morning, not instantly |
| Staged rollouts with automatic halt on crash spikes | Catches what tests miss, before most users see it | Full rollout takes 5 days |
| Feature flags for unfinished work | Lets us release weekly regardless of feature size | Flag clean-up becomes ongoing work |
| Make each product’s engineers own their tests, with no separate QA handoff | Manual testing can’t scale with the team | About 6 weeks of slower feature delivery while engineers picked up the test tooling |
How it was built
The pipeline runs in 4 stages, each a gate:
- 1Pull request. Lint, type checks, unit tests and a smoke end-to-end suite, in under 20 minutes.
- 2Merge and build. A successful merge is built for the staging or development environment and shipped internally for testing.
- 3Release candidate. Automated store builds, release notes and the release checklist.
- 4Staged rollout. 1% → 10% → 50% → 100% with crash and error monitoring in Crashlytics and Sentry, and automatic halt.
-
Pull request
Under 20 min
Lint, type checks, unit tests, smoke end-to-end suite.
All green to merge
-
Merge and build
Every merge
Built for staging or development and shipped internally for testing.
Internal build passes
-
Release candidate
Weekly
Automated store builds, release notes, release checklist.
Critical-path tests pass
-
Staged rollout
About 5 days
1% → 10% → 50% → 100%, monitored in Crashlytics and Sentry.
Crash-free ≥ 99.5%
- 1%
- 10%
- 50%
- 100%
Gate between every step: crash-free sessions ≥ 99.5% in Crashlytics and Sentry, or automatic halt.
Production defects per release, indexed
Results
-
−80%
Production defects per release
-
3 days → 40 min
Regression testing time
-
Bi-weekly → weekly
Release cadence
-
98.7% → 99.7%
Crash-free sessions
Blue: my team’s work. Black: feature- or product-level, shared with others.
Reflection
What I’d do differently
Set a flaky-test policy from day one. Unreliable tests cost us trust in the pipeline for the first two months.
What it taught me
Delivery speed and quality aren’t a trade-off; automation gives you both. Users never see the pipeline, but they feel it as an app they can trust with their money.