Case 01
DevOps & InfrastructureRecovering $46.9k in Silently Lost Orders
US e-commerce manufacturer with a custom 3D product configurator
- $46.9k
- in orders recovered across 80 affected orders
- 12k
- builds migrated with zero further data loss
- 50%
- admin memory footprint cut (2GB → 1GB)
The Challenge
Builds from the 3D configurator silently stopped saving for two days, and a separate bug was duplicating order numbers — both invisible until customers started complaining.
Approach
- Traced the build issue to a Redis object-cache layer silently dropping writes meant for MySQL, then migrated ~12,000 builds into a dedicated table
- Replaced the non-atomic order-number generator with an atomic counter to stop duplicate collisions
- Rebuilt CI/CD on Jenkins (57 plugins, Telegram alerts, a disaster-recovery plan) and stood up a 5-VM clustered infrastructure behind a load balancer
- Put the origin server behind Cloudflare with WAF and geo-blocking, and disabled password-based SSH after ~31k brute-force attempts
- Moved a Nuxt + FastAPI app to Dokploy with zero downtime