Diagnosing a Production-Scale Performance Failure

A real Starting Point turned a working feature into an HTTP 500.

Production contained approximately 1,004 Bank Activity rows. A real Starting Point activated the full historical read path, and subsequent Bank Activity reads returned HTTP 500.

The failure was tied to the conditions under which the feature ran. A prior proof using 300 rows had stayed below timeout; that result had not established behavior at the production scale.

Prior proof stayed below timeout

300 rows

Production-shaped reproduction

1,004 rows

Restore performance without weakening the rules.

• Preserve existing semantics and fail-closed behavior.
• Reproduce the production conditions with synthetic data.
• Do not mutate production or staging data during the hotfix investigation.
• Validate the change with broad regression coverage.

Test the scale hypothesis with the read path fully activated.

1. 01
   Incident

   Bank Activity reads return HTTP 500.

   ↓
2. 02
   Context

   A real Starting Point activates full historical reads.

   ↓
3. 03
   Hypothesis

   Work below the timeout at 300 rows may exceed it at production scale.

   ↓
4. 04
   Reproduce

   Use a production-shaped, 1,004-row synthetic dataset.

   ↓
5. 05
   Isolate

   Measure the database helper involved in row-level validation.

   ↓
6. 06
   Redesign

   Optimize the helper while preserving its semantics.

   ↓
7. 07
   Verify

   Measure the improvement and run broad regression coverage.

Repeated validation made helper cost a read-path problem.

1. 01
   Real Starting Point

   Activates the full historical read path

   ↓
2. 02
   Bank Activity history

   Approximately 1,004 production rows

   ↓
3. 03
   Row-level validation

   Calls a database helper

   ↓
4. 04
   Timezone catalog scan

   Expensive work inside that helper

The root cause was a database helper that scanned PostgreSQL timezone catalog data during row-level validation. The production-shaped reproduction exposed a scaling failure that the earlier 300-row proof had not crossed.

Change the cost of validation while keeping its meaning.

The fix focused on the measured helper bottleneck. It reduced the cost without relaxing the existing rules or changing the intended behavior. The improvement was evaluated using the production-shaped synthetic reproduction.

Performance and correctness were both part of the result.

• A 1,004-row synthetic reproduction exercised the scaling conditions.
• The deployed and optimized helpers were measured: 11,136.6 ms and 41.3 ms, respectively.
• Validation included broad regression coverage.
• Existing semantics and fail-closed behavior were preserved.
• No production or staging data was mutated during the hotfix investigation.

A measured improvement, grounded in the conditions that caused the failure.

Measured helper time fell from 11,136.6 ms to 41.3 ms—approximately a 99.6% reduction, or 270× faster. These figures describe the helper measurement, not an end-to-end application latency benchmark.

The investigation connected an incident, a context trigger, a scale hypothesis, a faithful reproduction, and a measured bottleneck. That chain of evidence made a focused fix possible without sacrificing financial application behavior.

• Root-cause analysis
• Production-shaped reproduction
• SQL
• Performance analysis
• Regression testing
• Change validation