React & Respond (Paradox Pair #68)
Hysteresis alerting, where we delay alerting until a threshold is consistently passed, normalizes our reactions, allowing us to respond in a more constructive manner.
The collective hard work and creativity that takes a product from inception to launch is a cause for celebration. Rarely, however, is our work done, it just enters a new phase. While we've tested and stressed our development systems it's only when a product meets the market that we finally have the full observable picture.
Prior to going live we've made our educated choices on what might go wrong and how we would respond to correct errors. We assembled a communication path and we've held our go/no-go reviews. We have instrumented our applications, collecting metrics that will give us near-time insights. We know our scaling factors and we mapped these against our expected usage. Yet, once we're live, we still find we've missed something: a bug, a forgotten requirement, a component failure.
"We know that through the combination of planning and training we weight randomness towards our favor. We shape our outcomes by being prepared and practicing our craft — being mindful and engaged."
- Intentional & Lucky (Paradox Pair #13)
One area that always needs adjustment post-launch are our alerting thresholds. We expect false positive alerts, opting to be hyper aware of anomalies in the early days so we can validate our assumptions and react quickly to unexpected events. As we get comfortable with the day-to-day performance we begin to lower our acceptance of false alerts and tune our thresholds accordingly. We introduce a lag to an alert, now requiring a consistent value above our threshold over a fixed period of time. Hysteresis alerting normalizes our reactions, trading a small amount of time in our response against the lost efforts of a false alarm.
New parents know this approach well, quickly recognizing that all noises from their new born are not the same, learning to pause and wait for a repeated cry before deciding to take action. We can apply this logic to our approach in other areas, especially those where we are prone to over react to an anomaly rather than a systemic failure: a drop in sales bookings, employee attrition, or when reviewing someone's performance. What we desire is to respond rather than react, moving from tense and fast to constructive and intentional.
The Paradox Pairs series is an exploration of the contradictory forces that surround us. A deeper study finds that these forces often complement each other if we can learn to tap into the strength of each. See the entire series by using the Paradox Pairs Index.
