Sometimes, it takes a storm to reveal the leaks. This incident was one such event that impacted our cloud-based access control solution, Salto KS. It was a high-impact incident that tested our systems, our communication, and our ability to respond under pressure. It wasn’t easy, but it became a turning point that has sparked meaningful change.
Today, we’re sharing how we faced it, what we learned, and how we’re making sure it doesn’t happen again.
The moment everything changed
When the incident hit, it wasn’t just a technical hiccup, it was a wake-up call. IR-60 was a high-severity incident we faced earlier this year that caused service disruptions across several customer-facing applications, including commissioning and integration portals. The issue led to delays in real-time updates and impacted user visibility into system status, making it harder for Salto KS customers to track progress during critical operations.
Our engineering teams jumped into action, diagnosing the issue and deploying fixes. Meanwhile, our communication channels were stretched thin, revealing gaps in how we keep our users and teams informed during critical moments.
What happened:
- Engineering launched a rapid response to contain and resolve the issue.
- Communication efforts were well-intentioned but lacked structure, predictability and consistency.
- Roles during the incident weren’t clearly defined, leading to some perplexity.
What we learned:
Every incident is a lesson in disguise. This incident taught us that even the best technology needs strong communication and clear processes to back it up.
Key takeaways:
- Visibility is everything: Users need timely, clear updates, especially when things go wrong.
- Templates and structure matter: Ad-hoc messages don’t cut it during a calamity.
- Defined roles are critical: Everyone should know who’s doing what, and when.
- Postmortems must lead to action: Reflection is good. Change is better.
Turning lessons into action
We didn’t just patch the problem, we’re rebuilding the way we work. Here’s how we’re making sure this incident becomes a step forward, not a repeat.
1. Smarter communication tools
- Status page: We’re turning it into a user-friendly, product-facing tool that serves both customers and integrators.
- In-app status banners: Real-time updates will be built into every customer-facing app, tailored to different user types.
- Dedicated channels: Clear, predictable updates for out techmunity and commercial teams via mailing lists, real-time tracking for internal teams through Slack, and refined messaging for external audiences.
2. Clearer incident management
- On-call process improvement: Defined roles for incident commanders and responders, with tailored approaches for day and night shifts.
- Incident flow mapping: Protocols for short, mid, and long incidents, so everyone knows what to expect.
3. Better postmortems
- Unified templates: One format for both technical and commercial audiences.
- Mandatory readership: High-severity incident reports are now required reading.
- Retrospective hosting: When needed, we’ll gather teams to reflect and improve.
4. Ongoing transparency
- Follow-up articles & blogs: We’ll continue sharing how we respond to incidents, what we learn, and how we grow.
Why this matters to you
We know that trust isn’t built by pretending everything is perfect. It’s built by showing up when things go wrong, being transparent, and proving that we’re learning, improving, and committed to doing better.
IR-60 was a challenge, but it’s also the reason we’re evolving. We’re making our systems stronger, our communication clearer, and our commitment to you even deeper.
We’re not just fixing things. We’re building something better.