Resolved -
The outage with the third party vendor has been resolved.
Jun 12, 16:42 EDT
Monitoring -
The third-party services have been recovered. All Hypercare services are operational. We're currently monitoring the services.
Jun 12, 16:29 EDT
Update -
Automated code escalation has been manually restored. All code team functions are now fully operational.
Jun 12, 16:09 EDT
Update -
In addition to the previously reported Doppler outage, we have now identified a separate issue affecting virtual paging. This is due to a service disruption with Twilio, a third-party vendor. As a result, virtual paging is currently unavailable and may affect message delivery for some users.
You can view the Twilio status here: https://status.twilio.com
We continue to monitor both the Doppler and Twilio incidents and will provide updates as services are restored. Thank you for your continued patience.
Jun 12, 15:51 EDT
Identified -
The engineering team is manually updating the secrets to restore the services. Everbridge integration has been restored.
Jun 12, 15:44 EDT
Investigating -
We are currently experiencing a partial service disruption caused by an ongoing outage with Doppler, one of our third-party service providers. As a result, the following Hypercare features are impacted:
- Escalation notifications related to code team activations (note: code team activations remain fully operational)
- Schedule change email notifications
- Schedule and Equity report downloads
- Everbridge file generation
We are actively monitoring the situation and will provide updates as Doppler resolves the issue.
You can view the status of the Doppler incident here: https://www.dopplerstatus.com/incidents/8jtzf7nrl9tv
Jun 12, 15:26 EDT