Intermittent Network Issue

Incident Report for Fonolo

Postmortem

On Friday, September 24, Fonolo experienced intermittent connectivity issues affecting some PSTN and Direct SIP customers between 10:30 and 13:45 EDT. This event may have been experienced as agents not being able to press DTMF1 to connect to callers or agents experiencing static or audio artefacts during Fonolo call-backs.

These issues were a result of a fibre cut on an intermediate third-party network beyond our redundant upstream carrier infrastructure, which manifested as increased latency on certain public IP routes being advertised by various internet carriers. Fonolo worked quickly to engage our carrier on the issue and have since confirmed that all affected services on the Fonolo platform have resumed without issue on Friday by 13:45 EDT. During the event, the carrier was able to re-route traffic around the nodes affected by the fibre cut. Fonolo received further confirmation on Saturday, September 25, that the fibre repairs had been successfully carried out as of 12:40 EDT, restoring full connectivity to their network.

Posted Sep 27, 2021 - 16:55 EDT

Resolved

Fonolo's monitoring have shown stable services across Voice Call-Backs, the Fonolo Portal, and SIP Services since 1:45 PM EST. Fonolo will continue to monitor and awaits additional information from the carrier, which will be shared in a post-mortem.

Posted Sep 24, 2021 - 17:26 EDT

Monitoring

Fonolo has received an update from our upstream carrier that services have stabilized, while work continues on the third-party fibre break.

Posted Sep 24, 2021 - 17:24 EDT

Update

Fonolo received confirmation from our upstream carrier that there was a third-party fibre cut and their technicians are currently working on repairs.

Posted Sep 24, 2021 - 15:30 EDT

Identified

Fonolo is currently experiencing intermittent network issues caused by our upstream carrier that is affecting some routes to customers. Fonolo is investigating and has initiated communication with our carrier. Customers utilizing PSTN and Direct SIP may experience increased latency that can cause static on calls or prevent agents from pressing 1 to connect. The Fonolo Portal may also experience slower loading times.

Posted Sep 24, 2021 - 13:50 EDT

This incident affected: Voice Call-Backs, Fonolo Portal, and SIP Services.