Optus has provided some further detail on the cause of this outage which took approx 10m Internet and mobile services offline for several hours last week.

Still not all of the technical details I would like to see but it was confirmed that the issue was a routing fault that occurred after maintenance. Possibly a full BGP table was received somewhere that it wasn’t expected (maybe due to filter config lost during the software update) triggering BGP max prefix protections. It also sounds like Optus don’t have a good out of band network to successfully recover services remotely so they required techs to visit sites.

From the news.com.au story:

“At around 4.05am Wednesday morning, the Optus network received changes to routing information from an international peering network following a routine software upgrade,”

“The routing information changes propagated through multiple layers in our network and exceeded preset safety levels on key routers which could not handle these.

The statement said the action resulted in routers disconnecting from the Optus IP Core network to protect themselves.

This resulted in a large scale effort to reconnect or reboot the routers physically, requiring “the dispatch of people across a number of sites in Australia”.