Fire impacting DNS SOA on nearly all sites

Due to a Fire in the data center of our primary Domain Name System host, we are currently havinng issues with outbound mail with error messages like Temporary server error [451 4.1.8 Domain of sender address you@yourdomain.com.au does not resolve]

Unfortunately there is nothing we can do other than escilating the issue to their technical desk, they have assured us that they are making every attempt to be back online online.

Update

Power is being restored to the data center in stages, power is back on to airconditioning and some of the infrastructure, networking kit is back up. More updates as they become available

Additional Update

Power has been restored to most of the data center, although we are still having the same issues with DNS and mail, we are looking into a work arround at this point although nothing new has come to hand. Onsite admins estimate about 6 hours as a tentitive date of repair

Tentative Resolution

The main issue with email seems to be solved (we are using alternate resolvers provided by the DR team) and mail appears to be flowing as normal. There is however a 4 hour scheduled outage for either June 7 or 14 (as the current solution is a temporary one and there will be a cut over when the change is made back). More as it comes to hand. Read more in this GawkWire report.

Relapse

Several customers have reported the symptoms returning, the data center have reported that there were issues with a faulty breaker on the backup generators currently powering thie network infrastructure. There is no ETA on repairs at this point. ThePlanet have proven to be a reliable provider thus far (with only 2 outages in the past 3 years), we will be re-evaluating our DR plans and considering other options in an attempt to reduce the impact of service outages in the future.

Resolution

We now believe this issue to be resolved, with normal service resuming with the exception of a possible outage when the temporary measures are replaced