All systems are operational again, and we're considering this issue closed.
Posted 12 months ago. Oct 21, 2016 - 20:59 CEST
We're no longer seeing errors. We will continue to monitor the situation.
Posted 12 months ago. Oct 21, 2016 - 20:21 CEST
Error levels are steadily dropping as our DNS is getting back to normal. We're still monitoring the situation.
Posted 12 months ago. Oct 21, 2016 - 20:08 CEST
We've located the root cause, and we're sad to report that in the short term there is nothing we can do to mitigate this issue. The issue arises because of the currently on-going DYN DDOS attack (http://gizmodo.com/this-is-probably-why-half-the-internet-shut-down-today-1788062835). This causes DNS issues with our internal queue and database systems, which in turn causes communication issues between our publicly facing servers and our backend servers.
We just saw a short event of about 20 minutes of elevated errors in our system. Everything should be back to normal now. We're monitoring for further regressions though, and investigating the root cause of this issue.