Edit - We’re back!
We’ve had an issue with our databases. One of our fast database servers ran out of space, and then the second fast server ran out of space whilst replicating to the first.
As a result, we have fallen over to our backup database server, which runs on spinny disks rather than SSDs. Spinny disks means that it’s got plenty of space to spare, but it’s not fast. The backup DB server is currently replicating to our two main servers to get things back up and running again, but whilst that’s happening, all of our services are running slow.
The good news is, we’ll be back up and running as if nothing happened because our backup server saved the day. The bad news is, it may take another 24 hours or so, because the backup server is reliable but not fast!
Appreciate the update! I think it would be helpful to have a designated space outside of blahaj.zone where the admins can provide info in these types of scenarios. I was getting a little anxious with the instance down all day and seemingly no way for the admins to communicate what was going on.
Normally, if it were a complete outage, we’d simply have stuck an outage notice up on the domains. In this case, it wasn’t an outage initially, but just a really laggy database, so we left the services up. Once we realised that wasn’t going to work, and we had to pull services down to fix the DB issue, we put up announcement pages on most of the services