Just to note as well for folks on #MastodonAppUK and #Universeodon

I am still looking into this issue with media uploads, I'm so far struggling to pin down a root cause as for most folks it's an intermittent error and I haven't actually found any errors in our logs yet other than what is returned to the apps / clients.

As soon as I can get to the bottom of it I'll get it sorted properly so media uploads work like normal again.

Just to note as well for folks on #MastodonAppUK and #Universeodon

I am still looking into this issue with media uploads, I'm so far struggling to pin down a root cause as for most folks it's an intermittent error and I haven't actually found any errors in our logs yet other than what is returned to the apps / clients.

As soon as I can get to the bottom of it I'll get it sorted properly so media uploads work like normal again.

Sorry for the blips there to #MastodonAppUK and #Universeodon - I managed to bork some of our HAProxy configuration in a bit of a stupid way.

MastodonAppUK is now running a single web server on the new cluster so I'm hoping that should make things feel a lot more responsive as I've removed some of the hops we were going through.

Tomorrow I'll get the other two temp web servers stood up, an additional for MastodonApp and an additional for Universeodon and get all the old servers on the old cluster decommissioned which should then mean all traffic is hitting our new owned dedicated hardware in the new environment!

Sorry for the blips there to #MastodonAppUK and #Universeodon - I managed to bork some of our HAProxy configuration in a bit of a stupid way.

MastodonAppUK is now running a single web server on the new cluster so I'm hoping that should make things feel a lot more responsive as I've removed some of the hops we were going through.

Tomorrow I'll get the other two temp web servers stood up, an additional for MastodonApp and an additional for Universeodon and get all the old servers on the old cluster decommissioned which should then mean all traffic is hitting our new owned dedicated hardware in the new environment!

Apologies folks on #MastodonAppUK and #Universeodon - It looks like we're seeing and have been seeing for the last week or two some issues with certain operating systems and clients trying to upload media to the sites.

We're currently (over the last 5 days) seeing around a 45% failure rate for iOS and 21% failure rate on MacOS devices.

It's not clear what could be causing this and it seems to be impacting multiple clients as far as I can see on my end.

I will continue to investigate and see if I can get to the bottom of the weirdness.

Apologies folks on #MastodonAppUK and #Universeodon - It looks like we're seeing and have been seeing for the last week or two some issues with certain operating systems and clients trying to upload media to the sites.

We're currently (over the last 5 days) seeing around a 45% failure rate for iOS and 21% failure rate on MacOS devices.

It's not clear what could be causing this and it seems to be impacting multiple clients as far as I can see on my end.

I will continue to investigate and see if I can get to the bottom of the weirdness.

Just to note that's maintenance full complete, there wasn't any downtime / interruptions as far as I or monitoring could detect.

I have added some more maintenance for #MastodonAppUK and #Universeodon now on https://status.atlas-media.co.uk to try to track some of the work that is currently prioritised for the instances.

I do also have a load of work for med-mastodon that I need to get around to doing.

Now however I've got a boat load of @atlasmediagroup work I need to crack on with.

Just to note that's maintenance full complete, there wasn't any downtime / interruptions as far as I or monitoring could detect.

I have added some more maintenance for #MastodonAppUK and #Universeodon now on https://status.atlas-media.co.uk to try to track some of the work that is currently prioritised for the instances.

I do also have a load of work for med-mastodon that I need to get around to doing.

Now however I've got a boat load of @atlasmediagroup work I need to crack on with.

MastodonAppUK and Universeodon Maintenance:

I'm going to be getting us upgraded to the latest patch release for Mastodon shortly, I don't expect this to cause any sort of noticeable downtime or impact as we can do the upgrades one server at a time.

#MastodonAppUK #Universeodon

I've made an adjustment to the #Universeodon infrastructure this afternoon to try to improve the traffic and the responsiveness of the site.

We seem to be having a lot of issues at the moment with connectivity between our new environment and the old environment.

I'm hoping this same change should reduce a lot of the traffic we're trying to route back and forth between the two and should hopefully make #MastodonAppUK a bit more responsive as well.

I am hoping to have some time on Sunday to make some further improvements ahead of the maintenance on Monday which should allow me to once and for all finish the migration in the near future!

I've made an adjustment to the #Universeodon infrastructure this afternoon to try to improve the traffic and the responsiveness of the site.

We seem to be having a lot of issues at the moment with connectivity between our new environment and the old environment.

I'm hoping this same change should reduce a lot of the traffic we're trying to route back and forth between the two and should hopefully make #MastodonAppUK a bit more responsive as well.

I am hoping to have some time on Sunday to make some further improvements ahead of the maintenance on Monday which should allow me to once and for all finish the migration in the near future!

We're going ahead and adding `miniwa.moe` to our list of suspended servers across #MastodonAppUK #Universeodon and #MedMastodon as there appears to be no active moderation taking place and reports of transphobic behaviour have been ignored by their moderators.

I'm not expecting this to impact the vast majority of folks across our instances but you might see a follower / following drop by 1 or 2.

#Fediblock

We're going ahead and adding `miniwa.moe` to our list of suspended servers across #MastodonAppUK #Universeodon and #MedMastodon as there appears to be no active moderation taking place and reports of transphobic behaviour have been ignored by their moderators.

I'm not expecting this to impact the vast majority of folks across our instances but you might see a follower / following drop by 1 or 2.

#Fediblock

I am aware of some stability issues on #Universeodon - I think this is a case of a badly tuned database backup that's causing things to lock up, part of this will be improved in the next week or so with some hardware upgrades to our servers but until then I'll do what I can to get things more stable.

I am aware of some stability issues on #Universeodon - I think this is a case of a badly tuned database backup that's causing things to lock up, part of this will be improved in the next week or so with some hardware upgrades to our servers but until then I'll do what I can to get things more stable.

Okay, I think #Universeodon should now be up, running and stable. There were a couple of bits I missed when I had deployed everything ready for the DB migration so we ran into some issues where we just got crushed with load and the server couldn't handle it properly.

It has flagged some other issues that I wasn't expecting when it comes to capacity and load but those are things I should be able to solve in due course and as part of moving MastodonAppUK over.

#MastoAdmin

Okay, I think #Universeodon should now be up, running and stable. There were a couple of bits I missed when I had deployed everything ready for the DB migration so we ran into some issues where we just got crushed with load and the server couldn't handle it properly.

It has flagged some other issues that I wasn't expecting when it comes to capacity and load but those are things I should be able to solve in due course and as part of moving MastodonAppUK over.

#MastoAdmin

For folks on #Universeodon - I've now ran #Wrapstodon so if you take a look on the website you should be able to view your Wrapstodon update.

Some apps will likely show an unknown notification type and it does only work currently on the web!

For folks on #Universeodon - I've now ran #Wrapstodon so if you take a look on the website you should be able to view your Wrapstodon update.

Some apps will likely show an unknown notification type and it does only work currently on the web!

I *think* I've now got a working replicated database running on our new infrastructure for #Universeodon which seems to be keeping in sync nicely with the primary. I need to do a load more checks to be 100% sure all the data is there. I'm probably going to look to take the site offline for a short window on Saturday to switch the site and everything over to the new DB server which I'm hoping should be the final big task to get Universeodon up and running on the new servers fully.

I need to get the MastodonAppUK DB Replicated as well which I'll probably look to do this evening / tomorrow at some point and once I'm happy that's also working I'll look to get that switched over.