Restart is complete, and we'll be monitoring the impact of our latest changes to staging performance.
Posted Jan 22, 2021 - 15:18 EST
We still need to restart oss.sonatype.org, and we'll be doing that within the hour. Expect 405 errors while we have staging operations paused leading up to the restart.
Posted Jan 22, 2021 - 14:43 EST
Staging operations have piled overnight. We will be pausing staging operations to allow those in flight to clear. This will cause new builds to fail with a 405 error. After that, we'll also be restarting oss.sonatype.org, so you can expect 5xx errors at that time.
Posted Jan 22, 2021 - 09:33 EST
oss.sonatype.org is on its way back up.
Posted Jan 21, 2021 - 22:24 EST
We will be restarting oss.sonatype.org one more time for one more config change. This will be the last service interruption for tonight.
Posted Jan 21, 2021 - 22:07 EST
We need to pause staging operations off and on for the next hour to allow some cleanup tasks to complete.
Posted Jan 21, 2021 - 20:37 EST
We're going to be pausing staging operations and restarting oss.sonatype.org on scaled up hardware in the next hour. Staging operations are still timing out, though at a slightly slower volume than this morning.
Posted Jan 21, 2021 - 17:11 EST
It seems like staging operations are slowing down again for no reason other than publishing volume. Staging operations have been paused temporarily.
Posted Jan 21, 2021 - 11:57 EST
A fix has been implemented and we are monitoring the results.
Posted Jan 21, 2021 - 11:24 EST
We noticed at least one case of a bulk drop of dozens of staging repositories. Drops happen during the normal course of releasing, but it appears that, given the current volume of projects hosted and actively publishing on oss.sonatype.org, bulk drops have an outsized impact on all staging operations.
Posted Jan 21, 2021 - 11:19 EST
It looks like staging repositories have once again piled up, so expect periods where we'll be pausing staging operations to allow whatever's running to complete. Builds will fail with 405 errors while staging operations are paused.
Posted Jan 21, 2021 - 10:21 EST
We suspect that builds are also being impacted, and we will restart the underlying services shortly to see if that provides relief.
Posted Jan 21, 2021 - 10:17 EST
This incident affected: Central (Publishing - oss.sonatype.org).