Degrading performance of staging operations on oss.sonatype.org

Incident Report for Maven Central

Resolved

We found that a timeout value had been rolled back to an incorrect value during yesterday's migration process. That has been corrected, and our monitors indicate that staging operations are no longer stacking up, and most seem to be completing in under a minute.
Posted Mar 27, 2020 - 15:26 EDT

Monitoring

We just completed a restart to migrate to scaled up infrastructure. We're monitoring to see if staging operations performance improves.
Posted Mar 27, 2020 - 13:53 EDT

Update

It appears that opening/closing/releasing staging repositories is being handled fine by oss.sonatype.org, but drops are being batched up and taking longer than the default Maven timeout.
Posted Mar 27, 2020 - 12:45 EDT

Identified

We will need to issue a series of restarts for some scaling and cleanup operations. You may see 504 errors replaced by 405s and 502s. If at all possible, you may want to consider pausing your deployments for the next hour.
Posted Mar 27, 2020 - 12:13 EDT

Investigating

The restart seems to have provided only temporary relief. Staging operations look like they're still stacking up, and users are reporting 504 timeouts.
Posted Mar 27, 2020 - 11:49 EDT
This incident affected: Publishing - oss.sonatype.org.