Staging Performance Restored Under Moderate Load
Incident Report for Sonatype
Resolved
oss.sonatype.org has survived the long weekend, and performance appears to be solid now that we have a manageable number of staging repositories.
Posted Oct 15, 2019 - 10:52 EDT
Update
After our most recent scheduled maintenance, which dropped over 2000 open staging repositories, it appears the oss.sonatype.org is keeping up with the volume of staging operations on a Friday afternoon. We will continue to monitor performance over the long weekend and will post updates to this incident as necessary.
Posted Oct 11, 2019 - 17:21 EDT
Monitoring
Things are looking good now, but we're still watching for staging operations to get blocked. We are still planning a bulk cleanup of open repos once our developers have finished testing some tools to do that quickly and safely.
Posted Oct 11, 2019 - 14:37 EDT
Update
We were hoping to get a larger volume of staging repositories on oss.sonatype.org cleaned up during the recently completed maintenance window, but were only able to clean up about 200. The blocks on staging operations have been lifted, but we may have to schedule another window for more cleanup later today.
Posted Oct 11, 2019 - 01:02 EDT
Investigating
Our changes on oss.sonatype.org from earlier today seem only to have had a minor positive impact on web UI responsiveness. At current staging operation levels, staging operations are still taking well over 5 minutes to complete, which means continued 504s for those currently publishing. If you can pause your activities for the next hour or two, that would help give us breathing room while we investigate.
Posted Oct 10, 2019 - 13:05 EDT
This incident affected: Central (OSSRH).