Between Jan 15, 21:10 UTC, and Jan 16, 05:40 UTC, we experienced sporadic run failures for source-to-target rivers on our US console, specifically during the source-to-FZ (File Zone) task. The issue was caused by a temporary drop in the shared disk’s ability to quickly update metadata. This delay led to occasional failures in the source phase of affected rivers, with runs encountering the error: "[Errno 17] File exists."
Impact: The issue only impacted rivers running on our US console. Sporadic failures occurred during the specified timeframe. Any affected runs need to be restarted from the last successful run.
Action Required: No action is required from users. Affected rivers will resume automatically on the next execution.
Resolution: The issue was fully resolved as of Jan 16, 05:40 UTC, and the system is now operating as expected.
We sincerely apologize for the inconvenience caused and appreciate your patience and understanding. If you have any questions, please don’t hesitate to reach out to our support team.