Yesterday at approximately 6:35 PM Eastern Time, the OpenWater platform went down. Our self-healing systems brought the systems back online within 15 minutes. We continued to investigate the issue and uncovered the newly added Email Log feature (added in June) received a flood of notices that emails were being opened. Several customers sent out large email blasts and had many deliveries and opens logged in a short window. The Email Log component was coupled with some other components.
We have deployed updates this morning so that the Email Log component now runs in the background and a surge in traffic cannot bring down the Public site (for Entrants / Judges). We will be issuing another update this week to fully decouple it from the admin portal as well.