The engineering team restored the server to a non-degraded state, and access to the system was restored.
Remediation: The offending cause due to an internal release was identified and rolled back from the system. Resources against the internal server will be increased overnight to limit the likelihood of overload in the future.
Posted Apr 19, 2023 - 14:48 NZST
Monitoring
An accelerated amount of requests were being received be an internal data storage server. This caused the server to overload and deny login attempts to live.ecoportal.com from any non-signed in users. The engineering team was alerted to this spike via internal tools and responded.