tag:ecoportal.statuspage.io,2005:/historyecoPortal Status - Incident History2024-02-23T23:24:17+13:00ecoPortaltag:ecoportal.statuspage.io,2005:Incident/200534452024-02-23T17:59:51+13:002024-02-23T23:24:17+13:00Entries without base tags not visible in Registers<p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>17:59</var> NZDT</small><br><strong>Resolved</strong> - Issue has been resolved and all entries should be visible in the next 30 minutes.</p><p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>14:29</var> NZDT</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>08:30</var> NZDT</small><br><strong>Identified</strong> - The engineering has identified the issue and is fixing the visibility of these entries inside the modules. An update will be made once full restored.</p><p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>08:00</var> NZDT</small><br><strong>Investigating</strong> - The engineering is actively investigating an issue with certain entries not being visible within specific modules.</p>tag:ecoportal.statuspage.io,2005:Incident/192091112023-11-30T09:06:32+13:002024-02-02T09:06:52+13:00Schedule exports delayed<p><small>Nov <var data-var='date'>30</var>, <var data-var='time'>09:06</var> NZDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>24</var>, <var data-var='time'>23:02</var> NZDT</small><br><strong>Identified</strong> - A small number of clients will have their scheduled exports delayed over the weekend due to an operational error. We are actively working to restore all affected scheduled exports.</p>tag:ecoportal.statuspage.io,2005:Incident/187220752023-10-09T11:05:30+13:002023-10-09T11:05:30+13:00Incorrect last updated at information for pages<p><small>Oct <var data-var='date'> 9</var>, <var data-var='time'>11:05</var> NZDT</small><br><strong>Resolved</strong> - No further issues were identified. The last updated information on the pages was fully restored. Email reports and scheduled exports have been restored</p><p><small>Oct <var data-var='date'> 9</var>, <var data-var='time'>08:54</var> NZDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Oct <var data-var='date'> 8</var>, <var data-var='time'>23:04</var> NZDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Oct <var data-var='date'> 8</var>, <var data-var='time'>19:51</var> NZDT</small><br><strong>Update</strong> - Schedule exports have also been disabled to avoid pages being included by mistake</p><p><small>Oct <var data-var='date'> 8</var>, <var data-var='time'>17:00</var> NZDT</small><br><strong>Update</strong> - Email reports sent in the last couple of hours should contain incorrect information. This feature has been disable while we restore the information. All email reports are going to be sent after the export is executed</p><p><small>Oct <var data-var='date'> 8</var>, <var data-var='time'>14:00</var> NZDT</small><br><strong>Identified</strong> - A migration has been executed against all pages this weekend to support a future release, unfortunately, it affected the "last updated at" information for all pages. We are currently restoring a copy from our production database to be able to restore the correct information.</p>tag:ecoportal.statuspage.io,2005:Incident/186824022023-10-04T13:17:40+13:002023-10-04T13:17:40+13:00Batch Jobs including exports<p><small>Oct <var data-var='date'> 4</var>, <var data-var='time'>13:17</var> NZDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'> 4</var>, <var data-var='time'>13:02</var> NZDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Oct <var data-var='date'> 4</var>, <var data-var='time'>11:27</var> NZDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:ecoportal.statuspage.io,2005:Incident/186605522023-10-02T11:23:33+13:002023-10-02T11:23:33+13:00Auto-logout increased anomaly<p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>11:23</var> NZDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>10:15</var> NZDT</small><br><strong>Monitoring</strong> - A fix has been deployed and we're monitoring the results</p><p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>10:04</var> NZDT</small><br><strong>Identified</strong> - We identified and issue with our login database cache. The fix is being deployed.</p><p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>10:03</var> NZDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:ecoportal.statuspage.io,2005:Incident/185717712023-09-22T18:42:09+12:002023-09-22T18:42:09+12:00Cloudflare Network Performance Issues<p><small>Sep <var data-var='date'>22</var>, <var data-var='time'>18:42</var> NZST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>22</var>, <var data-var='time'>12:53</var> NZST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Sep <var data-var='date'>22</var>, <var data-var='time'>12:41</var> NZST</small><br><strong>Update</strong> - Cloudflare is continuing to investigate this issue.</p><p><small>Sep <var data-var='date'>22</var>, <var data-var='time'>12:08</var> NZST</small><br><strong>Investigating</strong> - Cloudflare is investigating issues with network performance. Customers may see increased 5xx errors and/or latency.</p>tag:ecoportal.statuspage.io,2005:Incident/185948862023-09-22T17:00:00+12:002023-09-25T10:12:51+13:00Failed build<p><small>Sep <var data-var='date'>22</var>, <var data-var='time'>17:00</var> NZST</small><br><strong>Resolved</strong> - A minor build issue was identified in the build of the production application for live.ecoportal.com. The root cause was investigated and a code fix was written to remedy this issue.<br /><br />22 September 2023 3:12pm: The code fix was released to live.ecoportal.com. This code fix related to the abilities service in ecoPortal where certain permissions function. Due to the nature of the change a certain aspect of abilities affecting people accounts was regressed and made unavailable.<br /><br />This impacted the following areas: people manager, people fields on pages, cross-reference register reports when reporting against the people manager, and adding new people in bulk via batch tools.<br /><br />25 September 2023 7:58am: The regression was raised with internal engineering teams via client services after external reports. <br /><br />The decision was made to immediately rollback the system after receiving the internal report.<br /><br />25 September 2023 8:34am: The rollback was applied to the production environment and the abilities service was restored back to its original state.</p>tag:ecoportal.statuspage.io,2005:Incident/185704422023-09-22T09:52:18+12:002023-09-22T09:52:18+12:00Cloudflare Network performance Issues<p><small>Sep <var data-var='date'>22</var>, <var data-var='time'>09:52</var> NZST</small><br><strong>Resolved</strong> - Cloudflare has resolved this incident.</p><p><small>Sep <var data-var='date'>22</var>, <var data-var='time'>09:43</var> NZST</small><br><strong>Monitoring</strong> - A fix has been implemented and Cloudflare is monitoring the results.</p><p><small>Sep <var data-var='date'>22</var>, <var data-var='time'>09:33</var> NZST</small><br><strong>Identified</strong> - The issue has been identified, and a fix is being implemented by Cloudflare.</p><p><small>Sep <var data-var='date'>22</var>, <var data-var='time'>09:25</var> NZST</small><br><strong>Investigating</strong> - Cloudflare is investigating issues with network performance. Customers may see increased 5xx errors and/or latency.</p>tag:ecoportal.statuspage.io,2005:Incident/178852422023-04-23T15:30:00+12:002023-07-18T16:22:23+12:00Elevated Errors<p><small>Apr <var data-var='date'>23</var>, <var data-var='time'>15:30</var> NZST</small><br><strong>Resolved</strong> - A short term data storage service (Redis) exceeded its allocated memory capacity and began throwing errors. This prevented a small set of logged out ecoPortal users from logging in during this window. The cause of error has been identified and fixed including increasing the capacity of the service and additional monitoring implemented.</p>tag:ecoportal.statuspage.io,2005:Incident/168975642023-04-19T14:48:37+12:002023-04-19T01:39:06+12:00Outage<p><small>Apr <var data-var='date'>19</var>, <var data-var='time'>14:48</var> NZST</small><br><strong>Resolved</strong> - The engineering team restored the server to a non-degraded state, and access to the system was restored.<br /><br />Remediation:<br />The offending cause due to an internal release was identified and rolled back from the system. Resources against the internal server will be increased overnight to limit the likelihood of overload in the future.</p><p><small>Apr <var data-var='date'>18</var>, <var data-var='time'>14:30</var> NZST</small><br><strong>Monitoring</strong> - An accelerated amount of requests were being received be an internal data storage server. This caused the server to overload and deny login attempts to live.ecoportal.com from any non-signed in users. The engineering team was alerted to this spike via internal tools and responded.</p>tag:ecoportal.statuspage.io,2005:Incident/159469952023-01-26T19:30:00+13:002023-07-18T16:20:57+12:00Elevated Request Errors<p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>19:30</var> NZDT</small><br><strong>Resolved</strong> - Related to the major outage from 26 Jan 23 Morning. Auto scaling of servers to meet demand inadvertently deployed copies of the defective application version in some clusters which caused a small subset of users to again see the error response from Nginx. The Nginx layer has since been reviewed as no longer necessary and removed.</p>tag:ecoportal.statuspage.io,2005:Incident/159464612023-01-26T12:17:00+13:002023-07-18T16:19:58+12:00Outage<p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>12:17</var> NZDT</small><br><strong>Resolved</strong> - As part of our ongoing efforts to keep a high standard of security our system adopts a strict Content Security Policy (CSP). Some days prior to the incident one of our clients requested the ability to use certain features of our embedded google maps tool, this required the introduction of a new directive to our CSP. This introduction exceeded a hard cap on the size of the CSP HTTP header leveraged by Nginx as a reverse server proxy. This caused a server error response from Nginx. The system was rolled back and the system state was restored.</p>tag:ecoportal.statuspage.io,2005:Incident/178852172022-11-14T17:16:00+13:002023-07-18T16:17:14+12:00third-party tool impact<p><small>Nov <var data-var='date'>14</var>, <var data-var='time'>17:16</var> NZDT</small><br><strong>Resolved</strong> - CloudFlare, a third-party tool utilized to serve client requests inbound to the ecoPortal application, had a partial outage related to to DNS resolution https://www.cloudflarestatus.com/incidents/wb69sy7zb4r6.</p>tag:ecoportal.statuspage.io,2005:Incident/178851972022-11-03T16:00:00+13:002023-07-18T16:13:51+12:00Data Processing Delays - Reporting Tools Affected<p><small>Nov <var data-var='date'> 3</var>, <var data-var='time'>16:00</var> NZDT</small><br><strong>Resolved</strong> - A very large request spike to our data replication layer (ElasticSearch) caused by internal tooling usage caused the service to be delayed in processing incoming events. This led to data appearing as “not visible” in certain areas of the application for extended periods of time, namely register tables and dashboards. This tooling has been moved to a separate processing layer to not interfere with regular application use by users.</p>tag:ecoportal.statuspage.io,2005:Incident/178852492022-08-03T04:00:00+12:002023-07-18T17:27:15+12:00Data Processing Delays<p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>04:00</var> NZST</small><br><strong>Resolved</strong> - A very large request spike to our data replication layer (ElasticSearch) caused the service to start rejecting some incoming requests. This led to data appearing as “not visible” in certain areas of the application for a small period of time, namely register tables and dashboards.</p>