tag:status.vanillaforums.com,2005:/historyVanilla Forums Status - Incident History2024-03-29T03:52:29-04:00Vanilla Forumstag:status.vanillaforums.com,2005:Incident/203526602024-03-25T16:00:51-04:002024-03-25T16:00:52-04:00Knowledge Base Issues on Enterprise Sites<p><small>Mar <var data-var='date'>25</var>, <var data-var='time'>16:00</var> EDT</small><br><strong>Resolved</strong> - We have deployed a fix for this issue and full Knowledge Base access has been restored.</p><p><small>Mar <var data-var='date'>25</var>, <var data-var='time'>13:05</var> EDT</small><br><strong>Investigating</strong> - Navigating to Knowledge Base articles directly is resulting in an error. Navigating articles via the Knowledge Base page is working as expected. Our product team is investigating the issue.</p>tag:status.vanillaforums.com,2005:Incident/202167902024-03-12T00:35:35-04:002024-03-12T00:35:35-04:00Intermittent Connectivity Issues<p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>00:35</var> EDT</small><br><strong>Resolved</strong> - All hosted communities should be back to normal.</p><p><small>Mar <var data-var='date'>11</var>, <var data-var='time'>20:31</var> EDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Mar <var data-var='date'>11</var>, <var data-var='time'>19:04</var> EDT</small><br><strong>Update</strong> - The worst of the impact on hosted communities should be over at this time, but we are continuing to monitor and will update this status when the updates are complete</p><p><small>Mar <var data-var='date'>11</var>, <var data-var='time'>12:18</var> EDT</small><br><strong>Monitoring</strong> - Our host is performing some upgrades and this is leading to intermittent errors and poor performance for community sites.<br /><br />We are monitoring the situation.</p>tag:status.vanillaforums.com,2005:Incident/202085722024-03-10T15:04:37-04:002024-03-10T15:05:41-04:00Performance degradation in our Montréal (YUL) datacenter<p><small>Mar <var data-var='date'>10</var>, <var data-var='time'>15:04</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>10</var>, <var data-var='time'>14:55</var> EDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Mar <var data-var='date'>10</var>, <var data-var='time'>14:39</var> EDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Mar <var data-var='date'>10</var>, <var data-var='time'>11:00</var> EDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.vanillaforums.com,2005:Incident/201948262024-03-08T17:15:56-05:002024-03-08T17:15:57-05:00Security Updates<p><small>Mar <var data-var='date'> 8</var>, <var data-var='time'>17:15</var> EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'> 8</var>, <var data-var='time'>11:15</var> EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'> 8</var>, <var data-var='time'>11:04</var> EST</small><br><strong>Scheduled</strong> - We are performing maintenance updates to Enterprise clusters. We do not expect to see any down time for sites, but there may be a performance impact when the updates are being applied</p>tag:status.vanillaforums.com,2005:Incident/201856632024-03-07T17:42:05-05:002024-03-07T17:42:05-05:00Ongoing Maintenance on Enterprise Clusters<p><small>Mar <var data-var='date'> 7</var>, <var data-var='time'>17:42</var> EST</small><br><strong>Resolved</strong> - Upgrades are done for the day of 7 March.</p><p><small>Mar <var data-var='date'> 7</var>, <var data-var='time'>10:05</var> EST</small><br><strong>Monitoring</strong> - We are performing maintenance updates to Enterprise clusters. We do not expect to see any down time for sites, but there may be a performance impact when the updates are being applied</p>tag:status.vanillaforums.com,2005:Incident/199951752024-02-15T20:35:54-05:002024-02-15T20:35:54-05:00Partial Outage on Hosted Communities<p><small>Feb <var data-var='date'>15</var>, <var data-var='time'>20:35</var> EST</small><br><strong>Resolved</strong> - All hosted communities have been returned to full functionality.</p><p><small>Feb <var data-var='date'>15</var>, <var data-var='time'>16:51</var> EST</small><br><strong>Monitoring</strong> - We have identified the issue where some hosted communities were down. We have implemented a fix and are now monitoring the issue.</p>tag:status.vanillaforums.com,2005:Incident/199942812024-02-15T12:00:00-05:002024-02-15T14:33:15-05:00Database issues for certain sites in our YUL/Montréal datacenter<p><small>Feb <var data-var='date'>15</var>, <var data-var='time'>12:00</var> EST</small><br><strong>Resolved</strong> - We've identified some issues related to the databases upgrades we have done yesterday. Those are currently rebuilding, and a few customer's sites are affected. We're doing everything we can to bring those back up ASAP.</p>tag:status.vanillaforums.com,2005:Incident/199951582024-02-15T05:00:00-05:002024-02-15T16:49:33-05:00Partial Service Outage<p><small>Feb <var data-var='date'>15</var>, <var data-var='time'>05:00</var> EST</small><br><strong>Resolved</strong> - We have received reports that some clients cannot access their community sites.</p>tag:status.vanillaforums.com,2005:Incident/199253102024-02-14T20:45:56-05:002024-02-14T20:45:56-05:00Database and Application Software Upgrade on Corporate Public Sites in YUL region<p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>20:45</var> EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>19:30</var> EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>16:27</var> EST</small><br><strong>Scheduled</strong> - As part of our commitment to enhancing security and compliance, our Platform team will be performing essential database and application upgrades on client Corporate Public Sites in the YUL region. Please be advised that these upgrades may result in intermittent site disruptions. We appreciate your understanding and patience during this process, as these improvements are crucial for ensuring a safer and more compliant user experience.</p>tag:status.vanillaforums.com,2005:Incident/199252982024-02-13T20:45:56-05:002024-02-13T20:45:57-05:00Database and Application software Upgrade on Corporate Public Sites in AMS region<p><small>Feb <var data-var='date'>13</var>, <var data-var='time'>20:45</var> EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>13</var>, <var data-var='time'>19:30</var> EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>16:26</var> EST</small><br><strong>Scheduled</strong> - As part of our commitment to enhancing security and compliance, our Platform team will be performing essential database and application upgrades on client Corporate Public Sites in the AMS region. Please be advised that these upgrades may result in intermittent site disruptions. We appreciate your understanding and patience during this process, as these improvements are crucial for ensuring a safer and more compliant user experience.</p>tag:status.vanillaforums.com,2005:Incident/199252882024-02-12T20:00:57-05:002024-02-12T20:00:58-05:00Database and Application Software Upgrade on Corporate Public Sites in JSC region<p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>20:00</var> EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>19:30</var> EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>16:25</var> EST</small><br><strong>Scheduled</strong> - As part of our commitment to enhancing security and compliance, our Platform team will be performing essential database and application upgrades on client Corporate Public Sites in the JSC region. Please be advised that these upgrades may result in intermittent site disruptions. We appreciate your understanding and patience during this process, as these improvements are crucial for ensuring a safer and more compliant user experience.</p>tag:status.vanillaforums.com,2005:Incident/199310562024-02-07T11:15:21-05:002024-02-07T11:15:21-05:00Queue Server Outage<p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>11:15</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>10:19</var> EST</small><br><strong>Monitoring</strong> - We've resolved the issue and the queue is processing jobs again. We currently adding additional capacity and monitoring the situation.</p><p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>09:47</var> EST</small><br><strong>Identified</strong> - The worker servers that execute jobs in our queue service are currently degraded and not processing jobs. Expect delays with webhooks and addition of new records into search.</p>tag:status.vanillaforums.com,2005:Incident/199143242024-02-05T16:23:08-05:002024-02-05T16:23:08-05:00Queue Performance Degradation<p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>16:23</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>15:15</var> EST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>12:41</var> EST</small><br><strong>Identified</strong> - Our message queue system is currently experiencing a backlog, which is causing delays in notifications, webhooks, and analytics data. We have identified the issue and are actively working on a solution.</p>tag:status.vanillaforums.com,2005:Incident/198766252024-02-05T12:16:08-05:002024-02-05T12:16:08-05:00Database and Application software Upgrade on Sub-corporate Plan Public Sites<p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>12:16</var> EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>11:02</var> EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>10:24</var> EST</small><br><strong>Scheduled</strong> - Our Platform team will conduct essential database and application upgrades on client Sub-corporate Plan Public Sites next Monday to enhance security and compliance. We appreciate your understanding and patience during this process, as these improvements are vital for a safer and more compliant user experience.</p>tag:status.vanillaforums.com,2005:Incident/199038492024-02-03T22:50:21-05:002024-02-03T22:50:21-05:00Outage in YUL1 / Montréal<p><small>Feb <var data-var='date'> 3</var>, <var data-var='time'>22:50</var> EST</small><br><strong>Resolved</strong> - Systems are stable again. Provider is investigating the cause of the network issues.</p><p><small>Feb <var data-var='date'> 3</var>, <var data-var='time'>22:38</var> EST</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Feb <var data-var='date'> 3</var>, <var data-var='time'>22:31</var> EST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'> 3</var>, <var data-var='time'>22:21</var> EST</small><br><strong>Investigating</strong> - We're in contact with our hosting provider and actively investigating. Some VMs have become unreachable. Control plane suspected.</p>tag:status.vanillaforums.com,2005:Incident/198962112024-02-02T16:55:26-05:002024-02-02T16:55:26-05:00Performance issues in our YUL-1 datacenter<p><small>Feb <var data-var='date'> 2</var>, <var data-var='time'>16:55</var> EST</small><br><strong>Resolved</strong> - All systems have recovered and our hosting provider is working on replacing the faulty disk.</p><p><small>Feb <var data-var='date'> 2</var>, <var data-var='time'>13:30</var> EST</small><br><strong>Monitoring</strong> - We've identified performance issues in our YUL-1 datacenter that our provider identified as increased latency in our storage cluster, causing delays and timeouts. The disk have been taken offline and services are catching up, we're monitoring until full system recovery.</p>tag:status.vanillaforums.com,2005:Incident/198601222024-01-29T15:30:56-05:002024-01-29T15:30:56-05:00Database and Application software Upgrade in Staging<p><small>Jan <var data-var='date'>29</var>, <var data-var='time'>15:30</var> EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>29</var>, <var data-var='time'>14:30</var> EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>29</var>, <var data-var='time'>13:16</var> EST</small><br><strong>Scheduled</strong> - Our Platform team is scheduled to perform database and application upgrades on client staging environments next Monday to enhance security and compliance. We appreciate your understanding and patience during this process, as these improvements are crucial for a safer and more compliant user experience.</p>tag:status.vanillaforums.com,2005:Incident/198381722024-01-29T10:30:11-05:002024-01-29T13:08:24-05:00Database and Application Upgrades in Staging.<p><small>Jan <var data-var='date'>29</var>, <var data-var='time'>10:30</var> EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>29</var>, <var data-var='time'>09:32</var> EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>29</var>, <var data-var='time'>03:00</var> EST</small><br><strong>Scheduled</strong> - Our Platform team is scheduled to perform database and application upgrades on client staging environments next Monday to enhance security and compliance. We appreciate your understanding and patience during this process, as these improvements are crucial for a safer and more compliant user experience.</p>tag:status.vanillaforums.com,2005:Incident/198399732024-01-26T15:26:15-05:002024-01-26T15:26:15-05:00Queue Performance in Canada / Montreal datacenters is degraded<p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>15:26</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>14:27</var> EST</small><br><strong>Monitoring</strong> - We've added additional capacity to this queue system and the backlog is being cleared. Delays are down to ~5 minutes. We will monitor until everything is caught up.</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>13:32</var> EST</small><br><strong>Identified</strong> - Queued jobs such as webhooks, recording of analytics, and sending of email notifications may delayed by up to 30 minutes.</p>tag:status.vanillaforums.com,2005:Incident/196733032024-01-17T00:00:22-05:002024-01-17T00:00:23-05:00Hypervisor Upgrades (US/San Jose)<p><small>Jan <var data-var='date'>17</var>, <var data-var='time'>00:00</var> EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>16</var>, <var data-var='time'>19:00</var> EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>20:45</var> EST</small><br><strong>Scheduled</strong> - Our hosting will be performing software updates on our hypervisors. This will require some virtual machines to be rebooted, which may cause some sites to show intermittent errors especially as their databases are being rebooted and the cluster rebalances.<br /><br />We don't anticipate more than 5 minutes of downtime per site.</p>tag:status.vanillaforums.com,2005:Incident/196732952024-01-11T22:00:32-05:002024-01-11T22:00:33-05:00Hypervisor Upgrades (Europe/Amsterdam)<p><small>Jan <var data-var='date'>11</var>, <var data-var='time'>22:00</var> EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>11</var>, <var data-var='time'>17:00</var> EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>20:43</var> EST</small><br><strong>Scheduled</strong> - Our hosting will be performing software updates on our hypervisors. This will require some virtual machines to be rebooted, which may cause some sites to show intermittent errors especially as their databases are being rebooted and the cluster rebalances.<br /><br />We don't anticipate more than 5 minutes of downtime per site.</p>tag:status.vanillaforums.com,2005:Incident/191561402023-11-20T02:17:58-05:002023-11-20T02:17:58-05:00Intermittent 522 errors from Cloudflare's DEN datacenter<p><small>Nov <var data-var='date'>20</var>, <var data-var='time'>02:17</var> EST</small><br><strong>Resolved</strong> - Cloudflare has rerouted the traffic for that datacenter and the issue appear resolved. As of about 30 minutes ago, all traffic is back to normal and we're now seeing zero dropped requests from DEN.</p><p><small>Nov <var data-var='date'>17</var>, <var data-var='time'>21:51</var> EST</small><br><strong>Update</strong> - We've had a long chat with Cloudflare's support team and the issue is being escalated to their network team. There appear to be some occasional packet drops between Denver and our Montréal datacenter. We estimate about 4% of traffic coming in from Denver is taking more than 5 seconds to respond and/or times out. Traffic originating from all other regions are unaffected and operating normally.<br /><br />We continue to investigate and await further responses from Cloudflare.</p><p><small>Nov <var data-var='date'>17</var>, <var data-var='time'>09:30</var> EST</small><br><strong>Identified</strong> - Since this morning at approximately 9:30, we've been observing intermittent 522 errors from Cloudflare that are isolated to their DEN (Denver, Colorado, USA) datacenter. No other datacenters are impacted. Users located in Colorado will experience intermittent 522 errors from Cloudflare.<br /><br />We have opened a ticket with Cloudflare and are awaiting updates.</p>tag:status.vanillaforums.com,2005:Incident/190799092023-11-14T00:02:11-05:002023-11-14T00:02:12-05:00Hypervisor Upgrades<p><small>Nov <var data-var='date'>14</var>, <var data-var='time'>00:02</var> EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov <var data-var='date'>13</var>, <var data-var='time'>19:00</var> EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>21:06</var> EST</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>21:05</var> EST</small><br><strong>Scheduled</strong> - We will be performing hypervisor upgrades which requires instances to be rebooted. Customers may experience a few minutes of downtime as the database servers reboot, and potentially a few dropped requests as traffic gets rerouted internally.</p>tag:status.vanillaforums.com,2005:Incident/190798982023-11-10T21:00:14-05:002023-11-10T21:00:16-05:00Hypervisor Upgrades<p><small>Nov <var data-var='date'>10</var>, <var data-var='time'>21:00</var> EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov <var data-var='date'>10</var>, <var data-var='time'>19:00</var> EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>21:04</var> EST</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>21:03</var> EST</small><br><strong>Scheduled</strong> - We will be performing hypervisor upgrades which requires instances to be rebooted. Customers may experience a few minutes of downtime as the database servers reboot, and potentially a few dropped requests as traffic gets rerouted internally.</p>tag:status.vanillaforums.com,2005:Incident/190762932023-11-09T19:10:34-05:002023-11-09T19:10:37-05:00Database maintenance.<p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>19:10</var> EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>19:00</var> EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>13:28</var> EST</small><br><strong>Scheduled</strong> - We'll be performing maintenance on our MTL datacenter (YUL1). There might be a brief downtime, less than 5 minutes</p>