tag:status.cit-net.com,2005:/historyCIT Services Status Status - Incident History2024-03-29T00:22:56-05:00CIT Services Statustag:status.cit-net.com,2005:Incident/203189202024-03-21T08:53:58-05:002024-03-21T08:53:58-05:00Service Desk Phone Queue Outage<p><small>Mar <var data-var='date'>21</var>, <var data-var='time'>08:53</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>21</var>, <var data-var='time'>07:00</var> CDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.cit-net.com,2005:Incident/200682252024-02-24T21:00:56-06:002024-02-24T21:00:56-06:00ConnectWise Manage (PSA) Ticketing system Patching<p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>21:00</var> CST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>20:00</var> CST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>19:45</var> CST</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.cit-net.com,2005:Incident/200505012024-02-22T15:23:46-06:002024-02-22T15:23:46-06:00Control Server Emergency Maintenance<p><small>Feb <var data-var='date'>22</var>, <var data-var='time'>15:23</var> CST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>22</var>, <var data-var='time'>14:43</var> CST</small><br><strong>Update</strong> - We needed to do another reboot to complete the update of the EDR software. Server is back up and we are continuing to monitor and test for normal operations.</p><p><small>Feb <var data-var='date'>22</var>, <var data-var='time'>14:03</var> CST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'>22</var>, <var data-var='time'>12:23</var> CST</small><br><strong>Investigating</strong> - We are investigating a CW Control issue related to our recent patching. Server will be unavailable for a bit while we reboot and confirm settings.</p>tag:status.cit-net.com,2005:Incident/200303432024-02-20T14:39:10-06:002024-02-20T14:39:10-06:00Automate and Control Issues<p><small>Feb <var data-var='date'>20</var>, <var data-var='time'>14:39</var> CST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>20</var>, <var data-var='time'>10:42</var> CST</small><br><strong>Monitoring</strong> - Link between Automate and ScreenConnect should be working again now. We are continuing to push agent updates out to the managed computers, but core functionality should be back up and working and the servers are on the latest patches for security. Might be some slower response time on the load as the system is very busy pushing the agent updates out.</p><p><small>Feb <var data-var='date'>20</var>, <var data-var='time'>10:29</var> CST</small><br><strong>Identified</strong> - We have found that last nights ScreenConnect Patch broke the control connection to Automate. We are applying the newest patch to automate to better support the new Screenconnect version.</p><p><small>Feb <var data-var='date'>20</var>, <var data-var='time'>09:55</var> CST</small><br><strong>Investigating</strong> - We are currently looking into issues with CW Control and Automate working correctly. We will also be applying emergency patches to the system.</p>tag:status.cit-net.com,2005:Incident/197655062024-01-19T12:32:01-06:002024-01-19T12:32:01-06:00Automate / Proactive.cit-net.com is down<p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>12:32</var> CST</small><br><strong>Resolved</strong> - The rebalance of resources has completed and all services should be running at normal levels of performance.</p><p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>10:37</var> CST</small><br><strong>Update</strong> - We have identified the root cause of the issue. We have ordered a replacement for the problem component and will get back to full performance as soon as we can get that part. In the meantime most services should be operating with acceptable performance since we have shut down some non essential devices to provide the remaining resources to our critical services.</p><p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>10:03</var> CST</small><br><strong>Update</strong> - Still working to resolve the performance issues.</p><p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>08:36</var> CST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>07:41</var> CST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.cit-net.com,2005:Incident/193597262023-12-07T18:30:23-06:002023-12-07T18:30:23-06:00CIT CW Automate and Control Server Patching<p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>18:30</var> CST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>17:30</var> CST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Dec <var data-var='date'> 6</var>, <var data-var='time'>17:07</var> CST</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.cit-net.com,2005:Incident/193139472023-12-04T14:53:03-06:002023-12-04T14:53:03-06:00CIT Phones are down<p><small>Dec <var data-var='date'> 4</var>, <var data-var='time'>14:53</var> CST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'> 4</var>, <var data-var='time'>09:57</var> CST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Dec <var data-var='date'> 4</var>, <var data-var='time'>08:20</var> CST</small><br><strong>Identified</strong> - We have identified the issue and are working to apply a fix to the issue. Phones should be back up and running soon.</p><p><small>Dec <var data-var='date'> 4</var>, <var data-var='time'>08:19</var> CST</small><br><strong>Investigating</strong> - Please use email or texting to reach us until we are able to resolve the phone issues.</p>tag:status.cit-net.com,2005:Incident/192617052023-12-01T14:53:04-06:002023-12-01T14:53:04-06:00Automate Remote Session Inturruption<p><small>Dec <var data-var='date'> 1</var>, <var data-var='time'>14:53</var> CST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'> 1</var>, <var data-var='time'>14:38</var> CST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Dec <var data-var='date'> 1</var>, <var data-var='time'>09:02</var> CST</small><br><strong>Investigating</strong> - We are investigating an issue where remote sessions are not able to be started with an error of "Unable to retrieve connection information for this computer. Confirm Control is enabled".</p>tag:status.cit-net.com,2005:Incident/191509452023-11-20T07:40:59-06:002023-11-20T07:40:59-06:00Internal CIT Outbound Email Issue<p><small>Nov <var data-var='date'>20</var>, <var data-var='time'>07:40</var> CST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>17</var>, <var data-var='time'>11:52</var> CST</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Nov <var data-var='date'>17</var>, <var data-var='time'>10:43</var> CST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Nov <var data-var='date'>17</var>, <var data-var='time'>08:41</var> CST</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Nov <var data-var='date'>17</var>, <var data-var='time'>08:13</var> CST</small><br><strong>Identified</strong> - Vendor encryption service outage.</p><p><small>Nov <var data-var='date'>17</var>, <var data-var='time'>08:05</var> CST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.cit-net.com,2005:Incident/191075922023-11-13T12:13:13-06:002023-11-13T12:13:13-06:00ConnectWise Control Down<p><small>Nov <var data-var='date'>13</var>, <var data-var='time'>12:13</var> CST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>13</var>, <var data-var='time'>10:21</var> CST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Nov <var data-var='date'>13</var>, <var data-var='time'>07:47</var> CST</small><br><strong>Identified</strong> - We are investigating an issue this morning where remote sessions are being denied, stating a license error.</p>tag:status.cit-net.com,2005:Incident/188085612023-10-15T22:51:38-05:002023-10-15T22:51:38-05:00ConnectWise Manage (PSA) Ticketing system Patching<p><small>Oct <var data-var='date'>15</var>, <var data-var='time'>22:51</var> CDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Oct <var data-var='date'>15</var>, <var data-var='time'>22:44</var> CDT</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Oct <var data-var='date'>15</var>, <var data-var='time'>22:00</var> CDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct <var data-var='date'>15</var>, <var data-var='time'>21:59</var> CDT</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.cit-net.com,2005:Incident/183076452023-08-29T17:47:59-05:002023-08-29T17:47:59-05:00CW RMM Customer Logins<p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>17:47</var> CDT</small><br><strong>Resolved</strong> - Vendor has confirmed they rolled back changes and we have enabled Client Site manager accounts again.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>12:34</var> CDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>10:41</var> CDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>09:14</var> CDT</small><br><strong>Investigating</strong> - We are working with ConnectWise support on a customer login issue. During this time, we are disabling all customer logins until we can resolve the problem. If during this time you need remote access to one of your devices, please contact the service desk at 651-255-5799 and we will assist you with resolving the issue. We apologize for any inconvenience this may cause.</p>tag:status.cit-net.com,2005:Incident/181595662023-08-16T00:00:00-05:002023-08-16T01:03:09-05:00Proactive/Automate Incident<p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>00:00</var> CDT</small><br><strong>Resolved</strong> - Server stopped receiving updates from devices.<br />It generated offline server tickets for most servers. These were false alerts.<br />Problem is resolved and tickets are automatically closing.</p>tag:status.cit-net.com,2005:Incident/180882522023-08-09T19:46:41-05:002023-08-09T19:46:41-05:00CIT Ticketing System server maintenance.<p><small>Aug <var data-var='date'> 9</var>, <var data-var='time'>19:46</var> CDT</small><br><strong>Completed</strong> - We have completed the maintenance work and all services are back to normal operations.</p><p><small>Aug <var data-var='date'> 9</var>, <var data-var='time'>19:37</var> CDT</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Aug <var data-var='date'> 9</var>, <var data-var='time'>19:37</var> CDT</small><br><strong>Update</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Aug <var data-var='date'> 9</var>, <var data-var='time'>19:01</var> CDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Aug <var data-var='date'> 8</var>, <var data-var='time'>14:50</var> CDT</small><br><strong>Scheduled</strong> - We will be performing maintenance on our Ticketing system during this time. We will stop all new email requests from converting to ticket until our maintenance is complete. All pending emails will be processed into tickets after this maintenance is completed.</p>tag:status.cit-net.com,2005:Incident/179898142023-07-30T18:44:42-05:002023-07-30T18:44:42-05:00Automate access issue<p><small>Jul <var data-var='date'>30</var>, <var data-var='time'>18:44</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>30</var>, <var data-var='time'>18:19</var> CDT</small><br><strong>Update</strong> - We are continuing to monitor after making additional corrections this morning. System has been up and appears to be stable after the changes today.</p><p><small>Jul <var data-var='date'>29</var>, <var data-var='time'>22:34</var> CDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>16:56</var> CDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>16:55</var> CDT</small><br><strong>Investigating</strong> - We are aware of reported issues accessing the Automate web portal and desktop client.<br />We are investigating and will update.</p>tag:status.cit-net.com,2005:Incident/178919202023-07-18T13:30:11-05:002023-07-18T13:30:11-05:00Automate Maintenance<p><small>Jul <var data-var='date'>18</var>, <var data-var='time'>13:30</var> CDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jul <var data-var='date'>18</var>, <var data-var='time'>13:15</var> CDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jul <var data-var='date'>18</var>, <var data-var='time'>13:11</var> CDT</small><br><strong>Scheduled</strong> - Good afternoon,<br /><br />We are performing brief maintenance to resolve remote access issues within Automate.<br />This maintenance will NOT be service impacting your local networks.<br /><br />Best,<br />CIT Services team</p>tag:status.cit-net.com,2005:Incident/178565942023-07-14T17:12:17-05:002023-07-14T17:12:17-05:00ConnectWise RMM Slowness/Instability<p><small>Jul <var data-var='date'>14</var>, <var data-var='time'>17:12</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>14</var>, <var data-var='time'>16:12</var> CDT</small><br><strong>Monitoring</strong> - Further update from ConnectWise:<br />"A fix has been implemented for Asio Platform slowness and we are monitoring the results"</p><p><small>Jul <var data-var='date'>14</var>, <var data-var='time'>13:12</var> CDT</small><br><strong>Investigating</strong> - Notes from ConnectWise pertaining to slowness in RMM<br />"Update: <br />We are investigating reports of Asio Platform slowness. We will follow up with more information soon."</p>tag:status.cit-net.com,2005:Incident/178503142023-07-13T22:52:30-05:002023-07-13T22:52:31-05:00After-Hours Call Routing Issue<p><small>Jul <var data-var='date'>13</var>, <var data-var='time'>22:52</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>13</var>, <var data-var='time'>22:34</var> CDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.cit-net.com,2005:Incident/176939022023-06-26T21:00:29-05:002023-06-26T21:00:29-05:00Connectwise Manage Patching<p><small>Jun <var data-var='date'>26</var>, <var data-var='time'>21:00</var> CDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'>26</var>, <var data-var='time'>20:00</var> CDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>26</var>, <var data-var='time'>19:46</var> CDT</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.cit-net.com,2005:Incident/176312432023-06-20T10:30:56-05:002023-06-20T10:30:56-05:00Expired SSL Cert<p><small>Jun <var data-var='date'>20</var>, <var data-var='time'>10:30</var> CDT</small><br><strong>Resolved</strong> - Control.cit-net.com has been updated and all services should be back to normal operation.</p><p><small>Jun <var data-var='date'>20</var>, <var data-var='time'>09:42</var> CDT</small><br><strong>Update</strong> - Proactvie.cit-net.com is back to normal operation with the renewed Cert installed.</p><p><small>Jun <var data-var='date'>20</var>, <var data-var='time'>09:37</var> CDT</small><br><strong>Identified</strong> - Support.cit-net.com is back up and secured with renewed SSL Certificate. Still working to apply renewed cert to Proactive and Control sites.</p><p><small>Jun <var data-var='date'>20</var>, <var data-var='time'>09:17</var> CDT</small><br><strong>Investigating</strong> - We are working to renew the SSL Certificate as quickly as possible.</p>tag:status.cit-net.com,2005:Incident/175644372023-06-13T16:50:07-05:002023-06-13T16:50:07-05:00CIT Ticketing system performance issues.<p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>16:50</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>16:15</var> CDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>15:48</var> CDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>15:26</var> CDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.cit-net.com,2005:Incident/174897742023-06-06T10:50:45-05:002023-06-06T10:50:45-05:00Services - Phone Queue Issue<p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>10:50</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>08:39</var> CDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>08:36</var> CDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.cit-net.com,2005:Incident/174815982023-06-05T14:28:21-05:002023-06-05T14:28:21-05:00Services - Phone Queue Outage<p><small>Jun <var data-var='date'> 5</var>, <var data-var='time'>14:28</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'> 5</var>, <var data-var='time'>14:21</var> CDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jun <var data-var='date'> 5</var>, <var data-var='time'>13:30</var> CDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Jun <var data-var='date'> 5</var>, <var data-var='time'>13:23</var> CDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.cit-net.com,2005:Incident/174233062023-05-31T20:00:23-05:002023-05-31T20:00:23-05:00CIT Automate Maintenance<p><small>May <var data-var='date'>31</var>, <var data-var='time'>20:00</var> CDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May <var data-var='date'>31</var>, <var data-var='time'>18:30</var> CDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'>30</var>, <var data-var='time'>12:22</var> CDT</small><br><strong>Scheduled</strong> - Greetings!<br /><br />Please be advised that the CIT Services Support Team will be performing scheduled maintenance on Wednesday May 31st at 6:30 PM to an estimated 8:00PM CST. This maintenance will NOT be service impacting your local networks.<br /><br />During the maintenance window, CIT remote monitoring and management will be available but there may be slight performance loss. If you are actively working with our support team during this window, you may experience brief interruptions to remote troubleshooting control. If for any reason you experience any disruptions, please call our Service Desk at 651.255.5799.<br /><br />Best,<br />CIT Services team</p>tag:status.cit-net.com,2005:Incident/172667422023-05-16T10:18:05-05:002023-05-16T10:18:05-05:00Connectwise Manage and Control Outage<p><small>May <var data-var='date'>16</var>, <var data-var='time'>10:18</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>16</var>, <var data-var='time'>08:41</var> CDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>