Resolved -
Dear Customers,
All services at the GUBOV data centre have been restored, and there are currently no known outstanding customer issues.
While services are stable, our teams will continue to perform follow-up work, including the replacement of certain hardware components, as part of the full remediation process. This work is not expected to impact customer services.
We will continue to closely monitor the environment throughout the weekend to ensure ongoing stability and proactively address any potential issues.
If you experience any service disruptions or have concerns, please contact ThinkOn Support for assistance.
Next Update
This incident will be considered resolved, and no further updates are planned unless new issues arise.
We appreciate your patience and support throughout this incident.
Apr 17, 17:37 EDT
Update -
Dear Customers,
Most outstanding issues related to the GUBOV data centre incident have now been resolved. Our teams remain actively engaged and will continue to closely monitor the network to ensure ongoing stability. Monitoring systems are fully functional and operational.
We kindly ask that customers continue to report any issues they may encounter so our teams can investigate and assist promptly.
Next Update
We will provide further updates as needed.
We appreciate your continued patience and support.
Apr 17, 11:10 EDT
Update -
Dear Customers,
We continue to actively monitor the environment at the GUBOV data centre to ensure ongoing stability and confirm that all services remain fully operational. At this time, there are no new issues identified; however, our teams remain engaged and vigilant in observing the environment.
We kindly ask that customers continue to validate their services and confirm that everything is functioning as expected. If you encounter any issues, please contact ThinkOn Support so our teams can assist promptly.
Service Desk operations may still experience minor delays as we complete final monitoring and validation activities.
Next Update
We will provide further updates as needed.
We appreciate your continued patience and support.
Apr 16, 18:46 EDT
Update -
Dear Customers,
We are continuing to monitor the environment at the GUBOV data centre to ensure ongoing stability and to identify any residual issues. We kindly ask that customers continue to verify their services and confirm that everything is operating as expected. If you encounter any issues, please contact ThinkOn Support so our teams can assist promptly. Service Desk operations may still experience minor delays as we complete final monitoring and validation.
Next Update
We will provide further updates as needed.
We appreciate your continued patience and support.
Apr 16, 08:54 EDT
Update -
Dear Customers,
Services have been restored, and we are currently monitoring the environment to ensure ongoing network stability at the GUBOV data centre. We kindly ask that customers continue their validation efforts and confirm that services are operating as expected. If you encounter any issues, please contact ThinkOn Support so our teams can assist promptly. Service Desk operations may still experience minor delays as we complete final monitoring and validation.
Next Update
We will provide further updates as needed.
We appreciate your continued patience and support.
Apr 16, 07:52 EDT
Update -
We are continuing to monitor for any further issues.
Apr 16, 07:45 EDT
Monitoring -
Dear Customers,
Site connectivity has been restored, and we are continuing to validate all services at the GUBOV data centre. At this time, we ask that you please attempt to connect to your environments and confirm that your services are functioning as expected. If you encounter any issues, please contact ThinkOn Support so our teams can assist promptly. Service Desk operations may still experience minor delays as we complete final validations.
Next Update
We will provide further updates as needed.
We appreciate your continued patience and support.
Apr 16, 07:30 EDT
Update -
Dear Customers,
We are currently completing core system validation as site connectivity is being restored at the GUBOV data centre. Our teams are working to confirm that all services are functioning as expected. Service Desk operations may still experience delays, and response times for new and existing tickets may be impacted during this time.
Next Update
We will provide further updates as more information becomes available.
We appreciate your continued patience and understanding.
Apr 16, 06:59 EDT
Update -
Dear Customers,
As we continue to stabilize the environment at the GUBOV data centre, you may experience brief, intermittent periods of connectivity while we complete the restoration of site services. Our teams are actively monitoring and working to ensure a full and stable recovery. Service Desk operations may still experience delays, and response times for new and existing tickets may be impacted during this time.
Next Update
We will provide further updates as more information becomes available.
We appreciate your continued patience and understanding.
Apr 16, 06:41 EDT
Update -
Dear Customers,
We are currently validating the recent changes and completing the reconnection of all systems at the GUBOV data centre to confirm full restoration of connectivity. Our teams are actively monitoring the environment to ensure services are stable as they come back online. Service Desk operations may still experience delays, and response times for new and existing tickets may be impacted during this time.
Next Update
We will provide further updates as more information becomes available.
We appreciate your continued patience and understanding.
Apr 16, 06:14 EDT
Update -
Dear Customers,
We are currently in the final stages of configuring the replacement equipment at the GUBOV data centre. If no issues are encountered, we expect to restore site connectivity shortly and begin verification testing. Our teams remain actively engaged to ensure a stable restoration of services. Service Desk operations may still experience delays, and response times for new and existing tickets may be impacted during this time.
Next Update
We will provide further updates as more information becomes available.
We appreciate your continued patience and understanding.
Apr 16, 02:35 EDT
Update -
Dear Customers,
We continue to work on restoring services at the GUBOV data centre. As part of the recovery effort, our teams are proceeding with the installation and configuration of replacement network equipment.
The following activities are currently underway:
Installation and racking of replacement switches
Firmware updates on the new devices
Application of site-specific configurations
Migration of existing cabling from the affected devices to the new equipment
Testing of connections and dependent systems
Ongoing monitoring and validation to ensure services are fully restored
This process will take some time to complete, and our technicians and engineers remain fully engaged to ensure a safe and stable recovery.
Service Desk operations may still experience delays, and response times for new and existing tickets may be impacted during this time.
Next Update
We will provide further updates as more information becomes available.
We appreciate your continued patience and understanding.
Apr 16, 01:27 EDT
Update -
Dear Customers,
We continue to investigate the widespread outage at the GUBOV data centre. At this time, we are proceeding with the next remediation step, which involves replacing the affected hardware. A technician is expected to be on-site with replacement equipment within the next 20 minutes. Once on-site, our team will begin racking, cabling, installing, and configuring the devices, followed by validation checks. This process will take some time to complete. We understand the significant impact this incident is having on your production environments and your customers, and we want to assure you that our teams are working with urgency to restore services as quickly as possible. Service Desk operations may still experience delays, and response times for new and existing tickets may be impacted during this time.
Next Update
We will provide further updates as more information becomes available.
We appreciate your continued patience and understanding.
Apr 16, 00:48 EDT
Update -
Dear Customers,
We continue to investigate the widespread outage at the GUBOV data centre. At this time, replacement hardware is being brought on-site. Our teams will be working to install, configure, and perform validation checks as part of the recovery effort. This process will take some time to complete, and our technicians and engineers remain fully engaged in restoring services as quickly and safely as possible. Service Desk operations may still experience delays, and response times for new and existing tickets may be impacted during this time.
Next Update
We will provide further updates as more information becomes available.
We appreciate your continued patience and understanding.
Apr 15, 23:56 EDT
Update -
Dear Customers,
We continue to investigate the widespread outage at the GUBOV data centre. Our teams are actively performing detailed manual checks and verifying configurations across both primary and secondary network paths to further isolate the issue. At this time, there are no significant changes to report; however, our technicians and engineers remain fully engaged and continue their investigation. Service Desk operations may still experience delays, and response times for new and existing tickets may be impacted during this time.
Next Update
We will provide further updates as more information becomes available.
We appreciate your continued patience and understanding.
Apr 15, 23:09 EDT
Update -
Dear Customers,
We continue to investigate the widespread outage at the GUBOV data centre. Our teams are actively performing both physical (on-site) and logical inspections, reviewing each device to further isolate the issue and determine the appropriate remediation steps. At this time, there are no significant changes to report; however, our technicians and engineers remain fully engaged and continue their detailed investigation. Service Desk operations may still experience delays, and response times for new and existing tickets may be impacted during this time.
Next Update
We will provide further updates as more information becomes available.
We appreciate your continued patience and understanding.
Apr 15, 22:35 EDT
Update -
Dear Customers,
We continue to investigate the widespread outage at the GUBOV data centre. Our teams are actively performing both physical (on-site) and logical inspections to further isolate the issue and determine the appropriate remediation steps. At this time, there are no significant changes to report; however, our technicians and engineers remain fully engaged and are continuing their investigation. Service Desk operations may still experience delays, and response times for new and existing tickets may be impacted during this time.
Next Update
We will provide further updates as more information becomes available.
We appreciate your continued patience and understanding.
Apr 15, 21:56 EDT
Update -
Dear Customers,
We continue to investigate the widespread outage at the GUBOV data centre. The corrective actions performed thus far have not yet resolved the issue, and our teams are continuing to investigate and implement additional remediation steps. Our technicians and engineers remain fully engaged and are working to restore services as quickly as possible. Service Desk operations may still experience delays, and response times for new and existing tickets may be impacted during this time.
Next Update
We will provide further updates as more information becomes available.
We appreciate your continued patience and understanding.
Apr 15, 21:22 EDT
Update -
Dear Customers,
We continue to investigate the widespread outage at the GUBOV data centre. Our teams have identified the issue and are currently performing corrective actions, including controlled reboots of affected equipment, to restore services. Our technicians and engineers remain actively engaged in resolving the issue as quickly as possible. Service Desk operations may still experience delays, and response times for new and existing tickets may be impacted during this time.
Next Update
We will provide further updates as more information becomes available.
We appreciate your continued patience and understanding.
Apr 15, 20:56 EDT
Identified -
Dear Customers,
We continue to investigate the widespread outage at the GUBOV data centre. We have identified the underlying issue and are currently working to validate and implement corrective actions. Our teams are actively engaged in restoring services as quickly as possible. Service Desk operations may still experience delays, and response times for new and existing tickets may be impacted during this time.
Next Update
We will provide further updates as more information becomes available.
We appreciate your continued patience and understanding.
Apr 15, 20:23 EDT
Update -
Dear Customers,
We continue to investigate the widespread outage at the GUBOV data centre. At this time, we believe the root cause has been identified and are currently working to confirm our findings. Our on-site technicians and networking engineers remain actively engaged in validating the issue and determining the appropriate remediation steps.
Service Desk operations may still experience delays, and response times for new and existing tickets may be impacted during this time.
Next Update
We will provide further updates as more information becomes available.
We appreciate your continued patience and understanding.
Apr 15, 20:12 EDT
Update -
Dear Customers,
We continue to investigate the widespread outage at the GUBOV data centre. Our on-site technicians are actively assessing the physical hardware, while our networking engineers are working in tandem to troubleshoot any potential hardware or network-related issues.
Service Desk operations may still experience delays, and response times for new and existing tickets may be impacted during this time.
Next Update
We will provide further updates as more information becomes available.
We appreciate your continued patience and understanding.
Apr 15, 19:44 EDT
Update -
Dear Customers,
We continue to investigate the widespread outage at the GUBOV data centre. Our technicians are currently on-site and actively investigating the underlying physical hardware. Service Desk operations may still experience delays, and response times for new and existing tickets may be impacted during this time.
Next Update
We will provide further updates as more information becomes available.
We appreciate your continued patience and understanding.
Apr 15, 19:24 EDT
Investigating -
Dear Customers,
We are currently experiencing a widespread outage at the GUBOV data centre, which is also impacting our Service Desk operations.
As a result, response times for both new and existing tickets may be delayed while our teams work to manage and resolve the situation.
Next Update
We are actively investigating the issue and will provide further updates as more information becomes available.
We appreciate your patience and understanding.
Apr 15, 18:42 EDT