The Enterprise NOC confirmed service restored after a Force Majeure event caused by vandalism. This will serve as the final notification for this event.
Posted Feb 01, 2024 - 05:06 PST
Monitoring
Our datacenter has come back online. We are monitoring for additional outages as they continue repairs.
Here is the latest from our service provider:
As of 2:32 AM Eastern Time, the Enterprise NOC continues to work with Construction teams. The 2nd 432 count fiber is being spliced currently at 360 count completed on the east side and 144 count on the west side remaining to be spliced. There is no estimated time of resolution. Additional updates will be provided as information becomes available.
Posted Feb 01, 2024 - 00:02 PST
Update
Latest from our service provider:
"As of 12:08 PM Eastern Time, the Enterprise NOC continues to work with construction teams. Currently Construction teams have reported fiber techs have completed splicing 432 fibers at the west end and 360 fibers have been spliced on the east end on the 1st 432ct fiber. At this time fiber techs have also completed splicing 105 fiber on the east end and 72 fibers on the west end, on the 2nd 432 ct. Currently there is no estimated time of resolution, additional updates will be provided as information becomes available."
Posted Jan 31, 2024 - 21:50 PST
Update
Latest from our service provider:
"As of 11:08 PM Eastern Time, construction reports fiber techs have completed splicing 288 fibers at the west end and 360 fibers have been spliced on the east end, on the 1st 432ct fiber. Fiber techs have completed splicing 72 fiber on the east end and 36 fibers on the west end, on the 2nd 432 ct. ETR remains 1:00 AM EST. Further updates will be provided as information becomes available."
Posted Jan 31, 2024 - 20:30 PST
Update
Latest from our service provider:
"As of 6:24 PM Pacific Time, construction reports fiber techs have completed splicing 144 fibers on the 1st 432ct fiber. Splicers continue to prep the tails of the 2nd 432 ct. fiber. ETR remains 12:00 AM Central time. Further updates will be provided as information becomes available."
Posted Jan 31, 2024 - 18:38 PST
Update
Latest from our service provider:
"As of 3:20 PM Pacific Time, construction reports fiber techs are working to place the new fiber and have begun prepping both sides of the other fiber bundle. ETR remains 12:00 AM Central. Further updates will be provided as information becomes available."
Posted Jan 31, 2024 - 15:44 PST
Update
Latest from our service provider:
"At 3:11 PM Central Time, the Construction team continues to pull fiber through conduit. The current ETR remains at 12:00 AM Central Time. Further updates will be provided as information becomes available."
Posted Jan 31, 2024 - 13:34 PST
Update
Latest from our service provider:
Construction team continues to clear and pull fiber while digging for conduit on the west side of the damaged area. The current ETR has been set at 12:00 AM Central Time. Further updates will be provided as information becomes available.
Posted Jan 31, 2024 - 12:50 PST
Update
Our service provider has given us another status update:
"Construction team continues to clear out remaining fiber from existing conduit to place new fiber. The Fiber splicers are prepping existing enclosures. No current ETR. Further updates will be provided as information becomes available."
Posted Jan 31, 2024 - 11:53 PST
Update
Our ISP has informed us that they've located the damaged fiber lines, their construction team has begun to wreck out the remaining fiber and the splicers have arrived on site. Additional updates will be provided as they become available.
Posted Jan 31, 2024 - 10:38 PST
Identified
Latest update from the ISP is that "Construction teams are en-route. No ETA at this time. This outage has been escalated to VP level. Additional updates will be provided as they become available."
Posted Jan 31, 2024 - 10:22 PST
Investigating
At 7:15AM PST, our monitoring alerted us that our entire datacenter had gone offline. On-site technicians performed an initial diagnosis and determined that the issue was with our ISP, not our own hardware. We opened a ticket with the service provider, who has confirmed the outage and is working to resolve it.
They have not given us an ETA on restoration of service yet, but as soon as they do we will provide an update.
Posted Jan 31, 2024 - 08:00 PST
This incident affected: QA (Testray), Web, Customer, Files, and Internal (Dossiera, EVP, Patcher).