On October 26th between 10:00 and 14:30 CEST the LCS Platform experienced unexpected network errors and slow restart times. The network errors led to servers disconnecting from the LCS Platform, and displaying a warning to admin users.
Once the network issues were addressed, servers began reconnecting and warning messages were no longer shown to admin users. By the end of the incident, functionality has been fully restored.
After analyzing the incident, we found that we need to expand monitoring to detect earlier these types of issues, that are not yet covered by existing tools. To that end, the LCS team will create additional, custom monitoring methods, to better detect malfunctions such as this one.