» Published on Mon, 10 Oct 2022 10:14:00 +0000

• Resolved

  Current Status: Operational

  Affected Infrastructure Components: Website, Video playback, Search, Recording / Uploading, Processing / Encoding, Integrations Locations: Europe (EU)

  Update: On 2022-10-10 between 09:09 UTC and 10:37 UTC, the Panopto EU cloud experienced increased error rates and latency resulting in general website instability. During this time period, users may not have been able to perform operations such as logging in or editing content and there may have been a delay in new or updated content appearing in search results.

  RCA: Due to a usage pattern change, a frequently used database query began to use a suboptimal execution plan. This new execution plan caused excessive resource utilization, resulting in increased error rates as other queries were delayed and timed out. The Panopto operations team responded by removing the execution plan, which resolved the issue. Additionally, the time to resolution was delayed due to a gap in our escalation process that has since been closed.

  To prevent a recurrence of this issue, our engineering team has optimized the query to ensure better-performing execution plans are used.

  » Updated Wed, 26 Oct 2022 18:34:00 +0000

• Update

  Vendor notification:

  Current Status: Operational

  Affected Infrastructure Components: Website, Video playback, Search, Recording / Uploading, Processing / Encoding, Integrations Locations: Europe (EU)

  Update: The Panopto EU Cloud has been stable and metrics have remained normal since our last update.

  We are resolving this incident and Panopto engineers are completing a root cause analysis, which we will post here, within the next

  » Updated Mon, 10 Oct 2022 13:03:00 +0000

• Update

  Update: The source of the increased error rates has been mitigated. Our cloud operations team is monitoring cloud metrics to ensure the issue is resolved.

  Our next update will be within 60 minutes.

  » Updated Mon, 10 Oct 2022 10:46:00 +0000

• Investigating

  Vendor notification:

  Current Status: Degraded Performance

  Affected Infrastructure Components: Website, Video playback, Search, Recording / Uploading, Processing / Encoding, Integrations Locations: Europe (EU)

  Update: The Panopto EU Cloud is experiencing an increased error rate in general website usage.

  Our cloud operations team is urgently investigating the issue and we’ll update here as soon as we know more.

  » Updated Mon, 10 Oct 2022 10:14:00 +0000