Update - The amount of time taken for writes to become readable has returned to the normal operating level. We will continue to monitor and update.
Jan 26, 2023 - 22:33 UTC
Jan 26, 2023 - 22:33 UTC
Update - We are continuing to monitor for any further issues.
Jan 26, 2023 - 19:53 UTC
Jan 26, 2023 - 19:53 UTC
Update - The region is currently experiencing elevated times for written information to become queryable along with elevated query run times. All written information is still being safely queued. We are working to minimize disruptions and we will continue to update as the situation evolves.
Jan 26, 2023 - 18:38 UTC
Jan 26, 2023 - 18:38 UTC
Monitoring - The storage engine has completed processing the bursted traffic. Writes are now queryable within the normal time ranges. Our team will continue to monitor.
Jan 26, 2023 - 03:06 UTC
Jan 26, 2023 - 03:06 UTC
Identified - The region received another burst of write traffic causing backup into the queueing system. Heavily bursting traffic has now been limited on a per organization basis. All organizations impacted by this limitation have been notified. The storage system is currently processing the bursted traffic. We will update here as the storage engine completes this processing.
Jan 26, 2023 - 02:29 UTC
Jan 26, 2023 - 02:29 UTC
Investigating - We are seeing a return of the delayed reads and writes issue from a few hours ago. Our team is actively investigating.
Jan 26, 2023 - 01:21 UTC
Jan 26, 2023 - 01:21 UTC
Monitoring - We have identified the primary contributing factor for performance degradation (delayed writes and subsequent temporary read discrepancies). The impacted region has been receiving a large burst of writes twice daily which has saturated the storage layer. All writes to this region have been successfully recorded within our queuing system but are taking much longer than expected to become queryable. In response to this event, we have increased the resources available to the storage system. In addition, we have paused all tertiary background processes in the region to expedite recovery. We will continue to provide updates as the storage system recovers and will provide a complete root cause analysis when this incident has been resolved.
Jan 25, 2023 - 19:53 UTC
Jan 25, 2023 - 19:53 UTC
Update - We are continuing to work on a fix for this issue.
Jan 25, 2023 - 18:12 UTC
Jan 25, 2023 - 18:12 UTC
Update - We are continuing to work on a fix for this issue.
Jan 25, 2023 - 16:06 UTC
Jan 25, 2023 - 16:06 UTC
Update - We are continuing to work on a fix for this issue.
Jan 25, 2023 - 14:45 UTC
Jan 25, 2023 - 14:45 UTC
Update - We are continuing to work on a fix for this issue.
Jan 25, 2023 - 13:50 UTC
Jan 25, 2023 - 13:50 UTC
Update - We have deployed some minor updates to the cluster but still continue to investigate the issue
Jan 25, 2023 - 09:29 UTC
Jan 25, 2023 - 09:29 UTC
Update - We are continuing to work on a fix for this issue.
Jan 25, 2023 - 07:32 UTC
Jan 25, 2023 - 07:32 UTC
Update - We are continuing to work on a fix for this issue.
Jan 25, 2023 - 05:53 UTC
Jan 25, 2023 - 05:53 UTC
Update - We are continuing to work on a fix for this issue.
Jan 25, 2023 - 03:44 UTC
Jan 25, 2023 - 03:44 UTC
Update - We are continuing to work on a fix for this issue.
Jan 25, 2023 - 01:25 UTC
Jan 25, 2023 - 01:25 UTC
Update - We are continuing to work on a fix for this issue.
Jan 24, 2023 - 23:43 UTC
Jan 24, 2023 - 23:43 UTC
Update - We are continuing to work on this issue and a fix is being implemented.
Jan 24, 2023 - 21:00 UTC
Jan 24, 2023 - 21:00 UTC
Update - The issue has been identified and a fix is still being implemented.
Jan 24, 2023 - 19:50 UTC
Jan 24, 2023 - 19:50 UTC
Identified - The issue has been identified and a fix is being implemented.
Jan 24, 2023 - 17:41 UTC
Jan 24, 2023 - 17:41 UTC
Update - The AWS regions specified are experiencing delayed write/query operations and intermittent query failures.
We are continuing to investigate the issue.
Jan 24, 2023 - 15:57 UTC
We are continuing to investigate the issue.
Jan 24, 2023 - 15:57 UTC
Update - We are continuing to investigate this issue.
Jan 24, 2023 - 13:50 UTC
Jan 24, 2023 - 13:50 UTC
Investigating - We are currently investigating this issue.
Jan 24, 2023 - 13:11 UTC
Jan 24, 2023 - 13:11 UTC
Google Cloud: Iowa, US-Central
Operational
Web UI
?
Operational
API Writes
Operational
API Queries
Operational
Tasks
Operational
Persistent Storage
Operational
Compute
Operational
Other
?
Operational
Google Cloud: Belgium, Europe-West-1
Operational
Web UI
Operational
API Writes
Operational
API Queries
Operational
Tasks
Operational
Persistent Storage
Operational
Compute
Operational
Other
?
Operational
AWS: Frankfurt, EU-Central-1
Degraded Performance
Web UI
Operational
API Writes
Degraded Performance
API Queries
Degraded Performance
Tasks
Operational
Persistent Storage
Operational
Compute
Operational
Other
?
Operational
AWS: Oregon, US-West-2-1
Operational
Web UI
Operational
API Writes
Operational
API Queries
Operational
Tasks
Operational
Persistent Storage
Operational
Compute
Operational
Other
?
Operational
AWS: Oregon, US-West-2-2
Operational
Web UI
Operational
API Writes
Operational
API Queries
Operational
Tasks
Operational
Persistent Storage
Operational
Compute
Operational
Other
?
Operational
AWS: Virginia, US-East-1
Operational
Web UI
Operational
API Writes
Operational
API Queries
Operational
Tasks
Operational
Persistent Storage
Operational
Compute
Operational
Other
?
Operational
Azure: Amsterdam, West Europe
Operational
Web UI
Operational
API Writes
Operational
API Queries
Operational
Tasks
Operational
Persistent Storage
Operational
Compute
Operational
Other
?
Operational
Azure: Virginia, East US
Operational
Web UI
Operational
API Writes
Operational
API Queries
Operational
Tasks
Operational
Persistent Storage
Operational
Compute
Operational
Other
?
Operational
AWS: Sydney, AP-Southeast-2
Operational
Web UI
Operational
API Writes
Operational
API Queries
Operational
Tasks
Operational
Persistent Storage
Operational
Compute
Operational
Other
?
Operational
Other Services
?
Operational
Web UI Authentication (Auth0)
Operational
Marketplace integrations
?
Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.