The incident has been resolved

We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring the results.

TTBR for a single partition in the region has been growing uncontrollably. Current over 40 minutes. The impact is that when a user runs a query, any data bound for that partition will not be available. This may appear to them as dataloss, but the data is safe in kafka and will get written.

We suspect that the issue is that a user had their rate limits for writes increased and that user is targeting a single series. We have reduced their rate limits to normal, but this did not solve the problem as expected.

Recently discovered TTBR climbing on Prod01 due to spike in write. We are currently investigating the issue.