Background

InfluxDB stores data in TSM files. Within these files, series are identified by series key, which is a combination of tag values, field names, measurement, organization and bucket id. When processes such as compactions or queries need to access a series, they read the series key from the TSM file(s). There is an enforced limit of 64 KB for a series key, but few series keys ever reach close to this limit.

For each series, the TSM files store:

1. The key size in bytes
2. The key itself
3. The data for that series

The reader code uses the key size (stored in the first two bytes as a uint16) and then uses that to read in the bytes from the series key itself. The reader then reads in the contiguous bytes based on that number, but offset by 2 in order to account for the key size data .

InfluxDB has a design principle to return a 500 error if there is ever a risk of returning wrong or incomplete data without the user being aware that the data may be wrong or incomplete.

For redundancy and other reasons, InfluxDB Cloud implements a Kafka queue in front of writes, that can store unsuccessful writes for long periods of time.

Trigger

A user happened to upload data which through coincidence had a series key which was exactly 64 KB in length. The code to read, in pseudocode is such as:

read(start: 2, end: 2 + key_size)

However, since the ensuing value was stored in a uint16, the value wrapped to size 1. This resulted in a function such as:
read(start 2: end: 1)

This code panicked because the end point was before the start point, causing a restart loop in the storage engine. While this only impacted a single TSM file, the system, as designed, returned errors rather than potentially returning incomplete query results. Additionally, the Kafka queue ensured that writes were safe while the impacted storage pods were unavailable.

Customer Impact

Subscribers in the impacted region faced read interruptions during this incident. Writes were queued for some buckets, but quickly recovered as service was restored.

Mitigation

The following pseudocode has been modified:

2 + key_size

Became

2 + int(key_size)

We have ensured that this mitigation has been applied consistently throughout the codebase.