Background

• InfluxDB Cloud runs on Kubernetes, a cloud application orchestration platform.
• InfluxData uses an automated Continuous Delivery system to deploy both code and configuration changes to production. On a typical work day, the engineering team delivers from 5 to 15 different changes to production.
• The team uses a tool called “Argo CD”, developed by Intuit, to deploy code and configuration changes to Kubernetes clusters. Argo CD does this by reading a YAML configuration file, and then uses the Kubernetes API to make the cluster look consistent with what is specified in the YAML.
• In order to maintain multiple clusters, a language called jsonnet is used to templatize the configuration. The CD system detects changes in the jsonnet, converts the jsonnet into YAML, and then Argo applies the changes.
• There is a code review process in place to inspect the resulting YAML and ensure that it will do what is expected before it is applied.

Trigger

In this particular case, the development team applied a change intended to add an “application” to the aws-eu-central cluster. In this context an “application” can be thought of as a service or system component that contributes functionality to the overall InfluxDB Cloud application. The purpose of the additional service was to add the ability to collect performance and correctness data on the new IOx storage engine.

When the developers added the application, they mistyped the name field, using “idpe” instead of “IOx” for the name. “Idpe” being the name for the larger application in which most of the InfluxDB Cloud 2 services run.

In the jsonnet, the application name should have been defined like this:

     ApplicationName: 'iox-cd-aws-prod01-eu-central-1',

But was defined like this:

     ApplicationName: 'idpe-cd-aws-prod01-eu-central-1',

Argo CD interpreted the re-use of the application name as redefinition of the application. Because the new configuration was added below the existing functionality in the file, Argo CD deleted the existing application and reinstalled the new one. 

This, essentially, removed InfluxDB Cloud 2 from production.

Contributing Factors

1. The rendered YAML appeared to be adding functionality to the existing application; it was not clear that it was redefining it.
2. Automated tests that run in order to detect such problems were misconfigured and were not running in the location where this change was made.
3. Argo CD makes such changes with no warning or failsafe in place.

Customer Impact

1. All API endpoints, including all writes and reads, returned 404 or equivalent. No data was collected during this time.
2. No tasks ran during the outage and all scheduled task runs during the outage failed.
3. Data could not be queried externally.

While new writes were not collected, no existing data was lost. All time series data, and also all configuration metadata, that was written before the incident was preserved.

Timeline

The following times are in UTC:

10:22: The PR was merged

10:25: InfluxData monitoring started reporting API failures

10:37: Support Team responded to customer escalations and started the incident response process

10:42: The PR was reverted

10:45: Engineering teams started to plan the recovery process

10:49: status.influxdata.com was updated to reflect the issue

10:56: All senior managers were engaged

11:17: Teams started to redeploy base services (starting with Kafka) carefully in order to connect them with existing volumes, preserve state, and accelerate recovery. Additional services were re-deployed in parallel as data integrity was verified. In some cases, services were restored from backup instead of redeploying, if this was deemed a safer strategy by the team. Expecting a surge in traffic once restored, ingress points were proactively scaled up.

15:26  The team enabled the write service, tested, and validated functionality. They then re-enabled tasks to allow the backlog of task runs to complete in order to avoid overloading the cluster. Once backlogged tasks had completed, they re-enabled the query service.

16:04: All reads, and therefore full functionality, is fully restored.

Future Mitigations

1. Simplify the directory structure of the jsonnet configuration codebase and confirm that all pre-deployment tests are running on it.
2. Enhance the code review process by reviewing a diff generated by an Argo CD dry run. Develop and contribute this functionality upstream if needed.
3. Review disaster recovery runbooks in light of what was learned during the incident.
4. Review “near misses” and ensure that our backup strategy is sufficient.
5. Implement “game days” to practice disaster recovery on a cadence in the internal production cluster.