DoubleCloud helped Honeybadger boost analytics performance

Honeybadger.io, founded in 2012, is a company that provides developers with tools to monitor errors, performance issues, and the uptime of websites and applications. Honeybadger is dedicated to offering tools that are simple yet powerful, helping developers diagnose and fix problems with their web applications quickly and efficiently.

Honeybadger’s mission is to build the kind of tools they themselves would want to use, focusing on simplicity, developer experience, and customer service. They pride themselves on being a developer-focused company, with a commitment to transparency, honesty, and maintaining a sustainable business that respects its customers and employees.

In the process of developing their new Insights service, Honeybadger’s team faced the challenge of handling a substantial amount of event and log data at high speeds. Insights is designed to provide in-depth application health monitoring, requiring efficient processing of vast quantities of data.

They needed a data infrastructure that could efficiently handle the data that their customers were sending up and maintain high performance. However, their existing database solutions, such as PostgreSQL and Elasticsearch, were not up to the task. They were unable to handle the complex queries with a low response time without significant pre-aggregation.

In their quest for a more efficient technology, Honeybadger turned to ClickHouse. What drew them to it was ClickHouse’s focus on performance for analytical queries, which was particularly intriguing for them given the nature of their data.

The team set out to explore several platforms, including other managed offerings like DoubleCloud and ClickHouse Cloud. After running pilots on both DoubleCloud and ClickHouse Cloud, they found that DoubleCloud provided a more straightforward approach to cluster management and data migration, along with more reasonable pricing, which were considerable challenges with their previous solution.

The decision to choose DoubleCloud was also influenced by the fact that part of the DoubleCloud team had developed the first managed service for ClickHouse at Yandex, showcasing their deep expertise and familiarity with the technology. Honeybadger appreciated that DoubleCloud offered a “just-add-water” ClickHouse service that could be provisioned with just a few clicks.

The data pipeline was structured as follows: Logs are collected through a straightforward HTTPS API, which takes in JSON payloads, adds necessary tenant information, and pushes them to a Kafka topic. Then Go language formatter rearranges this data to fit the ClickHouse format. The formatted data is sent in batches to the Managed ClickHouse service. Concurrently, Honeybadger’s Rails application also sends error and user activity logs to the Kafka topic. This system allows Honeybadgers’ dashboards to show the data stored in ClickHouse in an easy-to-understand format, providing valuable analytics to their users.