DoubleCloud helped Honeybadger boost analytics performance

Highlights

HoneyBadger preferred DoubleCloud over ClickHouse Cloud for its simpler cluster management, easier data migration, and cost-effectiveness

Launching new product features with consistently high performance

The power of DoubleCloud lies in its simplicity. It’s a just-add-water managed ClickHouse service that can be provisioned in a few clicks, saving us the hassle of setup and maintenance
Ben Curtis,
Co-Founder, Honeybadger

About Honeybadger

Honeybadger.io, founded in 2012, is a company that provides developers with tools to monitor errors, performance issues, and the uptime of websites and applications. Honeybadger is dedicated to offering tools that are simple yet powerful, helping developers diagnose and fix problems with their web applications quickly and efficiently.

Honeybadger’s mission is to build the kind of tools they themselves would want to use, focusing on simplicity, developer experience, and customer service. They pride themselves on being a developer-focused company, with a commitment to transparency, honesty, and maintaining a sustainable business that respects its customers and employees.

The challenge

In the process of developing their new Insights service, Honeybadger’s team faced the challenge of handling a substantial amount of event and log data at high speeds. Insights is designed to provide in-depth application health monitoring, requiring efficient processing of vast quantities of data.

They needed a data infrastructure that could efficiently handle the data that their customers were sending up and maintain high performance. However, their existing database solutions, such as PostgreSQL and Elasticsearch, were not up to the task. They were unable to handle the complex queries with a low response time without significant pre-aggregation.

The solution

In their quest for a more efficient technology, Honeybadger turned to ClickHouse. What drew them to it was ClickHouse’s focus on performance for analytical queries, which was particularly intriguing for them given the nature of their data.

The team set out to explore several platforms, including other managed offerings like DoubleCloud and ClickHouse Cloud. After running pilots on both DoubleCloud and ClickHouse Cloud, they found that DoubleCloud provided a more straightforward approach to cluster management and data migration, along with more reasonable pricing, which were considerable challenges with their previous solution.

The decision to choose DoubleCloud was also influenced by the fact that part of the DoubleCloud team had developed the first managed service for ClickHouse at Yandex, showcasing their deep expertise and familiarity with the technology. Honeybadger appreciated that DoubleCloud offered a “just-add-water” ClickHouse service that could be provisioned with just a few clicks.

The data pipeline was structured as follows: Logs are collected through a straightforward HTTPS API, which takes in JSON payloads, adds necessary tenant information, and pushes them to a Kafka topic. Then Go language formatter rearranges this data to fit the ClickHouse format. The formatted data is sent in batches to the Managed ClickHouse service. Concurrently, Honeybadger’s Rails application also sends error and user activity logs to the Kafka topic. This system allows Honeybadgers’ dashboards to show the data stored in ClickHouse in an easy-to-understand format, providing valuable analytics to their users.

Honeybadger’s data pipelne

The outcome

The implementation of Managed ClickHouse by DoubleCloud had a significant impact on Honeybadger’s operations. It broadened the scope of projects they are willing to take on and enabled them to analyze larger amounts of data more efficiently.

The enhanced performance also enabled Honeybadger to add new data slices to their charts and diagrams.

Moreover, the improved query performance has led to brainstorming of innovative cross-project ideas that could provide richer data insights. The team is excited about the potential innovations that improved query performance could lead to.

Looking forward, Honeybadger is excited about the prospects of completely replacing their Elasticsearch cluster with ClickHouse, which would further improve their service offering. They are aiming to be production-ready with their integration of ClickHouse into Honeybadger via their new Insights module in Q1 2024.

Start building your data infrastructure today with our free trial

or contact us, and solution architects will help you with the request.

ClickHouse® is a trademark of ClickHouse, Inc. https://clickhouse.com