📢 Upcoming webinar | Using ClickHouse for real-time analytics Register now →

Neon increases data granularity with Managed ClickHouse

Highlights

Improved granularity for metrics:
from daily to 15-minute timestamps

Significant (up to 6x) increase in 
database query speed

New pricing plans with specific consumption limits

Thanks to the expertise of DoubleCloud’s team, we rapidly implemented ClickHouse. This architectural solution enabled a fundamentally new approach to pricing based on resource consumption, including customers with thousands of instances. It’s a solid foundation for our future projects
Daniil Efimov
Software Engineer, Neon

About Neon

Neon has established itself as a key player in the cloud database market, particularly with its serverless PostgreSQL service. The company’s rise in the industry is backed by a significant investment of $104 million to date, including a $46 million Series B funding round led by Menlo Ventures. This level of financial backing underscores the confidence investors have in Neon’s vision and technology.

The team at Neon is notable for its composition of renowned PostgreSQL contributors and technologists. This collective expertise has been pivotal in developing Neon’s cloud-native database service that is specifically tailored for modern workloads, including edge computing and AI. Their approach to Postgres, focusing on serverless architecture and innovative features such as code-like database branching and automatic scale-to-zero for idle databases, positions Neon at the forefront of database technology development.

The challenge

Neon’s partnership with its previous provider posed scalability issues as the number of users and their instances grew.

The primary pricing model for Neon’s managed service platform is consumption-based, along with a free tier with resource usage limits. In order to function properly, Neon’s billing system requires continuous monitoring of its resource consumption metrics, based on which the users are charged accordingly.

As the number of Neon’s customers grew along with projects and instances, the volume of metrics data also increased significantly. This problem led to a spike in data storage costs and a substantial increase in the time it takes to execute queries for aggregated metrics.

Daniil Efimov, the Software Engineer in charge of Neon’s billing system, researched time-series databases in the market to find a solution.

ClickHouse stood out for its data compression capabilities, fast visualization, and built-in connectors to data sources, meeting Daniil’s primary criteria for choosing a database to store consumption metrics.

The solution

Neon’s primary use case for ClickHouse by DoubleCloud involves storing a wide range of indicators with a high level of granularity. Metrics such as project storage, compute time, outgoing traffic, and the volume of written data are meticulously tracked. For instance, compute time is monitored every minute, while other metrics are tracked less frequently. Message services, particularly Vector, are used to transmit these metrics directly to ClickHouse and S3 storage. This use of hybrid storage solves two key tasks:

Task 1

ClickHouse allowed Neon to achieve much higher granularity of consumption metric storage, as well as an unmatched query processing speed. With this, Neon was able to implement new pricing plans with more complex settings and limits.

Aggregated up-to-date data in ClickHouse storage is used to monitor limits set by pricing plans. For example, if a free tier instance exceeds the maximum duration of 100 hours and 3 GB, billing automatically shuts down the instance upon reaching these limits.

Task 2

Additionally, thanks to the ability to store data with greater granularity, the service can provide its clients with extensive resource consumption analytics.

Neon’s data pipelne

The data pipeline integrates PostgreSQL, Vector, and ClickHouse. Data collected through Vector is stored in both ClickHouse for immediate processing and S3 for longer-term storage. This setup enables efficient data aggregation and querying, crucial for real-time billing and pricing plan management.

The outcome

The integration of DoubleCloud’s managed ClickHouse service with Neon has significantly transformed the way Neon handles data management.

Presently, Neon stores aggregated metrics in ClickHouse for two months, encompassing all clients, projects, and instances. This amounts to approximately 500 GB of data in compressed format and 5 TB in uncompressed format. To ensure the accuracy and effectiveness of pricing plan limits set by the billing system, an aggregated metric is requested every few minutes.

Looking ahead, Neon has charted out an ambitious roadmap to further harness the potential of ClickHouse. One of the primary objectives is to expand their data storage capacity. The plan involves storing data on solid-state drives (SSDs) for the two most recent months, and then transferring older data to S3 storage, with the ultimate aim of extending data storage to cover an entire year. This expansion is expected to enhance their analytics capabilities and enable more comprehensive monitoring of resource consumption. It will also help to implement consumption forecasts for customers, as well as implement new product features and even more diverse pricing plans.

Moreover, Neon intends to integrate Apache Superset and Snowflake with ClickHouse, broadening the scope of their analytics with aggregated data stored in ClickHouse. Additionally, thanks to the integration of ClickHouse with Grafana, Neon’s team is now able to build real-time dashboards and alerts on top of their billing data, for example, to track the latency and the number of billing events.

Finally, Neon is actively developing custom consumption reports. These reports are designed to provide customers with a more detailed and insightful view of their consumption. This ongoing innovation and expansion underscore Neon’s dedication to maintaining a leading position in data management technology and delivering state-of-the-art solutions to their clients.

Start building your data infrastructure today with our free trial

or contact us, and solution architects will help you with the request.

ClickHouse® is a trademark of ClickHouse, Inc. https://clickhouse.com