Neon increases data granularity with Managed ClickHouse

Neon has established itself as a key player in the cloud database market, particularly with its serverless PostgreSQL service. The company’s rise in the industry is backed by a significant investment of $104 million to date, including a $46 million Series B funding round led by Menlo Ventures. This level of financial backing underscores the confidence investors have in Neon’s vision and technology.

The team at Neon is notable for its composition of renowned PostgreSQL contributors and technologists. This collective expertise has been pivotal in developing Neon’s cloud-native database service that is specifically tailored for modern workloads, including edge computing and AI. Their approach to Postgres, focusing on serverless architecture and innovative features such as code-like database branching and automatic scale-to-zero for idle databases, positions Neon at the forefront of database technology development.

Neon’s partnership with its previous provider posed scalability issues as the number of users and their instances grew.

The primary pricing model for Neon’s managed service platform is consumption-based, along with a free tier with resource usage limits. In order to function properly, Neon’s billing system requires continuous monitoring of its resource consumption metrics, based on which the users are charged accordingly.

As the number of Neon’s customers grew along with projects and instances, the volume of metrics data also increased significantly. This problem led to a spike in data storage costs and a substantial increase in the time it takes to execute queries for aggregated metrics.

Daniil Efimov, the Software Engineer in charge of Neon’s billing system, researched time-series databases in the market to find a solution.

ClickHouse stood out for its data compression capabilities, fast visualization, and built-in connectors to data sources, meeting Daniil’s primary criteria for choosing a database to store consumption metrics.

Neon’s primary use case for ClickHouse by DoubleCloud involves storing a wide range of indicators with a high level of granularity. Metrics such as project storage, compute time, outgoing traffic, and the volume of written data are meticulously tracked. For instance, compute time is monitored every minute, while other metrics are tracked less frequently. Message services, particularly Vector, are used to transmit these metrics directly to ClickHouse and S3 storage. This use of hybrid storage solves two key tasks:

Task 1Task 1

ClickHouse allowed Neon to achieve much higher granularity of consumption metric storage, as well as an unmatched query processing speed. With this, Neon was able to implement new pricing plans with more complex settings and limits.

Aggregated up-to-date data in ClickHouse storage is used to monitor limits set by pricing plans. For example, if a free tier instance exceeds the maximum duration of 100 hours and 3 GB, billing automatically shuts down the instance upon reaching these limits.

Task 2Task 2

Additionally, thanks to the ability to store data with greater granularity, the service can provide its clients with extensive resource consumption analytics.

The integration of DoubleCloud’s managed ClickHouse service with Neon has significantly transformed the way Neon handles data management.

Presently, Neon stores aggregated metrics in ClickHouse for two months, encompassing all clients, projects, and instances. This amounts to approximately 500 GB of data in compressed format and 5 TB in uncompressed format. To ensure the accuracy and effectiveness of pricing plan limits set by the billing system, an aggregated metric is requested every few minutes.

Looking ahead, Neon has charted out an ambitious roadmap to further harness the potential of ClickHouse. One of the primary objectives is to expand their data storage capacity. The plan involves storing data on solid-state drives (SSDs) for the two most recent months, and then transferring older data to S3 storage, with the ultimate aim of extending data storage to cover an entire year. This expansion is expected to enhance their analytics capabilities and enable more comprehensive monitoring of resource consumption. It will also help to implement consumption forecasts for customers, as well as implement new product features and even more diverse pricing plans.

Moreover, Neon intends to integrate Apache Superset and Snowflake with ClickHouse, broadening the scope of their analytics with aggregated data stored in ClickHouse. Additionally, thanks to the integration of ClickHouse with Grafana, Neon’s team is now able to build real-time dashboards and alerts on top of their billing data, for example, to track the latency and the number of billing events.

Finally, Neon is actively developing custom consumption reports. These reports are designed to provide customers with a more detailed and insightful view of their consumption. This ongoing innovation and expansion underscore Neon’s dedication to maintaining a leading position in data management technology and delivering state-of-the-art solutions to their clients.