Traffic monetization platform migrates to Managed ClickHouse

OverviewOverview

A company runs an intelligent traffic monetization platform, working in online advertising.  It serves as a link between publishers and advertisers to buy and sell Ad traffic worldwide.

This platform is the company’s cornerstone and the key competitive advantage. Its performance and robustness is absolutely essential for the business.

Today the platform has more than 5000 advertisers, and the number is growing organically by 7-12% a month.

Problems, challenges, and migration objectivesProblems, challenges, and migration objectives

Growth-related challengesGrowth-related challenges

The company relied on services running from two data centers in Europe and the U.S. on on-prem hardware to reach partners from more than 200 countries.

The processed traffic generated more than 5 billion records per day. Every month, the traffic grew by 5-7%.

Technical challengesTechnical challenges

The company constantly needed to build up technical capacity, thus increasing support costs of the on-premise ClickHouse cluster hardware.

Also, in direct proportion to the scale of their business, hardware malfunctions kept piling up until they started affecting the end-user experience.

Migration objectivesMigration objectives

• Increase the stability of the cluster.
• Reduce support costs and infrastructure development to focus directly on the product migration and testing
• Alleviate growing maintenance workload by migrating to Managed ClickHouse.

Service testingService testing

The whole testing process was straightforward. The team quickly set up a cluster and used it to see how their platform performs under extreme circumstances:

• Read-write cycle reliability test during recurring load spikes.
• The service was placed in a stress test situation where the volume of records increased significantly after a pause to ensure it could catch up with the write-up.
• Validation of the service’s SaaS functions by performing end-to-end functionality tests.

Project deploymentProject deployment

Keeping in mind the on-premise nature of the initial project architecture, we developed a three-stage deployment sequence to ensure seamless migration that would have zero impact on end-user experience:

1. The team created an organization and project structure with clusters, transfers and Visualization dashboards from our web console.
2. After a testing period, the team used the data Transfer capabilities of our service to migrate all the relevant data from the on-premise storage into the cloud.
3. All the subsystems within the traffic monetization service were completely switched to working with the new cloud-based storage and monitoring solution.
4. The whole process is as easy as a few mouse clicks for the team to execute.

Key resultsKey results

This project created a new understanding of cloud-based Managed ClickHouse benefits:

• Reduced time and workload for cluster support. The team almost eradicated the time taken on cluster support and also noted an increase in their cluster stability due to the migration to the cloud, resulting in a huge cost saving.
• Cluster updates with minimal downtime. it is now possible to perform regular updates of their clusters to new versions with minimal downtime and in once case the team had a service outage for some time — however all the data in the cloud remained intact.
• Web-based console workflow makes mundane tasks easy.

Through the easy to use web interface, the company quickly increased quotas, expanded storage space and expanded the services with flexibility and reliability on a different level.