­čôó Upcoming webinar | Why use Terraform to stream data with Apache Kafka? Register now →

Loja Integrada & Pagali switched to DoubleCloud for processing a massive amount of financial data

Highlights

Saving 6 hours per week previously spent optimizing PostgreSQL. In some cases it used to take a whole week to update analytics!

Financial data required for investigating fraud can now be retained six times longer — up to 90 days.

The cost of the solution on the DoubleCloud platform is 2x cheaper than the cost of the previous solution based on Amazon RDS.

Made the entire hosting, maintenance, and synchronization process uncomplicated. We are very happy with the solution and will extend it to other operations.
Udlei Nati,
CTO, Pagali

About Loja Integrada & Pagali

DoubleCloud recently had the pleasure of speaking with Udlei Nati, the original founder and CTO of Loja Integrada, an e-commerce platform based in Brazil.

Loja Integrada was founded in 2013 and is Brazil’s largest e-commerce platform, supporting and empowering a community of small businesses and sole traders to sell their products online. It is a platform that is constantly growing and updating, realizing a huge surge in the number of sellers and buyers during the pandemic.

About 40,000 new stores are created every month, alongside with over 1300 agencies and partners in their ecosystem.

Pagali is a new business unit created by Loja Integrada to complement their growing platform with a payment gateway that is similar to PayPal. Pagali is already one of the biggest payment providers on Loja Integrada, as it makes the payment process much more efficient and profitable for all parties involved.

Pagali currently employs Udlei, and seven other people, including a front-end developer, two back-end developers, and a designer. Pagali processes 45% of all Loja Integrada’s transactions, needing an efficient and lightning-fast storage and means to analyze for all those transactions.

The challenge

As Loja Integrada grew very quickly, the volume of data also increased significantly, and the Pagali team began to face limitations with PostgreSQL. These include reduced performance and periodic crashes when executing queries. However, migrating to Amazon Aurora, which is an alternative to PostgreSQL and was created by the AWS team, solved the performance problem, but the Pagali team faced new business needs.

Firstly, one of the frequent tasks was analyzing transactions to fight fraud. This required having appropriate dashboards that can handle transaction queries up to 90 days deep and quickly obtain all necessary data for investigation. Secondly, for business development, it is very important to have a number of analytical reports with operational metrics for management.

Pagali can use the resources of the BI team, which exists in the parent company, Loja Integrada. However, this would significantly increase the development time for BI and would also limit the further development and evolution of these reports under the team’s needs.

The solution

The team led by Udlei Nati knew the ClickHouse technology, so there was no doubt about its performance for storing a large volume of financial data. The decisive factor for the team was that DoubleCloud had all the necessary services for their business tasks, which could combine into a custom solution like Lego blocks.

Firstly, a cloud cluster of ClickHouse was deployed, managed by the DoubleCloud service. Then data transmission was configured using Data Transfer connectors for PostgreSQL and Data Transfer for Amazon Aurora.

In the final stage, the team developed dashboards for transaction analysis and operational monitoring. A notable detail of their solution was that it was necessarily completed by the team without consulting Loja Integrada. Also, it is an impressive improvement compared to the previous data infrastructure with successes including:

  1. Performance increased six times.

  2. The size of the database was reduced by half.

  3. The cost of ownership was also reduced by half.

Pagali’s architecture:

Finally, the Pagali team felt a great deal of synergy with DoubleCloud. They are both start-ups with fast and efficient processes, building a reputation for great customer service while improving their respective platforms.

We can now make faster & immediate decisions thanks to the availability of faster and more granular metrics. Now we understand better what is happening in our business
Udlei Nati,
CTO, Pagali

Get started with DoubleCloud

ClickHouse® is a trademark of ClickHouse, Inc. https://clickhouse.com