Loja Integrada & Pagali switched to DoubleCloud for processing a massive amount of financial data

DoubleCloud recently had the pleasure of speaking with Udlei Nati, the original founder and CTO of Loja Integrada, an e-commerce platform based in Brazil.

Loja Integrada was founded in 2013 and is Brazil’s largest e-commerce platform, supporting and empowering a community of small businesses and sole traders to sell their products online. It is a platform that is constantly growing and updating, realizing a huge surge in the number of sellers and buyers during the pandemic.

About 40,000 new stores are created every month, alongside with over 1300 agencies and partners in their ecosystem.

Pagali is a new business unit created by Loja Integrada to complement their growing platform with a payment gateway that is similar to PayPal. Pagali is already one of the biggest payment providers on Loja Integrada, as it makes the payment process much more efficient and profitable for all parties involved.

Pagali currently employs Udlei, and seven other people, including a front-end developer, two back-end developers, and a designer. Pagali processes 45% of all Loja Integrada’s transactions, needing an efficient and lightning-fast storage and means to analyze for all those transactions.

As Loja Integrada grew very quickly, the volume of data also increased significantly, and the Pagali team began to face limitations with PostgreSQL. These include reduced performance and periodic crashes when executing queries. However, migrating to Amazon Aurora, which is an alternative to PostgreSQL and was created by the AWS team, solved the performance problem, but the Pagali team faced new business needs.

Firstly, one of the frequent tasks was analyzing transactions to fight fraud. This required having appropriate dashboards that can handle transaction queries up to 90 days deep and quickly obtain all necessary data for investigation. Secondly, for business development, it is very important to have a number of analytical reports with operational metrics for management.

Pagali can use the resources of the BI team, which exists in the parent company, Loja Integrada. However, this would significantly increase the development time for BI and would also limit the further development and evolution of these reports under the team’s needs.

The team led by Udlei Nati knew the ClickHouse technology, so there was no doubt about its performance for storing a large volume of financial data. The decisive factor for the team was that DoubleCloud had all the necessary services for their business tasks, which could combine into a custom solution like Lego blocks.

Firstly, a cloud cluster of ClickHouse was deployed, managed by the DoubleCloud service. Then data transmission was configured using Data Transfer connectors for PostgreSQL and Data Transfer for Amazon Aurora.

In the final stage, the team developed dashboards for transaction analysis and operational monitoring. A notable detail of their solution was that it was necessarily completed by the team without consulting Loja Integrada. Also, it is an impressive improvement compared to the previous data infrastructure with successes including:

1. Performance increased six times.

2. The size of the database was reduced by half.

3. The cost of ownership was also reduced by half.