Code Refactoring & Data Engineering Architecture

Implementamos projetos de dados em diferentes clientes em diversos continentes

Refatoração de Código

Context:

Our client maintained a SaaS with legacy code written in PHP, where data engineering jobs were orchestrated with cron jobs. This made management impossible, created intolerance to failures, lacked retry mechanisms, and had primitive, hard-to-access log visualization.

Challenge:

The code had a high dependency on specific developers due to the PHP language, making maintenance difficult. It also presented constant failures in API integrations and made it hard to monitor script execution.

Solution:

We implemented Apache Airflow for data pipeline orchestration. All data pipelines were rewritten in Go, focusing on performance optimization, complete refactoring of the main modules, applying clean code best practices, and standardized documentation.

Several data transformations for report generation were refactored into SQL dialects to simplify analytics.

We also implemented GitHub Actions to automate CI/CD.

Results:

Visual monitoring of ETL pipelines

Retry policies easily configurable

Automated deployments

Easy integration of new pipelines thanks to reusable code

Conclusion:

The company is now able to evolve its digital products without technical bottlenecks, ensuring agility and quality.