Construção de pipelines de dados sobre obras públicas em Pernambuco: abordagem prática com o Apache Airflow

Abstract

This study presents a practical approach to building data pipelines focused on collecting, transforming, and storing information related to public works in the state of Pernambuco. The central objective is to develop efficient and automated workflows for extracting data from public transparency portals and subsequently consolidating this information. Based on Data Engineering technologies, the Apache Airflow framework was chosen to orchestrate the processes, enabling the scheduling and monitoring of these workflows.