Senior Data Engineer
Board of Innovation
- Lisboa
- Permanente
- Horário completo
- Work together with a collaborative international team of engineers, designers and solution leads to understand customer business needs and translate them into end-to-end solutions.
- Develop and manage ETL pipelines to extract, transform, and load data from various internal and external sources into harmonized datasets.
- Design, optimize, and maintain cloud databases and data storage systems (PostgresSQL, MongoDB, Object storage).
- Ensure data integrity, quality, and security across all pipelines and workflows, and enforce data access and governance systems.
- Collaborate with AI Engineers to prepare data for use in model workflows (e.g. vector embeddings).
- 5+ years of experience building secure and scalable cloud data infrastructure, and a degree in Computer Science or related field.
- Experience in Python, SQL, and data transformation libraries like pandas or PySpark.
- Hands-on experience with data pipeline orchestration tools like Apache Airflow or Prefect.
- Solid understanding of database design and optimization for relational and non-relational databases.
- Experience with deployment on one of the major cloud providers (preferably Azure).
- Advanced English skills, both written and verbal, with the ability to communicate effectively in an international team.
- Experience working in a consulting or agency environment on project-based work.
- Passionate about working with data and building reliable scalable systems.
- Familiar with DevOps practices in CI/CD for infra-as-code (e.g. terraform, helm, argo), container orchestration (e.g. docker, k8s), and deployment automation.
- Passionate about continuously improving code quality by performing code reviews, and following best practices for testing and debugging.
- Able to take charge of a project from conception till production-level customer deployment.
- Willing to work with the team to develop knowledge around new technology as well as software development best practices.
- Within your first month, you will integrate with the team, help set up best development practices and initial data pipelines.
- Within your third month, you will be actively involved in a client project, scoping the project and implementing user stories.
- Within your six months, you will be involved in multiple client projects, having already brought one from inception to a scalable client deployment. You will be actively working with the team to develop internal software primitives and IP.