Data Engineer

payabl. empowers businesses through payments innovation and banking services. It is a licensed financial company specializing in global payments and multi-currency accounts, providing a portfolio of global financial services via its platform payabl.one.

• Minimum 3+ years of experience in data engineering or related roles.
• Strong experience with SQL and data modeling for analytics and reporting.
• Strong programming experience with Python, ideally including PySpark.
• Experience designing and maintaining ETL/ELT pipelines in production.
• Experience with real-time or near-real-time data ingestion.
• Experience working with Kafka or similar streaming technologies.
• Experience with CDC concepts and tools such as Debezium.
• Experience with data lake or lakehouse architectures on cloud platforms.
• Hands-on experience with AWS data services such as S3, Glue Catalog, Glue Jobs, EMR, IAM.
• Experience with Apache Iceberg, Delta Lake, or similar open table formats.
• Experience designing curated analytics layers like silver and gold layers in medallion architecture.
• Experience with orchestration tools such as Apache Airflow or Dagster.
• Experience working with relational databases such as MySQL, MariaDB, or PostgreSQL.
• Working knowledge of Unix/Linux environments and shell scripting.
• Understanding of data quality, governance, lineage, and production monitoring.

Nice to have:

• Experience with Apache Druid, ClickHouse, Snowflake, Databricks, or other analytical databases.
• Experience with Airbyte or similar data integration tools.
• Experience with dbt or collaboration with analytics engineering teams.
• Experience with Terraform and Terragrunt.
• Experience with Docker and Kubernetes.
• Experience optimizing Spark jobs on EMR or AWS Glue.
• Experience with data visualization tools such as Tableau, Power BI, Superset, AWS QuickSight.
• Experience with data observability, alerting, and monitoring tools.

Lakehouse Architecture and Data Platform Design:

• Design, build, and maintain scalable data lakehouse solutions on AWS using S3, Apache Iceberg, and AWS Glue Catalog.
• Contribute to the evolution of the medallion architecture ensuring reliable and performant bronze, silver, and gold layers.

CDC and Streaming Data Ingestion:

• Build and support real-time and near-real-time ingestion pipelines using Debezium, Kafka, Kafka Connect, and Iceberg sinks.
• Monitor and troubleshoot streaming pipelines including connector issues, schema changes, and data consistency.

Data Modelling and Business-Ready Layers:

• Design and implement silver and gold layer datasets transforming raw data into trusted business-ready products.
• Collaborate with analysts and business stakeholders to understand requirements and create reusable data models.

Batch and Distributed Processing:

• Develop and maintain batch and distributed processing jobs using PySpark on AWS EMR and AWS Glue.
• Optimize data transformation jobs for performance, scalability, reliability, and cost-efficiency.

Data Integration and Orchestration:

• Build and maintain data workflows with Apache Airflow for API ingestion, batch processing, and orchestration.
• Support integrations from external and third-party systems using tools like Airbyte.

Data Quality, Governance, and Reliability:

• Implement data quality checks, validation, and reconciliation to ensure trusted datasets.
• Contribute to data governance including documentation, ownership, lineage, access, and compliance.

Infrastructure and DevOps Collaboration:

• Collaborate on AWS infrastructure and deployment using services such as S3, Glue, EMR, IAM, EKS.
• Support infrastructure-as-code workflows using Terraform and Terragrunt.

Please note if hired in Poland or Portugal, the Talent Acquisition Partner will inform about specific working arrangements and conditions during the recruitment process.