Data Engineer (PySpark)

Brak informacji o wynagrodzeniu

MidFull-time

#422607·Dodano 6 dni temu·1

Źródło: YourITeams

Tech Stack / Keywords

PySparkETLAWSArchitectureSparkApache SparkSQLAgile

2–4 years of experience as a Data Engineer or in a similar role.
Practical, hands-on experience with PySpark / Apache Spark.
Solid understanding of key Spark concepts: partitioning, shuffle, repartition vs coalesce, broadcast joins, and data skew impact.
Practical experience with AWS services, especially AWS Glue, Amazon S3, and Amazon CloudWatch.
Experience building, maintaining, or enhancing ETL/data processing pipelines.
Ability to diagnose common Glue and PySpark issues from logs and basic metrics.
Experience handling ETL error scenarios: retries, reprocessing, partial failures, and preventing duplicate processing.
Basic understanding of idempotency and safe restart strategies after failures.
Experience processing large data volumes or many files and familiarity with batch migration processes.
Strong SQL skills and familiarity with joins, aggregations, filtering, grouping, and ranking.

Implement data transformations and ETL logic using PySpark.
Maintain and develop existing AWS Glue jobs and related pipelines.
Investigate and resolve pipeline failures using logs, error messages, and basic metrics.
Diagnose and mitigate Spark performance issues (partitioning, shuffle, skew, joins).
Choose appropriate partitioning and join strategies for given workloads.
Handle error scenarios in ETL processes, including retries, reprocessing, and partial failures to avoid duplicate processing.
Work with large volumes of data and many files without creating uncontrolled parallelism.
Follow established architectural patterns and implement solutions that integrate with the current system design.
Support data migration or batch processing tasks as needed.

Remote working.
Unique TEAL culture, relationship- and respect-driven community, non-corporate atmosphere.
Agile approach and no bureaucracy.
Outstanding integration trips to various places in Europe.
Activities to support your well-being and health.
Luxmed Gold Extended medical care and Multisport Plus benefit.

Opieka zdrowotna

Karta sportowa

YourITeams

79 aktywnych ofert