N-iX

Senior DataOps Engineer

4500 - 5600 USD/ mies./ mies.B2BB2B (netto)

SeniorZdalnie·KrakówFull-time·B2B

#447109·Dodano miesiąc temu·4

Źródło: justjoin.it

Aplikuj teraz

Tech Stack / Keywords

Data Engineering/DataOpsAWS Data Analytics stackApache IcebergApache Spark / PySpark and Spark StreamingMedallion Lakehouse ArchitectureRelational DatabasesCI/CD

Podsumowanie oferty

The Senior DataOps Engineer will work on accelerating the client's Data & AI initiatives through a secure hybrid cloud foundation on AWS and modernizing the IT estate as part of cloud migration. The role involves designing and maintaining scalable data platforms and pipelines, with focus on batch and streaming data processing. Key responsibilities include implementing data tokenization, real-time streaming architectures, and Medallion Lakehouse Architecture on the AWS platform.

Wymagania

4+ years experience as a Data Engineer or DataOps Engineer building enterprise-grade data platforms and pipelines.
Strong expertise with AWS Data Analytics stack: Amazon S3, AWS Glue Data Catalog, AWS Lake Formation, Amazon EMR (EMR-on-EKS / Serverless), Amazon Athena, AWS DataSync, and Amazon MSK.
Deep experience with Apache Iceberg table format, cataloging, compaction, and schema evolution.
Proficient in Apache Spark / PySpark and Spark Streaming for batch, micro-batch, and real-time processing.
Strong experience in Medallion Lakehouse Architecture design and implementation.
Practical experience in data tokenization and encryption at scale (e.g., Protegrity, Thales, FPE, Spark UDF).
Solid knowledge of event-driven architectures and streaming: Apache Kafka, Amazon MSK, Kafka Connect (SMT), Schema Registry, MirrorMaker2 / MSK Replicator.
Expertise with Relational Databases such as Amazon RDS, PostgreSQL, and Oracle.
Hands-on experience with DataOps CI/CD and Automation: GitLab CI/CD, Terraform, AWS CDK, schema-drift detection, and data contract validation.
Familiarity with data governance tools and enterprise data catalogs like Informatica Axon/EDC and AWS Lake Formation.

Nice to have:

AWS Certified Data Analytics – Specialty or AWS Certified Data Engineer – Associate.
Experience in telecom domain data models, CDR processing, and high-throughput real-time telemetry.
Experience with cloud-side tokenization/detokenization via Athena UDFs / AWS Lambda.
Familiarity with containerization technologies: Docker, EKS, Kubernetes for big data runtimes.
Experience with AWS Macie and FinOps cost-allocation and anomaly-detection frameworks.

Obowiązki

Design, implement, and maintain the AWS Data Platform foundation including Amazon S3, Apache Iceberg table format standardization, and AWS Glue Data Catalog integration.
Build and optimize scalable batch and streaming data pipelines using Amazon EMR (Serverless and EMR-on-EKS), Apache Spark/PySpark, Apache Flink, and Amazon Athena.
Implement data tokenization and de-identification pipelines using Protegrity/Spark UDFs, Spark Streaming, and Kafka Connect SMT.
Build and manage real-time streaming architectures with Amazon MSK, MSK Replicator/MirrorMaker2, and Schema Registry.
Implement Medallion Architecture (Bronze, Silver, Gold layers) and automate Iceberg table registration and backfill.
Execute data migrations across non-PII and PII datasets using AWS DataSync and streaming migration configurations.
Configure data access control and governance with AWS Lake Formation, cross-engine authorization, AWS Macie, and Informatica Data Catalog integration.
Build automated schema-drift identification components and integrate GitLab CI/CD pipelines for dataset synchronization.
Implement DataOps observability with Amazon CloudWatch, FinOps cost and anomaly monitoring, and alerting.
Author technical documentation, operational runbooks, disaster recovery procedures, and cutover/rollback playbooks for platform hardening.

N-iX

45 aktywnych ofert

Zobacz wszystkie oferty

Aplikuj teraz