Data Engineer (Azure)

We support recruitment for a US-based company that provides mission-critical background screening solutions for Fortune 100 clients. The company has over 4500 employees from more than 30 nationalities and focuses on digital background check and verification services globally.

• Strong experience with Python/PySpark and SQL.
• Hands-on experience building data pipelines using Databricks Spark.
• Experience processing large-scale datasets in production.
• Strong knowledge of Kafka or streaming platforms like Azure Event Hubs or Amazon Kinesis.
• Experience building streaming data integrations and event-driven pipelines.
• Good understanding of Spark Structured Streaming.
• Familiarity with CDC (Change Data Capture) concepts; experience with Debezium is a plus.
• Strong knowledge of Databricks Delta optimization techniques.
• Experience developing reusable Python libraries and packages.
• Hands-on experience with CI/CD pipelines.
• Good understanding of networking fundamentals.
• Familiarity with Agile/Scrum methodologies.

• Build reusable, metadata-driven data pipelines.
• Automate and optimize data platform processes.
• Develop integrations with data sources and consumers.
• Extend shared ETL libraries with transformation methods.
• Write unit tests.
• Create monitoring solutions for the Databricks platform.
• Proactively address ETL performance and quality issues.
• Collaborate with infrastructure teams on cloud resources.
• Update data platform wiki and documentation.
• Conduct code reviews to ensure quality.
• Initiate and implement architecture improvements.