Lead Data Engineer (PySpark)

EPAM is a leading global provider of digital platform engineering and development services. We are committed to having a positive impact on our customers, our employees, and our communities. We embrace a dynamic and inclusive culture. Here you will collaborate with multi-national teams, contribute to a myriad of innovative projects that deliver the most creative and cutting-edge solutions, and have an opportunity to continuously learn and grow. No matter where you are located, you will join a dedicated, creative, and diverse community that will help you discover your fullest potential.

• 9+ years of experience in data engineering with AI exposure
• Expertise in Azure Fabric and end-to-end Fabric experience
• Knowledge of OneLake (Delta / OpenLake)
• Advanced skills in Python, PySpark and SparkSQL
• Proficiency in Cosmos DB (NoSQL API) and other Cosmos DB variants
• Understanding of DF Gen2 and M-code
• Capability to implement CI/CD pipelines using Azure DevOps or equivalent tools
• Experience with generic Azure services and Power BI integration, semantic models and performance considerations
• Background in Agile or Scrum environments
• Strong ownership mindset and ability to lead by example
• Excellent communication skills for technical and non-technical audiences
• Good understanding of the financial domain

Nice to have:

• Experience with code generation, including non-AI and AI-assisted approaches
• Azure AI Foundry experience
• Data Science fundamentals and collaboration with DS teams
• Strong background in Big Data architectures and Spark ecosystems
• Familiarity with financial instruments and financial services data
• Hands-on experience with industry-standard LLMs (including GPT, Claude or similar)
• Exposure to AI-enabled data platforms and intelligent analytics use cases

• Lead the design, development and optimization of scalable data engineering solutions using Azure Fabric and cloud-native technologies
• Own end-to-end data pipelines including ingestion, transformation, storage and analytics
• Architect and implement solutions leveraging OneLake (Delta / OpenLake) and Fabric experiences
• Develop and optimize PySpark, SparkSQL and Python-based data processing pipelines
• Work with Cosmos DB (NoSQL API) and other Cosmos DB variants to support high-performance data access patterns
• Implement and maintain CI/CD pipelines and promote DevOps best practices
• Collaborate with data scientists, AI engineers and product stakeholders to enable AI-driven analytics and insights
• Mentor and guide junior engineers, setting coding standards and best practices
• Ensure data quality, security, governance and performance across platforms
• Contribute to technical decision-making and solution architecture discussions