Spark / Data Lakehouse Developer

• Minimum 4 years of experience with Apache Spark.
• Strong knowledge of PySpark and Spark SQL.
• Experience with MongoDB and MongoDB Spark Connector.
• Practical knowledge of Apache Iceberg.
• Familiarity with CI/CD practices and Jenkins.
• Hands-on experience with Prometheus and Grafana.
• Knowledge of Spark optimizations including partitioning, caching, broadcast joins, and Adaptive Query Execution (AQE).
• Experience working with large datasets.
• Analytical and independent problem-solving skills.

Nice to have:

• Experience using Scrum/Kanban and collaboration tools like Jira and Confluence.

• Develop and optimize Apache Spark processes.
• Build Data Lakehouse layers: raw/bronze/silver/gold.
• Implement data reconciliation and quality controls.
• Collaborate with Data, QA, and DevOps teams.
• Work using Scrum/Kanban methodologies and tools like Jira and Confluence.