Data Engineer (GCP, Pyspark)

• 10+ years experience in software design, GCP, and Pyspark development.
• Experience with automated testing in Agile, DevOps environments.
• Guidance on best practice environment setup.
• Comfortable presenting to business non-technical and senior technical audiences.
• Interested in overseeing and guiding engineers as an engineering manager.
• Comfortable working autonomously as lead of engineering pod.
• Experience with scheduling tools such as Airflow.
• Experience with technologies including Apache Hadoop, Pyspark, Apache Spark, YARN, Hive, Python, ETL frameworks, Map Reduce, SQL, RESTful services.
• Strong knowledge of Unix/Linux platforms.
• Hands-on experience building data pipelines in GCP and preferably Hadoop, including Hive, Spark SQL.
• Experience with version control tools (Git, GitHub), deployment tools (Ansible, Jenkins), and requirement management in JIRA.
• Experience debugging code issues and communicating findings.
• Understanding or experience of cloud design patterns.
• Interest in AI enablers to accelerate workflows.
• Strong verbal and written English skills.

Nice to have:

• Experience with Elastic search.
• Experience developing Java APIs.
• Experience with data ingestion.
• Exposure to DevOps & Agile methodologies such as Scrum and Kanban.

• Define and contribute to system architecture and design.
• Promote development standards, conduct code reviews, mentor, and share knowledge.
• Provide production support and troubleshooting.
• Implement tools and processes ensuring performance, scale, availability, accuracy, and monitoring.
• Liaise with Business Analysts to ensure requirements are correctly interpreted and implemented.
• Participate in planning, status meetings, sprint reviews, and retrospectives.
• Serve as technical SME on data migration projects, guiding approach and estimation.