Big Data Engineer Spark/Scala

Diverse CG Sp. z o.o. Sp.k. is a recruitment company that supports businesses by providing experienced professionals and continuous support through a partnership approach. The company is currently growing and undertaking multiple recruitment projects for its partners.

• 7+ years of experience in Spark and Scala
• 5+ years of development experience with Python
• 3+ years experience with Robot Framework
• 3+ years experience with CI/CD tools such as Jenkins
• Experience with distributed data processing engines like Spark
• Experience with Hadoop and related tools (Hive, Oozie, MapReduce, etc.)
• Strong SQL skills with experience in creating data flows
• Expertise in BitBucket and GIT
• Knowledge of AWS architectural components (Lambda, Step Functions, State Machine, etc.)
• Unit testing experience (JUnit 5, Mockito, Spark testing)
• Knowledge of code versioning and branching strategies
• Advanced English skills
• Strong working experience with Linux and bash
• Experience using Jenkins or Bamboo, Jira, Bitbucket
• Collaborative mindset with effective communication and teamwork in an Agile (SAFe) environment
• Curiosity and responsiveness to understand others' needs and ensure delivery of expected outcomes
• Strong focus on quality and continuous improvement
• High level of self-motivation and ownership in day-to-day work

Nice to have:

• Experience with containerization using Docker
• Knowledge of streaming and queue technologies such as Kafka and IBM MQ
• Working knowledge of DevOps

• Build distributed and highly parallelized Big Data processing pipelines that process massive amounts of structured and unstructured data in near real-time
• Leverage Spark to enrich and transform corporate data to enable searching, data visualization, and advanced analytics
• Analyze requirements and question unclear requirements
• Work closely with DevOps, QA, and Product Management teams in a Continuous Delivery environment