Senior Big Data Engineer (Databricks + AWS)

SoftServe is hiring a Senior Big Data Engineer to work on scalable cloud-based data solutions using AWS and Databricks. The role involves working on batch and streaming data pipelines within small international teams and in a flat organizational structure.

• 4+ years of experience in Big Data or Data Engineering
• Strong proficiency in Python (PySpark) and SQL
• Hands-on experience with AWS cloud services for data engineering
• Practical experience with Databricks on AWS, including building and managing data pipelines
• Strong knowledge of Apache Spark and large-scale data processing
• Experience with batch and streaming data processing
• Familiarity with Delta Lake and Databricks components such as Workflows and Jobs
• Experience with orchestration tools like Apache Airflow or MWAA
• Knowledge of streaming technologies such as Apache Kafka, Amazon MSK, or Kinesis
• Upper-intermediate or higher level of English

• Design, develop, and maintain scalable batch and streaming data pipelines
• Build and optimize data processing solutions using Python (PySpark) and SQL
• Develop and manage data workflows using Databricks on AWS
• Work with Apache Spark and Databricks for large-scale data processing
• Implement and maintain Delta Lake-based data architectures
• Utilize orchestration tools such as Apache Airflow or MWAA to manage workflows
• Support implementation of streaming solutions using Apache Kafka, Amazon MSK, or Kinesis
• Collaborate with technical and business stakeholders to deliver effective data solutions
• Participate in architecture discussions and continuous improvement initiatives
• Participate in the full project lifecycle from PoC and MVP to production implementation