Big Data Engineer (Scala, Spark)

Link Group specializes in building tech teams for Fortune 500 companies and startups, connecting professionals with career opportunities. The project involves building a high-performance data platform for the finance/stock exchange industry, focusing on real-time analytics and reporting with scalability and efficiency.

• At least 3+ years of experience in big data engineering.
• Proficiency in Scala and experience with Apache Spark.
• Strong understanding of distributed data processing and frameworks like Hadoop.
• Experience with message brokers like Kafka.
• Hands-on experience with SQL/NoSQL databases.
• Familiarity with version control tools like Git.
• Solid understanding of data architecture and ETL processes.
• Good command of English.

Nice to have:

• Experience with cloud platforms such as AWS, Azure, or GCP.
• Knowledge of containerization tools like Docker and orchestration with Kubernetes.
• Familiarity with CI/CD pipelines for big data workflows.
• Experience in optimizing Spark jobs for large-scale data processing.
• Academic background in Computer Science, Data Engineering, or a related field.

• Design, implement, and optimize big data processing pipelines using Scala and Spark.
• Work with massive datasets, ensuring scalability and performance.
• Collaborate with data scientists and analysts to develop data solutions.
• Monitor and troubleshoot data workflows to ensure reliability and efficiency.
• Integrate with data sources and ensure proper data ingestion and transformation.
• Research emerging big data technologies to improve the architecture.