Senior Data Engineer/Spark Developer

An established accounting firm seeks a senior-level data engineering professional to design, develop and maintain scalable Apache Spark applications.

• Minimum 7 years of hands-on experience in software development, with a strong focus on data engineering.
• Extensive experience with Apache Spark and working knowledge of Databricks and/or Azure Synapse/Fabric.
• Proficiency in Python, including strong data-manipulation skills using Pandas and/or Polars.
• Practical experience with columnar data storage formats, especially Parquet, and working with Delta Tables.
• Proven expertise in data processing, analysis and transformation workflows at scale.
• Solid understanding of microservices architecture and experience implementing scalable solutions.
• Strong analytical and problem-solving skills with a detail-oriented approach.
• Excellent organisational skills and the ability to self-manage, prioritise tasks and meet tight deadlines.
• Effective communication skills for collaboration with internal teams and external clients.

• Establish, document and enforce best practices and coding standards across the project.
• Conduct thorough code reviews to ensure high code quality and compliance with established guidelines.
• Design, develop and maintain robust, scalable Spark applications for large-scale data processing.
• Write clean, maintainable and efficient code following accepted engineering practices.
• Optimise code and data pipelines for performance and scalability, ensuring efficient handling of columnar formats.
• Develop and maintain Delta Table and Parquet-based data workflows where applicable.
• Collaborate with cross-functional teams to deliver reliable software solutions and integrate with microservices architectures.
• Identify, diagnose and resolve technical issues to maintain application reliability and performance.
• Create and maintain comprehensive technical documentation for code, processes and operational workflows.
• Manage day-to-day operational flow and communicate clearly with team members and stakeholders.
• Maintain regular communication with the client to align requirements, progress and deliverables.