DataBricks Developer (IoT sphere)

Our client is a global leader in access solutions and smart home technologies. They offer connected home solutions including garage door openers, gate systems, and smart access management tools powered by the myQ digital ecosystem. Headquartered in Oak Brook, Illinois, they serve residential and commercial customers worldwide.

• 5+ years of experience in Data Engineering.
• Strong experience with Databricks.
• Advanced proficiency in Python, PySpark, and strong SQL skills.
• Experience designing and implementing data pipelines based on the Medallion Architecture (Bronze, Silver, and Gold).
• Experience building scalable ETL/ELT data pipelines.
• Experience integrating data from REST APIs and third-party services.
• Experience developing real-time or near real-time data pipelines.
• Experience optimizing Spark workloads for performance and scalability.
• Understanding of data quality, governance, and monitoring best practices.
• Strong problem-solving and communication skills.
• English proficiency at B2 level or higher.
• Bachelor's degree in Computer Science or a related field.

Nice to have:

• Experience with Delta Lake and Delta Live Tables (DLT).
• Experience with Databricks Workflows.
• Experience with Unity Catalog.
• Experience with Apache Kafka or other streaming technologies.
• Experience with Azure Data Factory, AWS Glue, or similar data orchestration tools.
• Experience implementing CI/CD processes for data pipelines.
• Experience working in Agile teams.

• Design, build, and maintain scalable data pipelines using Databricks.
• Design and implement API integrations to ingest data from multiple internal and external systems.
• Implement and maintain data pipelines based on the Medallion Architecture (Bronze, Silver, and Gold) to support reliable and scalable analytics.
• Build and optimize batch and real-time data processing pipelines to meet performance, latency, and reliability requirements.
• Develop reusable data transformation frameworks using Python and PySpark.
• Optimize Spark jobs and Databricks workloads for performance and cost efficiency.
• Ensure data quality through validation, monitoring, and automated testing.
• Collaborate with architects, product owners, and engineering teams to design scalable data solutions.
• Troubleshoot production issues and continuously improve pipeline reliability and observability.
• Document data flows, architectural decisions, and operational procedures.

Background check and drug test required.