Senior Data Engineer - Databricks

Intetics Inc. is a global technology company specializing in custom software development, AI-powered solutions, cloud technologies, and digital transformation, with over 30 years of experience across a wide range of industries.

• 4+ years of data engineering experience.
• Minimum 2 years of experience with Databricks or the Apache Spark ecosystem on Azure and/or AWS.
• Proficient in PySpark, SQL, and Python with production pipeline experience under SLA constraints.
• Hands-on experience with Delta Lake including schema evolution, ACID transactions, and incremental/streaming patterns.
• Experience in pipeline performance tuning and compute optimization in Databricks production.
• Solid knowledge of PostgreSQL, including query optimization and schema design.
• Experience maintaining legacy ETL tools like SSIS, Informatica, or custom Python/SQL pipelines.
• Experience with large-scale multi-tenant architectures focusing on tenant isolation and data privacy.
• Proven collaboration across data science, product, and infrastructure teams.
• Strong understanding of data governance, security, and compliance principles.

Nice to have:

• Operating Databricks workspaces across Azure and AWS, including cost governance and cluster management.
• Optimizing Databricks workloads in a serverless environment.
• Experience with Microsoft SQL Server in data engineering contexts.
• Exposure to ML feature engineering or feature stores such as Databricks Feature Store or Feast.
• Experience with customer onboarding automation or Infrastructure as Code (IaC) for tenant pipelines.
• Databricks Certified Data Engineer Associate or Professional certification.

• Own Databricks production support including monitoring, alerting, and incident response for all production data flows.
• Maintain and report on SLA performance metrics for data pipeline delivery.
• Identify and implement pipeline optimizations to reduce compute costs and improve throughput.
• Migrate legacy ETL/ELT pipelines to Databricks with automation tooling.
• Support new customer onboarding by provisioning and hardening tenant data pipelines.
• Design and build high-performance Databricks pipelines across Azure and AWS.
• Own the Delta Lake architecture including schema design and incremental processing.
• Enforce data security best practices including role-based access control and secrets management.
• Implement data quality monitoring and observability for pipelines and ML model inputs.
• Apply multi-tenant data isolation patterns for secure data delivery.
• Partner with Enterprise Architecture for integration with AI and analytics ecosystem.
• Participate in on-call rotation and after-hours incident response for global operations.
• Maintain technical documentation, runbooks, and architectural decision records.
• Apply CI/CD best practices including version control, automated testing, and deployment tooling.