[C3F] Data Platform Engineer

Software Mind develops solutions impacting companies globally, working with tech giants and unicorns on transformative projects using emerging technologies. The company builds cross-functional engineering teams that take ownership and value passion and creativity.

• Commercial experience as a data or platform engineer working on streaming or CDC pipelines
• Practical knowledge of Kafka including topics, partitions, offsets, consumer groups, delivery semantics, and operating at least one Kafka Connect deployment
• Strong SQL and PostgreSQL skills including working knowledge of WAL, logical replication, replication slots, and replication lag
• Understanding of CDC concepts including snapshots, inserts, updates, deletes, event ordering, delivery semantics, and schema evolution
• Confident in Python for automation including orchestration, monitoring, recovery scripts, and automated testing
• Hands-on experience with Azure data services such as Event Hubs, ADLS, or Azure PostgreSQL
• Comfortable working with Kubernetes as a user, including deployment, configuration, secrets management, log reading, and debugging
• Ability to debug running pipelines from metrics and logs distinguishing various issues

Nice to have:

• Experience with Debezium including snapshot strategy on large tables, schema recovery, offset handling, and connector recovery
• Operating stateful workloads on AKS including StatefulSets and resource tuning
• Using infrastructure-as-code tools like Terraform or Bicep for data platform components
• Production experience with Databricks and Azure Data Factory at scale
• Implementing data protection controls such as masking, hashing, access control, and retention policies

• Build and operate change-data-capture pipelines from PostgreSQL into Azure using Kafka Connect and Debezium
• Configure, deploy, and scale connectors end to end, including setup, task management, offsets, schema history, and snapshot strategy
• Run pipelines as stateful workloads on Kubernetes (AKS), managing configuration, secrets, networking, and resource tuning
• Monitor and troubleshoot platform production issues such as connector failures, task rebalances, restarts, throughput, backpressure, message-size limits, retries, and recovery
• Automate platform tasks with Python for onboarding, orchestration, monitoring, alerting, recovery workflows, and automated testing
• Integrate CDC streams with wider Azure data services: Event Hubs, ADLS, Azure PostgreSQL, ADF, and Databricks
• Manage platform infrastructure as code for environment reproducibility and change review
• Apply data protection requirements on sensitive data such as masking, hashing, access control, and retention