Kafka Engineer

Upvanta is hiring a Kafka Engineer with expertise in building a hybrid Kafka environment integrating on-premise and AWS clusters. The role focuses on leveraging the Confluent Platform and AWS cloud services for a scalable, secure, and highly available data streaming platform.

• Minimum 4 years experience with Apache Kafka.
• Proficient in Confluent Platform (Enterprise), Confluent Kafka, and Confluent Cloud.
• Experience building, maintaining, migrating, and replicating Kafka clusters.
• Practical knowledge of MirrorMaker 2, Cluster Linking, and Confluent Replicator.
• Practical experience with AWS services: VPC, IAM, EC2, EKS, MSK, S3, KMS, Secrets Manager.
• Experience building and maintaining hybrid on-premise to AWS environments.
• Strong knowledge of Terraform including module creation, state/version management, testing (Terratest, tfsec).
• Knowledge of Ansible.
• Experience creating and maintaining CI/CD pipelines with GitHub Actions, GitLab CI, Jenkins.
• Experience automating integration tests in Kafka.
• Programming and scripting skills in Python, Bash, and Ansible.
• Experience with monitoring tools: Prometheus, Grafana, Confluent Control Center, OpenTelemetry, Elastic Stack.
• Knowledge of security protocols: TLS, SASL, ACL, RBAC, data encryption at rest and in transit.
• Experience working in Scrum and/or Kanban methodologies.
• Familiarity with tools such as Jira and Confluence.

Kafka and Confluent:

• Design hybrid Kafka topology connecting on-premise and AWS.
• Configure Confluent Platform components including Schema Registry, Flink, Control Center, Tableflow.
• Design architecture ensuring high availability, security, and scalability.
• Create and configure Kafka clusters including Amazon MSK and Confluent Cloud.
• Implement and configure MirrorMaker 2, Cluster Linking, and Confluent Replicator.

AWS and Infrastructure:

• Configure AWS services: VPC, IAM, EC2, EKS, S3, KMS, Secrets Manager.
• Integrate Kafka components with existing on-premise infrastructure.

CI/CD and Automation:

• Design and maintain CI/CD pipelines using GitHub Actions, GitLab CI, Jenkins.
• Automate Kafka, schema, and consumer application deployments.
• Implement Infrastructure-as-Code with Terraform including module creation, state management, versioning, and testing.
• Use Ansible for configuration and deployment automation.
• Automate Kafka integration tests.

Monitoring and Security:

• Implement monitoring with Prometheus, Grafana, Confluent Control Center, OpenTelemetry, Elastic Stack.
• Create runbooks, alerts, and SLA/SLOs.
• Troubleshoot platform incidents and optimize Kafka performance.
• Manage security configurations including TLS, SASL, ACL, RBAC.
• Implement data encryption at rest and in transit.
• Collaborate with Security teams on secure architecture.

Teamwork and Mentoring:

• Work within Data Platform team alongside Data Engineers, DevOps, and Security.
• Mentor junior team members.
• Conduct technical workshops and knowledge sharing.