AWS Data Engineer (DataBricks+DataLake)

H2B Group is a consulting company that helps clients automate business processes using AI and supports IT specialists in developing their careers in a B2B model. They offer projects in industries such as fintech, medtech, edutech, and industrial automation. The company collaborates closely to deliver fast and measurable results for clients while providing IT specialists with career development opportunities and valuable experience.

• Minimum 4 years of experience in data engineering.
• At least 2 years hands-on experience with Databricks.
• At least 2 years hands-on experience with AWS.
• Experience in at least one large data or platform migration.
• Strong knowledge of Spark and PySpark.
• Practical experience with Delta Lake features.
• Familiarity with Databricks Workflows, Jobs, Unity Catalog.
• Strong knowledge of AWS services including S3, IAM, KMS, VPC, PrivateLink, Secrets Manager.
• Experience integrating AWS Lambda, SNS/SQS, Glue.
• Proficient in SQL, with experience in SQL Server and PostgreSQL.
• Ability to convert T-SQL logic to Databricks/PostgreSQL.
• Experience migrating database schemas, procedures, functions, and validating data.
• Knowledge of data governance, security, compliance, and access control.
• Experience with CI/CD pipelines and Infrastructure as Code, preferably Terraform.
• Skills in data testing and monitoring pipelines.
• Strong project management and communication skills.

Nice to have:

• Databricks Certified Data Engineer Associate/Professional.
• AWS Certified Data Analytics or AWS Solutions Architect.
• Experience with Unity Catalog and governance in enterprise environments.

Data Engineering and Databricks:

• Strong knowledge of Spark and PySpark.
• Experience with Delta Lake features such as ACID, partitioning, Z-Ordering, optimize, vacuum.
• Building batch and near real-time data pipelines.
• Familiarity with Databricks Workflows, Jobs, Unity Catalog.
• Writing efficient transformations and debugging performance issues.

AWS and Data Lake Architecture:

• Strong knowledge of S3 as storage layer.
• Experience with IAM, KMS, VPC, PrivateLink, Secrets Manager.
• Integration with services like Lambda, SNS/SQS, Glue.
• Designing dev/test/prod environments and separation of accounts/permissions.
• Cost and performance optimization in the cloud including tagging and monitoring.

SQL and Database Migrations:

• Very good SQL skills, experience with SQL Server and PostgreSQL.
• Ability to convert T-SQL logic to Databricks/PostgreSQL compatible approaches.
• Experience migrating schemas, procedures, functions, and data validation.
• Performing reconciliation after migration including quality controls and reporting differences.

Data Governance, Security, Compliance:

• Designing permission models at catalog, schema, table, and column levels.
• Good practices in data lineage, auditing, and access policies.
• Understanding compliance requirements and least privilege principles.
• Working with sensitive data including masking, tokenization, and access control.

DevOps and Quality:

• CI/CD for data pipelines and infrastructure, Infrastructure as Code preferably with Terraform.
• Data testing: unit, integration, and data quality checks.
• Monitoring and alerting: SLA/SLO for pipelines, incident handling, runbooks.

Project and Communication Skills:

• Ability to work cross-team with application teams and architects.
• Risk management for client-to-client migrations.
• Clear communication of status, blockers, and technical decisions.
• Product thinking: delivering not just code but a stable migration path.

The data controller is H2B Group Sp. z o.o. based in Gdańsk, Poland. Candidates have rights regarding their personal data including access, correction, deletion, restriction, objection, portability, and complaint to supervisory authorities. Data processing is for recruitment purposes. Providing mandatory data is required by law; refusal may prevent recruitment. Data will be processed until recruitment ends and for potential claims. Consent for future recruitment can be withdrawn anytime.