Data Engineer

• Minimum 3 years commercial experience as a Data Engineer
• Practical knowledge of Python for building and maintaining production ETL/ELT pipelines
• Experience with REST API integrations and JSON/XML data processing
• Good knowledge of SQL
• Experience in data validation, schema drift handling, error logging, and data reprocessing using tools such as Pydantic, Pandera, or Great Expectations
• Hands-on experience with Azure services: Azure Data Lake Storage, Azure Data Factory, Azure Functions, Azure SQL, Azure Databricks, or Microsoft Fabric
• Familiarity with Git and CI/CD workflows using Azure DevOps or GitHub Actions
• English language proficiency at minimum B2+ level for recruitment and daily client collaboration
• Independence and ability to adapt within existing teams and technical environments

Nice to have:

• Experience integrating AI/LLM results with data pipelines and evaluating their quality
• Knowledge of preparing data models for Power BI
• Experience with legislative or regulatory data, e.g., APIs from Congress.gov or UK Parliament

• Integration with external REST APIs, including handling authorization, pagination, rate limits, and retries
• Fetching, parsing, and processing data in JSON and XML formats
• Implementing data transformations and structures as per architecture
• Ensuring data quality via validation, schema drift handling, and reprocessing capabilities
• Implementing logging, monitoring, and error handling for pipelines
• Working with Git repositories and client CI/CD processes
• Ongoing collaboration with client teams, independently refining requirements and reporting technical issues