Data Engineer

You will join the team behind a large-scale, centralized data platform built for a global consulting organization. The platform powers several internal products and supports Mergers & Acquisitions (M&A) engagements with firm-level and site-level data, including firmographics, technographics, hierarchical relationships, and business metrics such as revenue and web traffic. The goal is to maintain a unified, enterprise-grade dataset of 300M+ company records integrated from 10+ sources.

• Strong Python experience including data structures, typing, error handling, generators/iterators, context managers, standard library
• Fundamentals of software engineering: modular design, dependency management, packaging, API design
• Testing discipline including pytest, fixtures, mocking, and testable code design
• Hands-on experience building and operating ETL/ELT pipelines in production
• Strong experience with Snowflake and dbt
• Experience with Apache Airflow or similar orchestration tools
• Solid working knowledge of SQL and data modeling for schema design and queries
• Experience with Docker, Kubernetes, and CI/CD practices
• Debugging and profiling skills for Python performance, concurrency, and memory
• Experience with at least one public cloud: AWS or Azure
• Experience with version control systems (Git)
• Ability to explain technical trade-offs to technical and business audiences
• Experience using AI coding assistants such as Claude Code, Cursor, or similar daily
• Strong communication skills and good command of English (minimum C1 level)

Nice to have:

• Experience with Apache Spark, ideally on Databricks
• Experience with Pydantic or similar schema-validation libraries
• Experience with Python web/API frameworks (FastAPI, Flask)
• Asynchronous Python, multiprocessing, or other concurrency patterns
• Experience with Azure AI Search or AWS OpenSearch
• Knowledge of a second programming language: Go, Rust, Scala, or TypeScript
• Familiarity with LLMs, Azure OpenAI, or agentic AI systems

Engineering & delivery (~70%):

• Design, build, and maintain batch and streaming data pipelines in Python, including orchestration, scheduling, and monitoring with Airflow
• Write reusable, well-typed Python libraries and internal packages
• Build Python services and APIs using FastAPI that expose data to downstream applications and integrate with third-party/internal APIs
• Develop data transformation logic in SQL and dbt on Snowflake, build data models for fast querying
• Write unit, integration, and data-contract tests and maintain automated CI coverage
• Profile and optimize Python code and data processing jobs for performance and cost
• Deploy and operate code in the cloud using containers, infrastructure as code, and CI/CD systems
• Enforce access controls, secrets handling, and data protections throughout the data lifecycle
• Use AI coding assistants effectively while validating generated output and help build automated quality gates

Continuous improvement (~30%):

• Find and fix efficiency, reliability, cost, and correctness issues in pipelines with refactoring
• Replace scripts and notebooks with tested, packaged, scheduled code
• Implement data quality checks, validation, and monitoring
• Create matching logic to deduplicate and connect entities across multiple data sources
• Document data processes, system architecture, and maintain project documentation