Remote Senior Data Engineer

Varwise is an Adtech company focusing on building a scalable identity graph that reaches millions of users by leveraging machine learning and data science.

• 8+ years of professional software engineering experience focused on data engineering in big data environments.
• Expert-level proficiency in Python and at least one other high-level programming language (e.g., Java, Scala, C++, C#).
• Proven experience building and optimizing large-scale data pipelines using Apache Spark.
• Hands-on experience developing and deploying data solutions in major cloud platforms (AWS, GCP, or Azure).
• Experience working with AI, LLMs, Agents, and/or Generative AI technologies.
• Relevant technology experience in the advertising industry.
• Experience with Scala.
• Experience with big data visualization and analytics using OLAP tools.
• Familiarity with big data tools and frameworks such as MLFlow, dbt, Kafka, Airflow, or Databricks.
• Experience with large-scale data management formats like Parquet, Delta Lake, or Iceberg.

Nice to have:

• Experience with Kinesis, Jenkins, TensorFlow.

• Design, develop, and maintain highly scalable data pipelines, ETL processes, and data models using Python, Spark, and other big data technologies in a cloud environment (AWS/GCP).
• Lead the integration of AI and LLMs into data products, collaborating with data scientists to productionize machine learning models and agentic workflows.
• Utilize AI-driven software development tools (e.g., GitHub Copilot) to enhance productivity, code quality, and delivery speed.
• Maintain reliable and scalable distributed data processing systems.
• Become a core maintainer of the data lake, building searchable datasets for broader business uses.
• Scale, troubleshoot, and fix existing applications and services, ensuring data pipelines run 24/7.
• Lead technical discussions to improve tools, processes, or projects.
• Work on scaling the identity graph to deliver effective advertising campaigns.
• Handle data sets exceeding billions of records.
• Scale the MLOps platform using traditional ML and LLM/Generative AI applications.

Tools: Jira, GitHub, Git, Jenkins, Agile, Kanban.