Cloud Data Pipelines Engineer
Brak informacji o wynagrodzeniu
NieokreśloneFull-time
#455397·Dodano 6 dni temu·0
Źródło: LinkGroupTech Stack / Keywords
PythonRGCPAzureDockerKubernetesApache AirflowPrefectPostgreSQLBigQuery
Wymagania
- Advanced proficiency in Python and/or R, with experience writing clean, modular, and well-tested code.
- Hands-on production experience with Google Cloud Platform (GCP) or Microsoft Azure.
- Practical experience with Docker, Kubernetes, and CI/CD pipelines like GitHub Actions or GitLab CI.
- Proficiency with SQL and relational/non-relational data warehouses such as PostgreSQL, BigQuery, Snowflake, or Azure SQL.
- Solid grasp of version control (Git), automated testing practices, and API design principles.
Obowiązki
- Containerize, deploy, and orchestrate custom R and Python scripts in cloud-native serverless or containerized environments.
- Leverage managed cloud services on GCP (Cloud Functions, Cloud Run, BigQuery, Dataflow) or Azure (Azure Data Factory, Azure Functions, Databricks, Synapse Analytics).
- Implement and maintain workflow automation tools such as Apache Airflow or Prefect to schedule and monitor script execution.
- Implement automated logging, error handling, and testing frameworks to validate data integrity and script outputs.
- Tune data processing jobs and cloud resource allocation for cost-efficiency, execution speed, and scalability.
Link Group
495 aktywnych ofert