Informații principale

Descrierea și cerințele proiectului

Responsibilities:
- Develop and maintain modular, well-tested Python code, emphasizing object-oriented programming principles.
- Design and implement CI/CD pipelines, with a focus on automation and efficiency, preferably using Azure DevOps.
- Perform data engineering tasks on Azure Databricks, handling Spark batch processing, structured streaming, and PySpark transformations.
- Work with Databricks Lakehouse architecture, applying medallion design patterns for efficient data organization and processing.
- Manage relational databases effectively, with a preference for using Azure SQL Database.

Skills:
- Strong Python programming, especially in an object-oriented approach.
- Expertise in CI/CD practices and tools.
- Proficiency in Azure Databricks, Spark, and data engineering methodologies.
- Familiarity with data architecture concepts, such as the medallion model.
- Good knowledge of relational database management systems (RDBMS).

Tools:
- Programming Languages: Python
- CI/CD Platforms: Azure DevOps
- Data Engineering Frameworks: Azure Databricks, PySpark, Spark Structured Streaming
- Database Systems: Azure SQL Database
- Architecture: Databricks Lakehouse

Nice to Have:
- Azure Databricks Unity Catalog for advanced data governance
- Experience with Azure Data Factory for data orchestration
- Knowledge of Kafka for real-time data streaming
- Experience extracting SAP data using change data capture (CDC) techniques
- Familiarity with SAP HANA for database management and integration

Start: 11/2024
Duration: 12 Months+
Location: REMOTE

Categorie