Data Engineer

This role is for one of the Weekday's clients

If you are passionate about data engineering and cloud-based solutions, this role is for you.

Requirements

• Data Pipeline Development: Design, build, and maintain scalable ETL/ELT pipelines using Python and PySpark.
• Big Data Processing: Work with large datasets and optimize processing using distributed computing frameworks.
• SQL & Database Management: Write efficient SQL queries to extract, transform, and load data across various relational and non-relational databases.
• Azure Data Engineering: Leverage Azure Data Factory, Azure Databricks, Azure Synapse, and Azure Storage to manage and optimize data workflows in the cloud.
• Performance Optimization: Improve the efficiency and performance of data pipelines by tuning queries, caching, and leveraging big data best practices.
• Data Quality & Governance: Implement data validation, cleansing, and integrity checks to ensure high-quality data is available for business use.
• Collaboration: Work closely with data scientists, analysts, and software engineers to support data-driven decision-making and real-time analytics.
• Automation & Monitoring: Develop automated data ingestion and processing workflows, and set up monitoring and alerting mechanisms for data pipelines.

• 3+ years of experience in data engineering or a related field.
• Strong programming skills in Python and experience with PySpark for big data processing.
• Expertise in SQL for data querying, optimization, and transformation.
• Hands-on experience with Azure cloud services like Azure Data Factory, Azure Databricks, Azure Synapse Analytics, and Azure Storage.
• Experience with data modeling, ETL processes, and data warehouse concepts.
• Familiarity with orchestration tools such as Apache Airflow or Azure-native alternatives.
• Knowledge of distributed computing frameworks and big data technologies.
• Strong problem-solving skills and ability to optimize data workflows for efficiency and performance.
• Excellent communication and teamwork skills, with a collaborative mindset.

• Experience with containerization and orchestration (Docker, Kubernetes).
• Understanding of data security, governance, and compliance in a cloud environment.
• Familiarity with CI/CD pipelines and infrastructure-as-code (Terraform, ARM templates).
• Exposure to real-time data processing using Kafka or similar streaming platforms.