Big Data Engineer

Job Description

Job Title: Data Engineer

Job Summary:

We are looking for a Senior Data Engineer with expertise in building scalable data pipelines, designing data frameworks, and working with modern data technologies. The ideal candidate should have strong experience with Scala, Apache Spark, and streaming technologies such as Kafka, Kinesis, or Flink.

Key Responsibilities:

• Design, develop, and maintain data pipelines to support business and analytics needs.
• Build data frameworks for unit testing, data lineage tracking, and automation.
• Optimize and enhance existing data processing workflows for scalability and performance.
• Work with Apache Spark to process large-scale datasets efficiently.
• Implement real-time data streaming solutions using Kafka, Kinesis, or Flink.
• Collaborate with cross-functional teams, including Data Scientists, Analysts, and Software Engineers, to drive data-driven decision-making.

Required Qualifications:

• 5+ years of experience in building data pipelines.
• 5+ years of experience in developing data frameworks for unit testing, data lineage tracking, and automation.
• Proficiency in Scala is required.
• Strong working knowledge of Apache Spark.
• Experience with streaming technologies such as Kafka, Kinesis, or Flink.
• Familiarity with cloud platforms (AWS, Azure, or GCP) is a plus.

Preferred Qualifications:

• Experience with big data processing frameworks such as Hadoop or Databricks.
• Strong understanding of distributed computing principles.
• Knowledge of data modeling, ETL development, and performance tuning.