Lead Data Engineer - E-Commerce Technologies

Job Description

Skills: Big Data Technologies, Data Modeling, ETL Processes, Data Warehousing, SQL, Python, Cloud Platforms, Data Governance,

• Architect and scale a state-of-the-art data infrastructure capable of handling batch and real-time data processing needs with unparalleled performance.
• Collaborate closely with the data science team to oversee data systems, ensuring accurate monitoring and insightful analysis of business processes.
• Design and implement robust ETL (Extract, Transform, Load) data pipelines, optimizing data flow and accessibility.
• Develop comprehensive backend data solutions to bolster microservices architecture, ensuring seamless data integration and management.
• Engineer and manage integrations with third-party e-commerce platforms, expanding data ecosystem and capabilities.

• A background at a product-centric software company directly contributing to building products (vs providing data/analytics to business stakeholders)
• Software development experience with a focus on data engineering.
• Extensive experience in building ETL pipelines using tools such as Apache Spark, Databricks, or Hadoop.
• Proficiency in Python or Java, with a deep understanding of software engineering best practices.
• Expertise in distributed computing and data modeling, capable of designing scalable data systems.
• Proficiency with NoSQL databases, including MongoDB, Cassandra, DynamoDB, and CosmosDB.
• Proficiency in real-time stream processing systems such as Kafka, AWS Kinesis, or GCP Data Flow.
• Skilled in utilizing caching and search technologies like Redis, Elasticsearch, or Solr.
• Familiarity with message queuing systems, including RabbitMQ, AWS SQS, or GCP Cloud Tasks.
• Experience with Delta Lake, Parquet files, and AWS, GCP, or Azure cloud services.
• A strong advocate for Test Driven Development (TDD) and experienced in version control using Git platforms like GitHub or Bitbucket.
• Experience at a startup is preferred.
• Experience with consumer e-commerce data/technologies would be a bonus.

• Work Location: Remote
• 5 days working