Azure data engineer

Job Description

Please find below the company andJDdetails.

Company Name-StatusNeo

Company Website-https://statusneo.com/

LinkedInLink-https://www.linkedin.com/company/statusneo/

Client Name-EY

Employment Type: Full-time

Job Title:Azure Data Engineer

Work Location: EY Bangalore

Key Responsibility:

We are currently seeking a seasoned Azure Data Engineer with proven experience in Databricks PySpark to join our team of professionals. The successful candidate will play a key role in our Metadata Management team, providing cutting-edge data management strategies and leveraging their strong analytical skills to solve complex problems.

Primary Skills / Must have:

1. Should have strong programming skills in Python.
2. Experience in creating large scale data processing pipelines using a Python and Spark based framework.
3. Work with different aspects of the Spark ecosystem, including Spark SQL, DataFrames, Datasets, and Streaming
4. Should possess strong SQL skills.
5. Excellent understanding of Unix ecosystem and should have experience in creating the shell scripts.
6. Excellent understanding of Hive/Hadoop ecosystem.
7. Solid Understanding of data engineering concepts and best practices.
8. Excellent understanding of Job Scheduling mechanisms like Autosys, TWS.
9. Excellent problem solving and analytical skills.
10. Excellent verbal and written communication skills.
11. Experience in optimizing large data loads.
12. Secondary Skills / Desired skills
13. Exposure to an Agile Development environment would be a plus.
14. Strong understanding of Data warehousing domain.
15. Ability to architect an ETL solution and data conversion strategy.
16. Good understanding of dimensional modelling.
17. Roles and Responsibilities
18. As an Azure Data Engineer, the candidate is expected to
19. Design and develop highly optimized and scalable ETL applications using Python and Spark.
20. Undertaking end-to-end project delivery (from inception to post-implementation support), including review and finalization of business requirements, creation of functional specifications and/or system designs, and ensuring that end-solution meets business needs and expectations.
21. Development of new transformation processes to load data from source to target, or performance tuning of existing ETL code (mappings, sessions).
22. Analysis of existing designs and interfaces and applying design modifications or enhancements
23. Coding and documenting data processing scripts and stored procedures.
24. Providing business insights and analysis findings for ad-hoc data requests
25. Testing software components and complete solutions (including debugging and troubleshooting) and preparing migration documentation.
26. Providing reporting-line transparency through periodic updates on project or task status