Job Description

Role: Lead Data Engineer


Location: New York


Mode of Work: Onsite


60/hour.

Responsibilities


- Lead the design and implementation of a robust, scalable, and reusable data ingestion framework using Microsoft Fabric


- Building a metadata-driven solution for ingesting data from CSV sources into the Data Lake


- Work closely with stakeholders to understand data sources, ensure compliance, and guide development teams to deliver efficient and adaptable data ingestion pipelines.


Must Have:


- 8-10 Yrs of Experience in Building data architecture using various data engineering techniques


- In-depth knowledge and hands-on experience with Microsoft Fabric components, including Lakehouse, One Lake, Delta Lake format, Fabric Pipelines, Dataflows Gen2, and Spark notebooks/Synapse Data Engineering


- Strong understanding of data lake concepts, including optimal use of One Lake for data storage, Delta Lake


- Experience in designing and implementing metadata-driven solutions for data ingestion


- Expertise in various data ingestion patterns, including batch processing (specifically for CSV files)


- Proficient in data modeling standards, potentially including Medallion architecture (Bronze, Silver, Gold layers)


- Strong proficiency in PySpark for data processing and transformation within Fabric Spark environments


- Advanced SQL skills for querying data, managing database interactions


- Strong analytical and problem-solving skills to assess data requirements, troubleshooting issues


Good to Have:


- Proficiency in PySpark for data processing and transformation within Fabric Spark environments


- Understanding of data governance principles, data security measures


- Knowledge of monitoring data pipeline performance, troubleshooting issues

Job Tags

Similar Jobs