Lead Data Engineer Job at Maintec Technologies, New York, NY

UnVsVEhUR0V6ZFVFY3ZTbXlRanN1ZE5uQlE9PQ==
  • Maintec Technologies
  • New York, NY

Job Description

Role: Lead Data Engineer


Location: New York


Mode of Work: Onsite


60/hour.

Responsibilities


- Lead the design and implementation of a robust, scalable, and reusable data ingestion framework using Microsoft Fabric


- Building a metadata-driven solution for ingesting data from CSV sources into the Data Lake


- Work closely with stakeholders to understand data sources, ensure compliance, and guide development teams to deliver efficient and adaptable data ingestion pipelines.


Must Have:


- 8-10 Yrs of Experience in Building data architecture using various data engineering techniques


- In-depth knowledge and hands-on experience with Microsoft Fabric components, including Lakehouse, One Lake, Delta Lake format, Fabric Pipelines, Dataflows Gen2, and Spark notebooks/Synapse Data Engineering


- Strong understanding of data lake concepts, including optimal use of One Lake for data storage, Delta Lake


- Experience in designing and implementing metadata-driven solutions for data ingestion


- Expertise in various data ingestion patterns, including batch processing (specifically for CSV files)


- Proficient in data modeling standards, potentially including Medallion architecture (Bronze, Silver, Gold layers)


- Strong proficiency in PySpark for data processing and transformation within Fabric Spark environments


- Advanced SQL skills for querying data, managing database interactions


- Strong analytical and problem-solving skills to assess data requirements, troubleshooting issues


Good to Have:


- Proficiency in PySpark for data processing and transformation within Fabric Spark environments


- Understanding of data governance principles, data security measures


- Knowledge of monitoring data pipeline performance, troubleshooting issues

Job Tags

Similar Jobs

House Keeper

Housekeeper Wanted - Job Opportunity: North Attleboro House Cleaner Job at House Keeper

What We're Looking For:Nice to meet you! My name is Tommy. I am interviewing a home cleaner working in North Attleboro, Massachusetts. My aim is to find a capable, polite employee who has bi-weekly availability.Services And Training:My home requires surface polishing... 

Trinity Health

Medical Office Manager Job at Trinity Health

 ...Office Manager Employment Type: Full time Shift: Outpatient, Full time M-F Location: Albany, NY If you are looking for a Management...  ...day-to-day practice operations including maintenance of medical records, human resource management, property and facilities,... 

Yew Chung International School

K8 Teacher (English) Job at Yew Chung International School

Job Title: K-8 Teacher (English)Program: K-8Reports to: Co-PrincipalsFLSA Status: ExemptCompensation: $68,723 - $105,549 AnnuallyPosition SummaryThe K-8 Teacher at Yew Chung International School Silicon Valley plays a vital role in providing a comprehensive education... 

MH Energy

Class A Container Delivery Driver Job Job at MH Energy

Class A Container Delivery Driver JobCDL driver using flatbed to deliver sandboxes to oilfield well sites. Pay: $1865 per WeekWeekly pay, 100% paid health benefits, 401k, paid trainingHow to apply for this Driving JobFill out a Gary's Job Board application. That... 

Alliance Center for Education

School Bus Driver Job at Alliance Center for Education

 ...School Bus Driver Alliance Center for Education - CHARLOTTE, NC 28205 Overview Job Shift Day Description Alliance Center for Education is committed to providing a safe and nurturing environment for children to learn and grow. We are looking for a reliable...