Overview
We are looking for a highly experienced Lead Data Engineering Developer to join our team. This role involves building, optimizing, and maintaining complex data pipelines, leveraging your expertise in Python, Spark, Hive, and cloud environments. You'll play a critical role in delivering high-quality data solutions for dynamic projects.
Responsibilities
- Design, develop, and maintain complex data pipelines to support large-scale data processing
- Write clean, efficient, and test-driven code using Python and SQL
- Implement unit testing with mock frameworks as part of the development process
- Work extensively with Spark and Hive for data processing and transformation
- Collaborate with teams to optimize data structures and algorithms for performance
- Utilize AWS components (EC2, S3, Lambda, DynamoDB, SQS, RDS) for cloud-based data processing and storage
- Ensure the scalability and reliability of data workflows and pipelines
Requirements
At least 7 years of hands-on experience in data engineering or related fieldsStrong coding skills in Python with deep expertise in SQLExperience with test-driven development and writing unit tests using mock frameworksProven ability to design and manage complex data pipelinesExtensive knowledge of Spark and Hive data processing frameworksSolid understanding of data structures and algorithmsHands-on experience with AWS cloud services including EC2, S3, Lambda, DynamoDB, SQS, and RDSEnglish language proficiency at B2+ levelNice to have
Experience with data orchestration tools like AirflowFamiliarity with other cloud platforms or additional AWS servicesBackground in optimizing data pipelines for machine learning or analytics use casesWe offer
International projects with top brandsWork with global teams of highly skilled, diverse peersEmployee financial programsPaid time off and sick leaveUpskilling, reskilling and certification coursesUnlimited access to the LinkedIn Learning library and 22,000+ coursesGlobal career opportunitiesVolunteer and community involvement opportunitiesEPAM Employee GroupsAward-winning culture recognized by Glassdoor, Newsweek and LinkedIn#J-18808-Ljbffr