Data Engineer Fte, 150-160K, 20% Bonus, 20K Signon, 401K, Benefits -- Direct Full Time Hire By Client --

Data Engineer Fte, 150-160K, 20% Bonus, 20K Signon, 401K, Benefits -- Direct Full Time Hire By Client --
Company:

Projas Technologies, Llc


Place:

California


Area:

Programmer

Data Engineer Fte, 150-160K, 20% Bonus, 20K Signon, 401K, Benefits -- Direct Full Time Hire By Client --

Details of the offer

Full time employee role (direct hire) with our customer. Work can be done offsite as of now until COVID, later work needs to be done onsite in bay area
Responsibilities
Assemble large, complex data sets in the format fit for each use case
Strong knowledge on file based data lake platform. Incremental updates to data lake
Delta Lake technology knowledge is highly desired
Parquet based data lake technology expertise
Write generic Python/Pyspark modules for processing data from various data sources (XML, Parquet, CSV, Relational)
Demonstrable experience architecting, developing and optimizing ETL pipelines using Python, Spark, EMR and Airflow
Develop and optimize big data pipelines for data scientists (requires a basic understanding of data science concepts and ML)
Research and recommend new innovative methods and systems to manage data for business improvement
Participate in internal governance to drive the data quality business cycle and roadmap
Required Skills
Python, Spark, ETL/Data engineering, S3 based datalake in AWS. Development and management of Airflow based data flows
Bachelor s or Master s degree in computer science or software engineering
3+ years of programming experience (including functional programming); must be advanced in Python
Experience building and optimizing big data pipelines using Spark
Experience with AWS cloud services: S3, EC2, EMR, RDS, Redshift, PySpark, Airflow
Experience with relational SQL and NoSQL databases, including Postgres
Solid understanding of how to design robust data workflows including optimization and user experience
Strong analytical and problem-solving skills
Excellent oral and written communication skills
Able to work in teams and collaborate with others to clarify requirements
Strong co-ordination and project management skills to handle complex projects
Experience developing and working with XML, JSON, and external web services
Strong experience working with a variety of relational SQL and NoSQL databases
Strong experience working with big data tools: Hadoop, Spark, Kafka, etc.
Experience with at least one cloud provider solution (AWS, GCP, Azure)
Strong experience with object-oriented/object function scripting languages: Python, Java, C++, Scala, etc.
Ability to work in Linux environment
Experience working with APIs
Strong knowledge of data pipeline and workflow management tools
Expertise in standard software engineering methodology, e.g. unit testing, code reviews, design documentation
Experience creating ETL processes that prepare data for consumption appropriately
Experience in setting up, maintaining and optimizing databases for production usage in reporting, analysis and ML applications
Working in a collaborative environment and interacting effectively with technical and non-technical team members equally well
Relevant working experience with Docker and Kubernetes preferred
Ability to work with ML frameworks preferred

Requirements


Knowledges:

  • Basic
  • Python
  • SQL
  • Linux

Is data engineer i

LOC_1300_MKT:Wanamaker BuildingReq ID:42481 Shift: Days Employment Status:Regular : Full Time Job Summary 13; We are looking for a Data Engineer to join our...


Pennsylvania

Published a month ago

Java developer

Company Name:Kroger General Office Position Type:Employee FLSA Status:Exempt Line of Business:Infrastructure Services See what life is like at Kroger...


Ohio

Published a month ago

Services developer

Company Name:Kroger General Office Position Type:Employee FLSA Status:Exempt Line of Business:Digital Technology See what life is like at Kroger Technology...


Ohio

Published a month ago

Software developer/sr. software developer job

Company Information 13; PACCAR is a Fortune 500 company established in 1905. PACCAR Inc is recognized as a global leader in the commercial vehicle, financial...


Washington

Published a month ago