Lead Data Engineer

Lead Data Engineer
Company:

E-Solutions, Inc.


Lead Data Engineer

Details of the offer

Lead Data Engineer
Location Appleton,WI
Long term Contract

Description: Architects, designs and provides governance of large-scale Data Integration solutions that deliver business value across the organization.

Collects the data from a variety of traditional and non-traditional sources, stores it in a data repository, cleanses and integrates the data (data prep) for analysis.

Designers, builders and managers of the information and big data infrastructure. They co-develop the architecture that helps analyze and process data that the organization required and further optimize those systems to perform smoothly.

Evaluates, compares and improves the different approaches including design patterns innovation, data lifecycle design, data ontology alignment, annotated datasets, and elastic search approaches.

Prepares the data for the data scientist exploration and discovery process. For example, we have data containing 30 attributes where two attributes are used to compute another attribute (for example, an index), and that computed feature is used for further analysis. In this way, the data could be changed according to the requirement of the applied model.

"Wrangles" the data, sometimes called data munging or data schmarzing (just checking to see if you're reading) which transforms, maps and "munges? the raw data using algorithms (e.g. sorting, parsing) into predefined data structures, and depositing the results into a data lake for the downstream analytics.

Much of the data engineering work tends to be domain specific (requires an understanding of the unique meaning of the data given the business or operational problem being solved) and involves the business stakeholders. Insurance background is preferred

Searching for a Data Engineer with 3+ years of experience in implementing modern data architectures and 2+ years of a Data Streaming. We have a data streaming platform to enable frictionless data flow from business systems and 3rd party sources using publish/subscribe model.

WHAT WE NEED YOU TO DO

Partner on the design, deployment, and persistence of our new reporting and analytics platform that unifies the data across organization, using Confluent Kafka.

Develop containerized microservices (Python/Flask ) to transform and enrich topic data

Lead by example best practices reusable materialization of data

Partner on setting standards, implementing tools, and creating documentation for self-serve data pipeline services supporting core engineering and professional services use cases.

Work with existing engineering teams to become data producers and consumers to and from Confluent Kafka

Work with business intelligence engineers, analysts and Data Scientists

WHAT WE'RE LOOKING FOR

Strong experience with event streaming such as (Kafka or Amazon Kinesis)

Experience with SQL and NoSQL databases (PostgreSQL, MongoDB, etc)

Desire to show ownership of problems you identify and proven ability to empower others to get more done

Proficient delivering services within AWS

The ability to communicate cross-functionally, derive requirements and architect shared datasets; ability to synthesize, simplify and explain complex problems to different types of audiences, including executives WHAT GIVES YOU AN EDGE

Experience with Confluent Kafka

Demonstrated ability to develop containerized microservices (Docker with Kubernetes)

Experience developing serverless data pipelines

(e-Solutions provides equal employment opportunities (EEO) to all employees and applicants for employment without regard to race, color, religion, gender, sexual orientation, gender identity or expression, national origin, age, disability, genetic information, marital status, amnesty, or status as a covered veteran in accordance with applicable federal, state and local laws. We especially invite women, minorities, veterans, and individuals with disabilities to apply. EEO/AA/M/F/Vet/Disability)

Requirements


Knowledges:

  • SQL

Java developer

Location: Kingstowne, VA About Us: Radiant Digital delivers technology consulting and business solutions for commercial and government clients. Our flexible...


From Radiant Digital - Virginia

Published a month ago

Fullstack developer (50/50 ui and backend)

Note: Here is a new req in New Hampshire for a Full-Stack Engineer. This is 50/50 UI and backend development. Needs strong Angular and strong Java We are...


From Resiliency Llc - New Hampshire

Published a month ago

Oracle scm lead (ptm)

VDart – We are a Global Information Technology Services & Workforce Solutions firm headquartered out of Atlanta, GA with presence in US, Canada, MX, UK, Belgium...


From Vdart, Inc. - New Jersey

Published a month ago

Business system analyst

Cerebra Consulting Inc is a System Integrator and IT Services Solution provider with a focus on Big Data, Business Analytics, Cloud Solutions, Amazon Web...


From Cerebra Consulting Inc - Michigan

Published a month ago