The selected candidate will perform the following duties: Daily Tasks:
• Design, develop, and maintain robust and scalable data pipelines for extracting, transforming, and loading data from various sources, including databases, and streaming platforms.
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and translate them into technical specifications.
• Resolve complex data integration platform related issues. Coordinate with vendors for product related issues, patches, bug fixes and upgrades.
• Implement data governance best practices to ensure data quality, consistency, and security throughout the data lifecycle.
• Document data pipelines, and technical specifications to facilitate knowledge sharing and collaboration among team members.
• Collaborate with appropriate stakeholders on defining and implementing technology direction/roadmap within data integration technology space.
• Optimize data pipelines for performance, scalability, and reliability, using techniques such as parallel processing, caching, and error handling.
• Meet with application teams to understand requirements and recommend high-level direction and approach to upcoming projects.
• Administer Data Integration product platforms - including running periodic backups and performing deployments.
• Participate in code reviews, agile ceremonies, and cross-functional meetings to contribute to the overall success of the team and the organization.
Position Success Criteria (Desired) - 'WANTS' Required Work Experience:
• Bachelor's degree in computer science, information technology, or a related field.
• 5+ years of experience in data engineering, database design, ETL processes, and data warehousing.
• 3+ in programming languages such as Python, Java, or Scala.
• 3+ years of experience with AWS tools and technologies (S3, EMR, Glue, Athena, RedShift).
• Strong knowledge of data storage and processing technologies, including databases (SQL and NoSQL), data lakes, and distributed computing frameworks (e.g., Hadoop, Spark).
• Strong knowledge with one or more Database platforms (DB2, SQL Server, Postgres).
• Solid understanding of data security, privacy, and compliance.
• Excellent problem-solving and communication skills.
• Ability to work collaboratively in cross-functional teams.
• Attention to detail and a commitment to data quality.
• Continuous learning mindset to keep up with evolving technologies and best practices in data engineering.
