LightFeather is currently seeking a skilled Databricks Engineer to join our dynamic team and play a pivotal role in our data engineering efforts. The successful candidate will be responsible for designing, implementing, and optimizing data pipelines that integrate data from multiple sources into Databricks. In this role, your primary focus will be to ensure seamless data flow and enable efficient data processing, storage, and analysis.
This Position is Full Time, Remote.
Responsibilities:
Develop and maintain ETL processes to extract, transform, and load data from various sources including Google Analytics (GA4), Splunk, Medallion, and others into Databricks. Design and implement data pipelines and workflows using Databricks, ensuring scalability, reliability, and performance. Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and provide appropriate data solutions. Develop and maintain Python Notebooks within Databricks for data analysis and processing, optimizing data workflows for efficiency and accuracy. Optimize and tune data processing jobs for performance and cost-efficiency. Ensure data quality and consistency through robust data validation and cleansing techniques. Monitor and troubleshoot data pipeline issues, ensuring timely resolution and minimal downtime. Leverage Terraform for infrastructure as code (IaC) practices to automate and manage infrastructure provisioning and scaling. Stay updated with the latest trends and advancements in data engineering and Databricks technologies. Qualifications:
US Citizenship. Active IRS clearance - Public Trust or higher. Bachelor's degree preferred or equivalent experience. 5+ years of hands-on experience with Databricks, including designing and managing large-scale data pipelines. Proficiency in ETL tools and techniques, with a strong understanding of data integration from sources like Google Analytics (GA4), Splunk, and Medallion. Solid experience with SQL, Python, and Spark for data processing and transformation. Familiarity with cloud platforms such as AWS, Azure, or Google Cloud, with a focus on their data services. Experience with other big data technologies such as Apache Airflow. Knowledge of data warehousing concepts and best practices. Familiarity with data visualization tools such as Tableau, Power BI, or Looker. Proven experience in designing and deploying Databricks infrastructure on cloud platforms, preferably Amazon AWS. Deep understanding of Apache Spark, Delta Lake, and their integration within the Databricks environment. Proficient in Terraform for implementing infrastructure as code (IaC) solutions. Strong expertise in Python, especially in developing Notebooks for data analysis within Databricks. Demonstrated ability to design and implement complex data pipelines with ETL processes for large-scale data aggregation and analysis. Knowledge of best practices for infrastructure scaling and data management, with a keen focus on security and robustness. Strong problem-solving skills and the ability to troubleshoot complex data issues. Excellent communication and collaboration skills to work effectively with cross-functional teams. Why Join LightFeather?
You'll be part of a team dedicated to meaningful impact, working on solutions that address mission-critical needs. Experience variety, fulfillment, and the opportunity to work with some of the best in the industry. We are committed to fostering a diverse and inclusive environment where everyone is valued and respected.
Commitment to Diversity
LightFeather is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees, regardless of race, color, religion, sex, sexual orientation, gender identity, national origin, veteran, or disability status.
