AI Data Engineer II

Candidates must be willing to participate in at least one in-person interview, which may include a live whiteboarding or technical assessment session.

You will address the challenge of transforming complex, fragmented data requirements into scalable, production-ready AI infrastructure using advanced Databricks and Spark architectures. By implementing rigorous software engineering disciplines and automated CI/CD workflows, you will eliminate bottlenecks in data delivery and ensure the reliability of mission-critical pipelines.

Your role is pivotal in harmonizing cross-functional goals with high-quality code standards to drive the next generation of our data platform's evolution.

- Deliver scalable, high-performance data pipelines using Databricks and Spark that meet rigorous departmental OKRs for performance and cost-efficiency

- Build fully automated CI/CD workflows within Gitlab to reduce deployment friction and ensure 100% version-controlled data infrastructure

- Apply SOLID engineering principles and modular design to create a reusable testing framework that guarantees data pipeline stability and quality

- Foster a culture of excellence by leading technical code reviews and mentoring peers to elevate the team's overall software engineering maturity

- Integrate AI-driven automation tools to proactively monitor pipeline health and optimize resource allocation across the AWS ecosystem

- Facilitate seamless collaboration between Data Science and Product teams to transform experimental models into fault-tolerant production solutions

- Advanced proficiency in Python and SQL alongside a deep understanding of distributed systems architecture and modern data patterns

- Expertise in Databricks, Spark, and Delta Lake orchestration to manage large-scale, high-velocity data environments

- A strong foundation in DevOps methodologies, specifically regarding infrastructure-as-code and containerization using Docker or Kubernetes

- AI Application literacy, with the ability to leverage machine learning libraries and NLP frameworks to enhance data processing capabilities

- Proven capability in applying design patterns and testing frameworks to ensure the integrity of complex software ecosystems

- Critical experience in building and managing highly available, fault-tolerant systems within an enterprise AWS environment

- Background in handling sensitive data and maintaining strict security protocols

- Minimum Education: Bachelor's Degree in Computer Science, Electrical Engineering, or a related field

- Minimum Experience: 3+ years of experience in data engineering

- Required Technical Skills: Must have at least 3+ years of experience with:

- AI Platforms - Databricks and Spark (including Delta Lake)