Lead Software Engineer, Full Stack

About the Team: We are seeking a Lead Software Engineer to own the lifecycle of a massive, high-velocity data ecosystem. We currently manage a large Cloudera-based stack (HBase, Kafka, NiFi) hosted on AWS EC2 and over the next 24 months, we are migrating this entire infrastructure to a modern, AWS-native serverless/managed architecture stack.

Maintain & Optimize: Ensure the current Cloudera/HBase/Kafka production environment remains performant and stable during the transition.

- Lead the Migration: Design and execute the phased migration of billions of records to AWS Stack with zero data loss and minimal downtime.

- Enable Application Teams: Work directly with internal application developers to refactor data consumers, ensuring a smooth cutover from HBase APIs to new AWS-native access patterns.

- Enforce Compliance: Implement and maintain strict data governance and security protocols (encryption, auditing, and lineage) to ensure the system remains compliant throughout the migration.

- Modernize Ingestion: Transition legacy NiFi and Kafka workflows into streamlined, cloud-native pipelines using AWS MSK, Lambda, and Glue.

- At least 4 years of experience in software engineering (Internship experience does not apply)

- At least 1 year experience with cloud computing (AWS, Microsoft Azure, Google Cloud)

- 5+ years of experience with the Hadoop ecosystem, specifically HBase (RegionServer tuning, schema design) and NiFi.

- 3+ years of experience architecting and deploying AWS DocumentDB or DynamoDB at a multi-billion record scale.

- 3+ years of experience with Apache Kafka (both self-managed and AWS MSK) and AWS Lambda.

- 3+ years of experience maintaining live production systems.

- 1+ years of experience leveraging interactive AI tooling to accelerate productivity, utilizing capabilities beyond basic code completion

- AWS Certified CloudOps Engineer (Associate) or SysOps Administrator.

At this time, Capital One will not sponsor a new applicant for employment authorization, or offer any immigration related support for this position (e.g. H1B, F-1 OPT, F-1 STEM OPT, F-1 CPT, J-1, TN, E-3, and O-1, or any other forms of work authorization that require immigration support from an employer).