Experience: 5+ years
* Design, develop, and maintain scalable ETL/ELT pipelines using Databricks, PySpark, and Python to support enterprise analytics, AI, and application use cases.
* Build and manage curated data layers following Lakehouse Medallion architecture best practices (Bronze / Silver / Gold).
* Develop reusable, modular data transformation frameworks to accelerate delivery across domains.
* Design and develop Databricks-native data applications, including notebook-based apps, Databricks dashboards, and interactive data experiences for analytics and business users.
* Build data APIs, parameterized pipelines, and app-integrated data services leveraging Databricks and Lakehouse capabilities.
* Partner with analytics, AI, and application teams to embed data and insights directly into workflows and applications.
* Ensure Databricks apps meet performance, security, governance, and usability standards.
* Optimize Apache Spark jobs and Databricks workloads for performance, cost efficiency, scalability, and reliability.
* Proactively address challenges related to data volume, schema evolution, and compute optimization.
* Implement robust data quality checks, validations, and anomaly detection within pipelines and apps.
* Own and support production data pipelines and Databricks applications, including monitoring, troubleshooting, and root-cause analysis.
* Ensure high availability, data correctness, and SLA adherence for business-critical datasets and apps.
* Contribute to observability, alerting, and operational automation.
* Collaborate with BI, analytics, AI/ML, platform, and application teams to deliver end-to-end data solutions.
* Enable data consumption across dashboards, reports, Databricks apps, AI models, APIs, and downstream applications.
* Translate business and analytical requirements into well-designed data pipelines and data applications.
* Act as a technical leader and mentor, defining best practices for data engineering and Databricks app development.
* Participate in architecture reviews, design discussions, and technical roadmaps.
* Continuously evaluate and adopt modern Databricks features, GenAI capabilities, and automation patterns to improve developer productivity.
* 5+ years of hands-on data engineering experience, owning production-grade pipelines and data solutions.
* Proven hands-on experience working with Databricks in production, including Databricks application development.
* Experience building and supporting Databricks notebooks, dashboards, and data-driven applications.
* Experience operating and supporting data pipelines and data apps in production environments.
* Solid understanding of data quality, reliability, security, and governance.
* Experience with AWS cloud services (e.g., S3, IAM, EC2, Glue, or equivalent).
* Exposure to Unity Catalog, access controls, metadata management, and governed data sharing.
* Experience with streaming data pipelines (e.g., Structured Streaming, Kafka).
* Familiarity with CI/CD, Git-based workflows, and Data/Analytics DevOps.
* Experience enabling BI, AI/ML, or application-embedded analytics using Databricks.
* Owns complex data pipelines and Databricks applications end-to-end with minimal oversight.
* Drives improvements in performance, reliability, cost efficiency, and usability across data and app layers.
* Influences architecture, standards, and best practices beyond immediate assignments.
* Serves as a trusted technical partner to analytics, AI, platform, and application teams.
Search Staff Data Engineer / Full-Stack Data Developer (Databricks / Python) jobs near San Diego, CA → Browse all live jobs
This posting was published by Qualcomm on their own careers system and is shown here with a direct link to apply there. Employers: for corrections or removal, contact jobs@veritahire.com.