Databricks Engineer

About Position:

We are seeking a skilled Databricks Engineer AI & Analytics to design, develop, and deliver scalable cloud-based data solutions that support modern analytics, business intelligence, and AI-driven initiatives. The ideal candidate will have strong expertise in Databricks, Python, PySpark, SQL, and cloud data platforms, along with exposure to Generative AI technologies.

• Role: Databricks Engineer AI & Analytics
• Location: Bengaluru
• Experience: 6 to 8 Years
• Job Type: Full Time Employment

What You'll Do:

• Design, build, and optimize batch and real-time data pipelines using Python, PySpark, SQL, and Databricks.
• Develop scalable ETL and ELT frameworks to support enterprise data and analytics platforms.
• Perform source-to-target analysis, data mapping, data profiling, and requirement gathering activities.
• Design and implement data transformation and data integration solutions.
• Support cloud-based data platforms and architecture enhancement initiatives on AWS and Azure.
• Build and maintain data pipelines that support analytics, reporting, machine learning, and AI workloads.
• Collaborate with business stakeholders, architects, and project teams to understand business requirements and deliver data solutions.
• Ensure data quality, consistency, governance, and security across data platforms.
• Participate in data modeling, data architecture, and performance optimization activities.
• Develop and support Generative AI and Large Language Model (LLM) use cases.
• Contribute to Retrieval-Augmented Generation (RAG) implementations and AI-powered productivity solutions.
• Support deployment and operationalization of AI-ready data platforms.
• Monitor pipeline performance and troubleshoot data processing issues.
• Implement best practices around DevOps, DataOps, DevSecOps, and Agile delivery methodologies.
• Participate in code reviews, testing activities, and continuous improvement initiatives.
• Create technical documentation and support knowledge-sharing activities.

Expertise You'll Bring:

• 6-8 years of experience in Data Engineering and modern data platform development.
• Strong hands-on experience with Databricks.
• Expertise in Python, PySpark, and SQL development.
• Experience designing and building large-scale ETL and ELT pipelines.
• Strong understanding of distributed data processing frameworks.
• Experience working with structured and unstructured datasets.
• Knowledge of data warehousing, dimensional modeling, and data lake concepts.
• Experience implementing scalable and high-performance data solutions.
• Hands-on experience with AWS or Azure cloud platforms.
• Experience working with cloud-native data and analytics services.
• Understanding of cloud architecture principles and best practices.
• Knowledge of data storage, data ingestion, and cloud-based processing frameworks.
• Experience supporting enterprise-scale analytics and BI solutions.
• Strong expertise in Databricks workspace development and optimization.
• Experience with Apache Spark and distributed computing concepts.
• Knowledge of Delta Lake and modern lakehouse architecture.
• Experience optimizing PySpark jobs and large-scale workloads.
• Familiarity with performance tuning and workload management.
• Working knowledge of Generative AI technologies and enterprise AI use cases.
• Understanding of Large Language Models (LLMs) and GPT-based solutions.
• Experience with Retrieval-Augmented Generation (RAG) concepts and implementations.
• Exposure to embeddings, vector search, and semantic retrieval frameworks.
• Ability to support AI-enabled analytics and intelligent data solutions.
• Experience working in Agile development environments.
• Ability to collaborate effectively with globally distributed teams.
• Strong stakeholder engagement and communication skills.
• Experience gathering requirements and translating them into technical solutions.
• Strong analytical and problem-solving capabilities.
• Ability to manage multiple priorities and deliver high-quality outcomes.
• Experience with DevOps, DataOps, and DevSecOps practices.
• Knowledge of CI/CD pipelines and automated deployment processes.
• Experience with testing, monitoring, and operational support activities.
• Strong focus on data quality, security, governance, and compliance.

Benefits:

• Competitive salary and benefits package
• Culture focused on talent development with quarterly growth opportunities and company-sponsored higher education and certifications
• Opportunity to work with cutting-edge technologies
• Employee engagement initiatives such as project parties, flexible work hours, and Long Service awards
• Annual health check-ups
• Insurance coverage: group term life, personal accident, and Mediclaim hospitalization for self, spouse, two children, and parents

Values-Driven, People-Centric & Inclusive Work Environment:

Persistent is dedicated to fostering diversity and inclusion in the workplace. We invite applications from all qualified individuals, including those with disabilities, and regardless of gender or gender preference. We welcome diverse candidates from all backgrounds.

• We support hybrid work and flexible hours to fit diverse lifestyles.
• Our office is accessibility-friendly, with ergonomic setups and assistive technologies to support employees with physical disabilities.
• If you are a person with disabilities and have specific requirements, please inform us during the application process or at any time during your employment

Let's unleash your full potential at Persistent - persistent.com/careers

Persistent is an Equal Opportunity Employer and prohibits discrimination and harassment of any kind.