DataBricks Data Engineering

Required Skills and Experience

• Hands-on experience developing data engineering solutions on Databricks.
• Strong proficiency in Python PySpark and SQL including complex transformations and query optimization.
• Working knowledge of Apache Spark execution partitioning joins caching and performance troubleshooting.
• Experience with Delta Lake including merge operations schema management and transactional data processing.
• Experience building ETL/ELT pipelines implementing incremental loads and applying data warehousing and dimensional modeling concepts.
• Experience with at least one cloud platformAzure AWS or Google Cloudand its storage and identity services.
• Working knowledge of Databricks jobs notebooks compute Unity Catalog and access management.
• Proficiency with Git code reviews automated testing and deployment practices.
• Strong analytical troubleshooting documentation and stakeholder communication skills.

Key Responsibilities

• Design build and maintain batch and streaming data pipelines using Databricks PySpark Python and SQL.
• Ingest and integrate data from databases files APIs and cloud storage into a lakehouse architecture.
• Develop bronze silver and gold data layers using Delta Lake and implement incremental processing deduplication and change data capture where required.
• Optimize Spark workloads SQL queries compute configurations and storage layouts for performance and cost efficiency.
• Schedule and orchestrate data workflows; implement monitoring alerting error handling and recovery mechanisms.
• Implement data validation reconciliation automated testing and data quality controls.
• Apply data governance and access controls using Unity Catalog and organizational security standards.
• Collaborate with architects analysts and business stakeholders to translate requirements into technical solutions.
• Maintain version-controlled code support CI/CD processes and document pipelines data models and operational procedures.
• Troubleshoot production issues and contribute to root-cause analysis and preventive improvements.

Required Skills and Experience

• Hands-on experience developing data engineering solutions on Databricks.
• Strong proficiency in Python PySpark and SQL including complex transformations and query optimization.
• Working knowledge of Apache Spark execution partitioning joins caching and performance troubleshooting.
• Experience with Delta Lake including merge operations schema management and transactional data processing.
• Experience building ETL/ELT pipelines implementing incremental loads and applying data warehousing and dimensional modeling concepts.
• Experience with at least one cloud platformAzure AWS or Google Cloudand its storage and identity services.
• Working knowledge of Databricks jobs notebooks compute Unity Catalog and access management.
• Proficiency with Git code reviews automated testing and deployment practices.
• Strong analytical troubleshooting documentation and stakeholder communication skills.

Key Responsibilities

• Design build and maintain batch and streaming data pipelines using Databricks PySpark Python and SQL.
• Ingest and integrate data from databases files APIs and cloud storage into a lakehouse architecture.
• Develop bronze silver and gold data layers using Delta Lake and implement incremental processing deduplication and change data capture where required.
• Optimize Spark workloads SQL queries compute configurations and storage layouts for performance and cost efficiency.
• Schedule and orchestrate data workflows; implement monitoring alerting error handling and recovery mechanisms.
• Implement data validation reconciliation automated testing and data quality controls.
• Apply data governance and access controls using Unity Catalog and organizational security standards.
• Collaborate with architects analysts and business stakeholders to translate requirements into technical solutions.
• Maintain version-controlled code support CI/CD processes and document pipelines data models and operational procedures.
• Troubleshoot production issues and contribute to root-cause analysis and preventive improvements.

Required Skills

Databricks & Lakehouse

Databricks Lakehouse Platform

• Delta Lake
• Unity Catalog
• Databricks Workflows
• Auto Loader
• Structured Streaming
• Databricks SQL

Data Engineering

PySpark

• Advanced SQL
• Data Modeling
• ETL/ELT Frameworks
• CDC Architectures
• Real-Time Data Processing

Governance & Quality

Row-Level Security

• Data Masking
• Data Lineage
• Data Quality Frameworks

Cloud & DevOps

AWS/ GCP / Azure

• Git CI/CD
• Terraform (nice to have)

• Extensive experience (5-8 years) in data engineering with a strong emphasis on DataBricks platform technologies.
• Proficient knowledge of software development lifecycle processes with hands-on experience in coding testing and deploying scalable data applications.
• Strong analytical and problem-solving skills focused on root cause investigations and resolution in complex distributed systems.
• Exceptional communication skills enabling effective collaboration across technical and non-technical teams.
• Demonstrable ability to work in a client-centric environment driving passionate results-oriented execution.
• Value-driven approach fostering learning agility problem-solving decision-making and cross-team collaboration.

Wipro Technologies is at the forefront of digital innovation spearheading transformative solutions across industries. As part of our Applied AI & Data group you will contribute to cutting-edge projects that shape the future of technology working alongside passionate professionals dedicated to excellence and reinvention.

We offer an inclusive dynamic workplace that empowers individuals to drive their career progression and cultivate skills within a supportive learning environment. Here your expertise in DataBricks and data engineering will be leveraged to its full potential driving meaningful impact for clients and communities.

Join us to be part of a forward-thinking culture where purpose fuels innovation and every team members contribution is valued in building the modern Wipro experience.