Azure Data Engineer

Role Name: Data Engineer for Rialto

Experience: 6 to 8 Years

Primary Location: Mumbai
Other Acceptable Locations: Kolkata Pune Hyderabad Bangalore Chennai Noida

Key Skills: Python PySpark Microsoft Fabric AI & Gen AI Products & Tools Azure Data Factory

Must-Have Technical/Functional Skills
Proficiency in Python TensorFlow PyTorch and other popular libraries for AI development.

Experience Required

Conversational AI and LLM Experience Hands-on experience with conversational AI systems including Retrieval-Augmented Generation (RAG) models fine-tuning large language models (LLMs) at an enterprise scale and developing dialogue management systems to handle dynamic user interactions effectively.

NLP and Language Model Specialization Proficiency with NLP techniques and a deep understanding of language models like GPT BERT T5 and other transformer-based models.

Data Engineering and Analytics Substantial experience in data engineering and analytics with strong expertise in Databricks and PySpark. Hands-on knowledge of Apache Spark fundamentals including Spark Architecture its APIs and how to leverage it for data processing and analysis.

Programming and Query Skills Proficiency in PySpark Python and SQL is essential for data manipulation analysis and transformation.

Cloud Platforms Demonstrated experience with cloud platforms (AWS Azure Google Cloud) and their integration with Databricks.

Data Solutions Architecture Experience as a data solutions architect is a significant advantage demonstrating the ability to design comprehensive data solutions.

Soft Skills Excellent communication and team collaboration skills.

Roles & Responsibilities

Machine Learning and Deep Learning Expertise Strong foundation in ML and DL concepts including model architectures such as transformers RNNs and LSTMs along with training techniques like supervised unsupervised and reinforcement learning.

Strategic Data Solution Development Serve as a primary contributor to the development of data solutions with a strategic outlook focused on building a data platform that serves as the single source of truth data lakehouse.

Design and Development Architect and manage data solutions ensuring integration of Databricks with existing data systems. Design ELT processes and data models in line with business requirements.

Data Management and Optimization Optimize data storage and processing in Databricks. Implement data lakehouse solutions ensuring performance and scalability.

Data Engineering Expertise Lead the development of data tables tailored to specific use cases by engineering critical elements from multiple data domains. Ensure ingestion of third-party data is well-structured compliant with data quality standards and traceable from the consumption layer back to the raw data layer.

Strategic Integration and Security Partner with the Information Security and Infrastructure teams to streamline data integration maintain data security and follow access best practices. Contribute to the creation of end-to-end data analytics solutions.