Databricks Architect

Position- Databricks Architect

Location- Portland OR

Duration- 6Months-Onsite 5 days

Experience- 13

Roles & Responsibilities

JD:

• We are seeking highly skilled and experienced Hands-on Databricks Architect to design build and optimize our modern data streaming platform. This platform built on Azure and leveraging the Databricks Lakehouse is central to our Data Availability Initiative. It provides comprehensive data ingestion real-time streaming batch processing analytics and AI/ML capabilities. The ideal candidate will be a technical lead with deep expertise in Databricks Azure cloud services and a strong understanding of data architecture principles like the Medallion Architecture and Data Mesh. This role requires a hands-on approach to implementing solutions driving robust data governance through Unity Catalog and integrating advanced AI/ML models.

• Lead the architectural design development and implementation of scalable and robust data solutions on the Databricks Lakehouse Platform within an Azure environment.
• Design and implement a robust multi-workspace strategy by manually configuring and automating the deployment of Development QA and Production Databricks environments.
• Standardize environment setup using Terraform ensuring consistent repeatable and version-controlled infrastructure across the entire SDLC.
• Architect and implement data ingestion strategies for real-time streaming (using Debezium Apache Kafka Confluent Platform) and batch processing leveraging Databricks Lake Flow and Auto Loader.
• Design and develop data pipelines following the Medallion Architecture (Lead Bronze Silver Gold tiers) using Databricks Delta Live Tables (DLT) and Structured Streaming ensuring data quality lineage and governance.
• Optimize Databricks clusters Delta Lake tables and Spark jobs for performance scalability and cost efficiency including leveraging auto-scaling spot instances compression and partitioning techniques.
• Establish and enforce comprehensive data governance security and access control policies utilizing Databricks Unity Catalog and Azure Purview across multi-tenant workspaces and business units.
• Develop and integrate MLOps pipelines using Databricks MLflow and Azure Machine Learning for model training serving and monitoring including custom Large Language Model (LLM) integrations.
• Collaborate with business stakeholders to translate complex data requirements into technical designs and scalable solutions.
• Implement CI/CD practices for Databricks solutions using Databricks Asset Bundles Terraform GitHub and GitHub Actions ensuring automated deployment and version control.
• Integrate data from diverse sources including SQL databases (PostgreSQL SQL Server) MongoDB Snowflake and various flat/multimedia files into the data platform.
• Design and implement API-first data consumption patterns serverless functions (Azure Functions) and application integrations for various data products and real time services.
• Provide technical leadership mentorship and best practices to data engineering and development teams.
• Contribute to the continuous evolution of the data platform incorporating future considerations such as the Databricks Ingestion Gateway advanced AI integration and Data Mesh evolution.

• Databricks Lakehouse Apache Spark Delta Live Tables (DLT) Auto Loader Microsoft Azure ADLS Gen2 Azure Functions Azure Machine Learning Apache Kafka Confluent
• Platform Debezium (CDC) Databricks Unity Catalog Azure Purview Python SQL Scala (Advantage) Databricks Asset Bundles Terraform GitHub Actions Medallion Architecture Data Mesh PostgreSQL MongoDB Snowflake MLOps LLM Integration Azure Networking

• Bachelors or Masters degree in Computer Science Data Engineering or a related quantitative field.
• 8 years of progressive experience in data architecture data engineering or a related role within an enterprise environment.
• 5 years of hands-on experience specifically with the Databricks Lakehouse Platform including designing and implementing complex data solutions.
• Proven track record of successfully designing and implementing large-scale high performance and cost-optimized data platforms on Microsoft Azure. Databricks Certified Data Engineer (Associate or Professional) and/or Azure Data Engineer Associate (DP-203) certifications are highly desirable.
• Excellent communication presentation and interpersonal skills with the ability to influence technical and non-technical stakeholders effectively.

Essentials skills - Databricks Apache Spark Auto Loader Azure ADLS Gen2 Apache Kafka LLM PostgreSQL MongoDB Snowflake MLOps.

Vishal Gautam Technical Recruiter

Momento USA Exceeding Customer Expectations

440 Beningo Blvd Unit#A-5 2^nd Floor Bellmawr NJ 08031

Interstate Business Park.

(About US )

Exceeding customer expectations.

Desk - / Extn 1011

Email: vishal@ Linkedin - (

Minority Certified by SWAM
National Minority Certified by NMSDC
One of the fastest growing company in NJ
Awarded fastest growing Asian American business by
E-verified Company

Information transmitted by this e-mail is proprietary to Momento USA and/ or its Customers and is intended for use only by the individual or entity to which it is addressed and may contain information that is privileged confidential or exempt from disclosure under applicable law. If you are not the intended recipient or it appears that this mail has been forwarded to you without proper authority you are notified that any use or dissemination of this information in any manner is strictly such cases please notify us immediately at and delete this mail from your records.

Note: Momento USA is an Equal Opportunity/Affirmative Action Employer. All qualified applicants will receive consideration for employment without regard to race color religion sex pregnancy sexual orientation gender identity national origin age protected veteran status or disability status.