Databrick Architect

Position : Databrick Architect

Location : Remote

Term : C2C/W2 role

Duration : 12 months

Job Description:

We are seeking an experienced and highly skilled Databricks Architect to design develop and optimize scalable data platforms and cloud-based analytics solutions. The ideal candidate will have strong expertise in Databricks Spark cloud technologies data engineering and modern data architecture practices. This role involves collaborating with business stakeholders data engineers analysts and architects to deliver robust big data and AI/ML solutions.

Key Responsibilities

• Design and implement scalable data architectures using Databricks and Apache Spark.
• Develop and optimize ETL/ELT pipelines for structured and unstructured data.
• Architect cloud-based data solutions on AWS Azure or GCP.
• Build and manage Lakehouse architecture using Databricks.
• Integrate multiple data sources including APIs databases streaming and cloud storage.
• Optimize Spark jobs and Databricks workloads for performance and cost efficiency.
• Implement data governance security data quality and compliance standards.
• Collaborate with data scientists and analytics teams for AI/ML workloads.
• Design batch and real-time data processing solutions.
• Lead technical discussions architecture reviews and best practice implementations.
• Mentor junior engineers and provide technical leadership.
• Create architecture documentation technical diagrams and solution designs.
• Work closely with DevOps teams for CI/CD and deployment automation.
• Troubleshoot production issues and provide long-term scalable solutions.

Required Skills

Must Have Skills

• Strong hands-on experience with Databricks
• Expertise in Apache Spark (PySpark / Scala Spark)
• Experience with Delta Lake and Lakehouse architecture
• Strong SQL and data modeling skills
• Experience with cloud platforms:
  • AWS / Azure / GCP
• Knowledge of data warehousing concepts
• Experience building ETL/ELT pipelines
• Hands-on experience with:
  • Python
  • Scala
  • SQL
• Experience with data orchestration tools:
  • Airflow
  • Azure Data Factory
  • AWS Glue
• Knowledge of performance tuning and optimization
• Understanding of CI/CD and DevOps practices
• Experience with Git/version control systems

Preferred Skills

• Experience with streaming technologies:
  • Kafka
  • Spark Streaming
• Knowledge of AI/ML integration on Databricks
• Experience with Snowflake or Redshift
• Infrastructure as Code experience:
  • Terraform
  • CloudFormation
• Familiarity with containerization:
  • Docker
  • Kubernetes
• Experience with Unity Catalog and data governance
• Certification in Databricks or Cloud platforms

Qualifications

• Bachelors or Masters degree in Computer Science Information Technology or related field.
• 10 years of overall IT experience.
• 5 years of experience in Big Data/Data Engineering.
• 3 years of hands-on Databricks architecture experience.

Nice to Have

• Databricks Certified Professional certification
• Experience in Healthcare Banking Retail or Telecom domains
• Exposure to Generative AI/Data AI solutions
• Experience with Medallion Architecture

Cloud BC Labs Inc is a digital transformation organization aimed at creating seamless solutions for clients to effectively manage their business operations. The company specializes in Business and Management Consulting AI/ML Data Analytics & Visualization Cloud Data Warehouse Migration Snowflake Implementation Informatica Implementation & Upgrade Staffing Services and Data Management Solutions