Databricks Technical Lead

Title : Databricks Technical Lead

Location : New Brunswick NJ on Site

Employment : Fulltime Only

Certification: Databricks certified Data Engineer Professional SQL Azure data Engineer Python

Primary Skillset:

• Apache Spark: Strong understanding and handson experience with Apache Spark including Spark SQL Spark Streaming.
• Databricks: Proficiency in using Azure Databricks including setting up and managing clusters notebooks and jobs.
• Python and SQL: Strong programming skills in either Python and SQL as these are the primary languages used for writing Spark applications.
• Data Processing and Analysis: Experience in designing and implementing data processing and analysis pipelines using Spark and Databricks.
• Distributed Computing: Knowledge of distributed computing concepts and experience with distributed computing frameworks like Spark.
• Cloud Platform: Familiarity with Azure cloud platform and its services including Azure Data Factory Azure access Azure Storage Azure Data Lake and Azure SQL Database.
• Data Engineering: Understanding of data engineering concepts and experience with ETL processes data modeling and data warehousing.
• DevOps: Experience with CI/CD pipelines version control systems (e.g. Git) and automated deployment of Spark applications.
• Performance Optimization: Ability to optimize Spark applications for performance and scalability including tuning Spark configurations and leveraging Spark optimizations.
• Data Security and Governance: Understanding of data security and governance practices including data encryption access controls and compliance regulations.
• Monitoring and Troubleshooting: Proficiency in monitoring and troubleshooting Spark applications identifying and resolving performance bottlenecks or issues.
• Collaboration and Communication: Excellent communication skills and the ability to collaborate effectively with crossfunctional teams including data scientists data engineers and business stakeholders.

Secondary Skillset:

• Machine Learning: Familiarity with machine learning concepts and experience with implementing machine learning algorithms using Spark MLlib or other machine learning frameworks.
• Data Visualization: Knowledge of data visualization tools and libraries like Matplotlib Plotly Power BI or Tableau for presenting insights derived from data analysis.