Title: Databricks Engineer
Location: Boston MA (On-site)
Duration: 6 months (possibility of extension)
Implementation Partner: Infosys
End Client: To be disclosed
JD:
Minimum Years of Experience: 8-10 Years
Mandatory Skills: Databricks Hadoop Python Spark Spark SQL PySpark and Airflow. IBM StreamSets (good to have knowledge)
Key Responsibilities:
- Develop Data Engineering and ML pipelines using Databricks and various AWS services including S3 EC2 API Gateway RDS Kinesis/Kafka and Lambda to build serverless applications
- Solid understanding of Databricks fundamentals/architecture with hands-on experience setting up clusters and working across modules (Data Engineering ML and SQL Warehouse)
- Familiarity with Medallion architecture DLT and Unity Catalog within Databricks
- Experience migrating data from on-prem Hadoop to Databricks/AWS
- Strong understanding of core AWS services and architecture best practices
- Hands-on experience across domains like database architecture BI ML advanced analytics and big data
- Strong knowledge of Airflow and CI/CD pipelines in AWS
- Experience migrating RDBMS Java/Python applications models and Elasticsearch
- Strong programming background in Scala Python and SQL