Sr Data Engineer

Job Title Data Engineer

Location Edison NJ (Onsite)

Duration : Fulltime

Job Description :

• Design develop monitor and operate data pipelines and maintenance of existing data pipelines.
• Collaborating with product owner and analysts to provide datadriven solutions and insights.
• Implementing data transformations and aggregations for large datasets.
• Ensuring data security and compliance with data governance policies.
• Developing in languages like Pyspark and Python.
• Preprocess of structured and unstructured data.
• Analyze transform and create SQL queries.
• Developing and optimizing Sparkbased data processing applications on the AWS Databricks platform.
• Troubleshooting and debugging datarelated issues on the AWS Databricks platform.
• Integrating Databricks with other data storage and processing systems.

What we are looking for:

• Expertise in Apache Spark along with Spark streaming & Spark SQL.
• Highly proficient with Apache Spark Scala/ Python and PySpark.
• Good understanding & handson experience Gitlab with CI/CD.
• Handson experience in AWS Services like S3 Glue is required.
• Working knowledge of AWS Databricks Cluster Management.
• Design build and test data pipelines and workflows using AWS Databricks.
• Having the knowledge about Optimizing AWS Databricks Streaming jobs.
• Have a deep understanding of AWS Cloud platforms and data engineering technologies.
• Strong SQL skills to perform data analysis and understanding of source data.
• Familiarity with NoSQL databases and JSON formats.
• Experience in building ETL / data warehouse transformation processes.
• Strong knowledge of Data Management principles.
• Good to have experience in analytical tool.
• Good to have experience in Alteryx/Tableau.