Data Engineer

Job Title: Data Engineer

Designation: Data Engineering

Location: Mph-ClientSite North Carolina US

Job Summary: We are seeking a skilled Data Engineer with expertise in PySpark to join our dynamic team. The ideal candidate will be responsible for designing developing and maintaining robust data pipelines and architectures that support our data-driven decision-making processes. This role requires a strong understanding of data engineering principles and a passion for working with large datasets.

Responsibilities:

• Design develop and implement scalable data pipelines using PySpark to process and analyze large volumes of data.
• Collaborate with data scientists and analysts to understand data requirements and deliver high-quality data solutions.
• Optimize data processing workflows for performance and efficiency.
• Ensure data quality and integrity through rigorous testing and validation processes.
• Monitor and troubleshoot data pipeline performance implementing improvements as necessary.
• Document data engineering processes architectures and workflows for future reference.
• Stay updated with the latest industry trends and technologies in data engineering and analytics.

Mandatory Skills:

• Proficiency in PySpark for data processing and transformation.
• Strong understanding of data modeling ETL processes and data warehousing concepts.
• Experience with big data technologies such as Hadoop Spark or similar frameworks.
• Familiarity with SQL and NoSQL databases.
• Ability to work with cloud platforms (e.g. AWS Azure Google Cloud) for data storage and processing.
• Strong analytical and problem-solving skills.

Preferred Skills:

• Experience with data visualization tools (e.g. Tableau Power BI).
• Knowledge of machine learning concepts and frameworks.
• Familiarity with programming languages such as Python or Java.
• Experience with containerization technologies (e.g. Docker Kubernetes).
• Understanding of data governance and security best practices.

Qualifications:

• Bachelors degree in Computer Science Information Technology or a related field.
• Relevant certifications in data engineering or big data technologies are a plus.
• Strong communication skills and the ability to work collaboratively in a team environment.