Associate Backend Engineer - PivotConsult

Agency :

Job Description Summary :

Job Description :

Job description

• Developing ETL Pipelines: Designing developing and maintaining scalable and adaptable data pipelines using Python or PySpark to facilitate the smooth migration of data from diverse data sources . Host these ETL pipelines in AWS EC2 AWS Glue or AWS EMR and store this data to cloud database services like Google BigQuery AWS S3 Redshift RDS Delta Lake etc. This includes managing significant data migrations and ensuring seamless transitions between systems.
• Implementing Data Quality Check Framework: Establishing and executing data quality checks and validation pipelines using different tools like Python PySpark Athena or BigQuery S3 Delta Lake to uphold the integrity and accuracy of our datasets.
• Creating Mechanisms for Generating ETL Migration Status Reports: Devising a framework to generate concise summary reports detailing data migration progress alongside promptly alerting stakeholders to any failures within ETL pipelines. This ensures swift resolution of data discrepancies arising from pipeline failures. Implement this using standard SMTP Python AWS SNS AWS SES AWS S3 Delta Lake etc services.
• Data Transformations and Processing: Implementing various data encryption and decryption techniques using Python and PySpark libraries in addition to generating insightful reports and analyses derived from processed data to aid in informed business decisionmaking.
• Development of APIs: Building APIs using frameworks such as Flask or Django incorporating diverse authentication and authorization techniques to safeguard the exchange of data. Host these APIs on EC2 server using services like Gearman etc or Write API logics in lambda and host these APIs using API Gateway services of cloud.
• Code Versioning and Deployment: Leveraging GitHub extensively for robust code versioning deployment of the latest code iterations seamless transitioning between different code versions and merging various branches to streamline development and code release processes.
• Automation: Designing and implementing code automation solutions to streamline and automate manual tasks effectively.

Required Candidate profile

Soft Skills

Must Have:

• Demonstrates adept problemsolving skills to efficiently address complex challenges encountered during data engineering tasks.
• Exhibits clear and effective communication skills facilitating seamless collaboration and comprehension across diverse teams and stakeholders.
• Displays proficiency in both independent and collaborative work dynamics fostering productivity and synergy within a fastpaced team environment.
• Demonstrates a high level of adaptability to changing requirements customer dynamics and work demands.
• Selfmotivated and responsible individual who takes ownership and initiative in tasks.
  

Good to Have:

• Demonstrates project management experience offering valuable insights and contributions towards efficient project and delivery.
• Good Presentation skills
• Excellent customer handling skills.

Technical Skills

• Proficiency in SQL (Structured Query Language) for querying and manipulating databases. Experience with relational database systems like MySQL PostgreSQL or Oracle and NoSQL databases like Mongo.
• Proficiency in objectoriented programming concepts such as encapsulation inheritance and polymorphism.
• Knowledge of data warehousing concepts and experience with data warehousing solutions like Amazon Redshift Google BigQuery or Snowflake.
• Experienced in developing ETL pipelines using Python PySpark.
• Knowledge of Python libraries/frameworks like Pandas NumPy or Spark for data processing and analysis.
• Familiarity with big data processing frameworks like Apache Hadoop and Apache Spark for handling largescale datasets and performing distributed computing.
• Knowledge of cloudbased services like AWS S3 AWS Glue AWS EMR AWS Lambda Athena Azure Data Lake Google BigQuery etc.
• Familiarity with version control systems like Git for managing codebase changes collaborating with team members and maintaining code quality.
• Experience with web sing libraries and frameworks like BeautifulSoup Sy Puppeteer Selenium etc. is highly beneficial. Knowledge of regular expressions is useful for pattern matching and extracting specific data formats from text. Understanding of HTTP protocols and how web servers respond to requests how to send requests to web servers handle responses and manage sessions and cookies is essential. Familiarity with XPath expressions or CSS selectors is important for targeting specific elements within the HTML structure.

Required Experience

• The ideal candidate will have a minimum of 13 years of relevant experience in data engineering roles with a demonstrated history of successfully developing and maintaining ETL pipelines handling big data migrations and ensuring data quality and validation.
• Must have excellent knowledge and programing capability using Python PySpark working on any of the Cloud Platforms like AWS Azure or Google.

Role

• Industry Type: Engineering
• Functional Area: Data Engineering Software Development Automation
• Employment Type: Full Time Permanent
• Role Category: System Design/Implementation
• Education : A minimum educational requirement is graduation.

Here at Havas across the group we pride ourselves on being committed to offering equal opportunities to all potential employees and have zero tolerance for discrimination. We are an equal opportunity employer and welcome applicants irrespective of age sex race ethnicity disability and other factors that have no bearing on an individuals ability to perform their job.

Contract Type :

Here at Havas across the group we pride ourselves on being committed to offering equal opportunities to all potential employees and have zero tolerance for discrimination. We are an equal opportunity employer and welcome applicants irrespective of age sex race ethnicity disability and other factors that have no bearing on an individuals ability to perform their job.