Data Engineer (1 2 years of experience in Big Data, Spark, Hadoop, Shell scripting)
Share
Job Location:
Monthly Salary:
Posted on:
8 hours ago
Vacancies:
1 Vacancy
Job Summary
At Visa the Corporate Information Technology Billing & Incentives Platforms team enables Visas revenue growth through flexible pricing engines and global revenue platforms built on next-generation technologies. This includes managing system requirements evaluating cutting-edge technologies design development integration quality assurance implementation and maintenance of corporate revenue applications. The team works closely with business owners of these services to deliver customer developed solutions as well as implement industry leading packaged software. This team has embarked on a major transformational journey to build and implement best of breed revenue and billing applications to transform our business as well as technology.
The candidate should enjoy working on diverse technologies and should be excited to take initiatives to solve complex business problems and get the job done while taking on new challenges. You should thrive in team-oriented and fast-paced environments where each team-member is vital to the overall success of the projects.
Key Responsibilities
Design develop and maintain automated test frameworks and scripts using PySpark for large-scale Big Data applications.
Collaborate closely with data engineering development and DevOps teams to understand data pipelines workflows and transformation logic.
Build and execute robust tests to validate data ingestion processing and transformation in Hadoop and Spark ecosystems.
Perform comprehensive data validation data integrity and performance testing across Spark Hive and RDBMS data layers.
Apply deep understanding of Spark concepts (RDDs DataFrames Datasets Spark SQL) to create reusable test automation components.
Leverage Shell scripting and Unix-based tools for automation log analysis and environment configuration.
Integrate test automation into CI/CD pipelines to ensure continuous testing and high-quality data delivery across environments.
Identify document and track defects working proactively with developers to ensure quick resolution. Knowledge of Jira Standards.
Optimize and tune Spark jobs and query performance for scalability and efficiency.
Maintain detailed documentation including test plans test cases and results aligned with evolving business requirements.
Conduct manual and functional testing for critical payment and financial data applications to ensure compliance and accuracy.
Operate independently in a fast-paced distributed environment balancing multiple priorities to meet service-level goals.
Collaborate effectively across teams and time zones fostering transparent communication trust and long-term partnerships with business stakeholders.
This is a hybrid position. Expectation of days in office will be confirmed by your hiring manager.
Qualifications :
Basic Qualifications:
Bachelors degree OR 3 years of relevant work experience
Preferred Qualifications:
- Bachelors degree in computer science Information Technology or related field.
- Relevant certifications in Big Data Spark QA.
- Experience with on premise infrastructure
- Knowledge of ETL processes and tools.
Additional Information :
Visa is an EEO Employer. Qualified applicants will receive consideration for employment without regard to race color religion sex national origin sexual orientation gender identity disability or protected veteran status. Visa will also consider for employment qualified applicants with criminal histories in a manner consistent with EEOC guidelines and applicable local law.
Remote Work :
No
Employment Type :
Full-time
Key Skills
- Apache Hive
- S3
- Hadoop
- Redshift
- Spark
- AWS
- Apache Pig
- NoSQL
- Big Data
- Data Warehouse
- Kafka
- Scala
About Company
Visa (NYSE: V) is a world leader in digital payments, facilitating transactions between consumers, merchants, financial institutions and government entities across more than 200 countries and territories. Our purpose is to uplift everyone, everywhere by being the best way to pay and b ... View more
