Data Engineer III (Databricks)
Data Engineer III (Databricks)
Employer Active
Job Alert
You will be updated with latest job alerts via email
Share
Job Alert
You will be updated with latest job alerts via email
Job Description
LivePerson (NASDAQ: LPSN) is the global leader in enterprise conversations. Hundreds of the worlds leading brands including HSBC Chipotle and Virgin Media use our awardwinning Conversational Cloud platform to connect with millions of consumers. We power nearly a billion conversational interactions every month providing a uniquely rich data set and safety tools to unlock the power of Conversational AI for better customer experiences.
At LivePerson we foster an inclusive workplace culture that encourages meaningful connection collaboration and innovation. Everyone is invited to ask questions actively seek new ways to achieve success and reach their full potential. We are continually looking for ways to improve our products and make things better. This means spotting opportunities solving ambiguities and seeking effective solutions to the problems our customers care about.
Overview:
We are looking for an experienced Data Engineer to provide data engineering expertise and support to various analytical products of LivePerson and assist in migrating our existing data processing ecosystem from Hadoop (Spark MapReduce Java and Scala) to Databricks on GCP. The goal is to leverage Databricks scalability performance and ease of use to enhance our current workflows.
You will:
- Assessment and Planning:
- Review the existing Hadoop infrastructure including Spark and MapReduce jobs.
- Analyze Java and Scala codebases for compatibility with Databricks.
- Identify dependencies libraries and configurations that may require modification.
- Propose a migration plan with clear timelines and milestones.
- Code Migration:
- Refactor Spark jobs to run efficiently on Databricks.
- Migrate MapReduce jobs where applicable or rewrite them using Spark DataFrame/Dataset API.
- Update Java and Scala code to comply with Databricks runtime environment.
- Testing and Validation:
- Develop unit and integration tests to ensure parity between the existing and new systems.
- Compare performance metrics before and after migration.
- Implement error handling and logging consistent with best practices in Databricks.
- Optimization and Performance Tuning:
- Finetune Spark configurations for performance improvements on Databricks.
- Optimize data ingestion and transformation processes.
- Deployment and Documentation:
- Deploy migrated jobs to production in Databricks.
- Document changes configurations and processes thoroughly.
- Provide knowledge transfer to internal teams if required.
Required skills:
- 6 years of experience in Data Engineering with focus on building data pipelines data platforms and ETL (Extract transform Load) processes on Hadoop and Databricks.
- Strong Expertise in Databricks (Spark on Databricks Delta Lake etc.) preferably on GCP.
- Strong expertise in the Hadoop ecosystem (Spark MapReduce HDFS) with solid foundations of Spark and its internals.
- Proficiency in Scala and Java.
- Strong SQL knowledge.
- Strong understanding of data engineering and optimization techniques.
- Solid understanding on Data governance Data modeling and enterprise scale data lakehouse platform
- Experience with test frameworks like Great Expectations
Minimum Qualifications :
- Bachelors degree in Computer Science or a related field
- Certified Databricks Engineer Preferred
You should be an expert in:
- Databricks with spark and its internals (3 years) MUST
- Data engineering in Hadoop ecosystem (5 years) MUST
- Scala and Java (5 years) MUST
- SQL MUST
Benefits:
- Health: Medical Dental and Vision
- Time away: vacation and holidays
- Development: Access to internal professional development resources.
- Equal opportunity employer
Why youll love working here:
As leaders in enterprise customer conversations we celebrate diversity empowering our team to forge impactful conversations globally. LivePerson is a place where uniqueness is embraced growth is constant and everyone is empowered to create their own success. And were very proud to have earned recognition from Fast Company Newsweek and BuiltIn for being a top innovative beloved and remotefriendly workplace.
Belonging at LivePerson:
We are proud to be an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to age ancestry color family or medical care leave gender identity or expression genetic information marital status medical condition national origin physical or mental disability protected veteran status race religion sex (including pregnancy) sexual orientation or any other characteristic protected by applicable laws regulations and ordinances. We also consider qualified applicants with criminal histories consistent with applicable federal state and local law.
We are committed to the accessibility needs of applicants and employees. We provide reasonable accommodations to job applicants with physical or mental disabilities. Applicants with a disability who require reasonable accommodation for any part of the application or hiring process should inform their recruiting contact upon initial connection.
***The talent acquisition team at LivePerson has recently been notified of a phishing scam targeting candidates applying for our open roles. Scammers have been posing as hiring managers and recruiters in an effort to access candidates personal and financial information. This phishing scam is not isolated to only LivePerson and has been documented in news articles and media outlets.
Please note that any communication from our hiring teams at LivePerson regarding a job opportunity will only be made by a LivePerson employee with an @ email address.
LivePerson does not ask for personal or financial information as part of our interview process including but not limited to your social security number online account passwords credit card numbers passport information and other related banking information.
If you have any questions and or concerns please feel free to contact
Employment Type
Full Time
