Senior Data Engineer

We are looking for a Senior Data Engineer with minimum 5 years of professional experience to design build and maintain scalable high-performance data platforms that power our analytics and machine learning initiatives. You will work in a distributed data-intensive environment and collaborate with cross-functional teams to ensure data accuracy reliability and security.

Roles & Responsibilities:

• Design develop and maintain scalable data processing pipelines and workflows using frameworks such as Apache Spark PySpark and Apache Beam.
• Build and maintain microservices in Python that serve data-driven features in production.
• Develop internal tools to support CI/CD pipelines experiment tracking and data versioning.
• Collect process and integrate large datasets from multiple sources including databases file systems and APIs.
• Ensure data integrity consistency and quality through robust validation and monitoring processes.
• Optimize data systems for performance scalability and high availability.
• Implement best practices for data security access control and privacy.
• Collaborate with data scientists analysts and engineers to support analytics and ML workflows.
• Lead complex migration initiatives involving the transition from on-premise Cloudera (HDFS Hive Impala HBase) environments to cloud-native platforms (Snowflake or Databricks) ensuring zero data loss and minimal downtime.
• Architect and build a large-scale greenfield data platform from the ground up including the setup of end-to-end infrastructure data ingestion layers transformation frameworks and high-throughput integrations with Snowflake or Databricks.

Must have:

• Strong understanding of distributed systems and modern data architectures.
• Proven hands-on experience with large-scale data platform migrations specifically transitioning from Cloudera (CDH/HDP) to either Snowflake or Databricks. This includes re-architecting Hive/Impala workloads migrating HDFS data and refactoring legacy ETL processes to leverage cloud-native features (e.g. Snowpipe Auto-ingestion or Delta Live Tables).
• Deep technical expertise in building and orchestrating a high-performance data platform from scratch including the implementation of complex integrations (Reverse ETL CDC API ingestions) specifically targeting Snowflake (using Snowpipe Streaming Tasks Streams) or Databricks (using Auto Loader Delta Sharing Unity Catalog).
• 5 years of professional experience in software engineering or data engineering.
• Strong software engineering skills with Python in large-scale high-performance production environments.
• Hands-on experience with Spark/PySpark and other big data frameworks.

About Opplane

Opplane specializes in providing advanced data-focused solutions for financial services telecommunication and reg-tech to accelerate their digital transformation journey. Opplane leadership team is comprised of Silicon Valley serial entrepreneurs and experienced executives. Its expertise comes from years of specific industry experience at some of the worlds top companies such as PayPal Xerox Parc Amazon Wells Fargo SoFi in the areas of product management data technology data governance data privacy security machine learning and risk management.

Global & Multicultural Diverse perspectives global collaboration (US Portugal India and Singapore offices)
Startup Energy Fast-moving impact-driven environment
Ownership Mindset Engineers own what they build
Collaborative & Friendly Open curious and supportive culture