Fullstack Data Engineer (Telecom domain)
Share
Job Location:
Middletown, DE - USA
Monthly Salary:
Not Disclosed
Posted on:
7 hours ago
Vacancies:
1 Vacancy
Job Summary
Job Summary:
- Location & Work Mode: Hybrid role; applicants must be local to Texas (TX) or New Jersey (NJ).
- Industry Experience: Must have recent experience in the telecom domain.
- Location & Work Mode: Hybrid role; applicants must be local to Texas (TX) or New Jersey (NJ).
- Industry Experience: Must have recent experience in the telecom domain.
- Role Overview:
- Responsible for designing and implementing a real-time data platform using Medallion Architecture.
- Manage end-to-end development of data pipelines from ingestion to consumption-ready output.
- Act as a Full Stack data professional handling infrastructure automation (DataOps) and complex data modeling.
Key Responsibilities:
- Build scalable real-time ingestion pipelines using Spark Structured Streaming and Auto Loader for sources like Kafka/Event Hubs/CDC.
- Develop ELT logic to cleanse deduplicate and normalize data into relational formats with ACID compliance.
- Design optimized JSON data structures in Delta tables for low-latency API consumption.
- Implement and manage workflows using Delta Live Tables (DLT) streaming tables and Databricks Workflows.
- Enforce data governance and security using Unity Catalog for access control and maintain a searchable data catalog.
- Own and automate deployment lifecycle with Databricks Asset Bundles (DABs) and CI/CD tools (GitHub Actions/Azure DevOps).
- Optimize streaming processes for performance and cost efficiency.
Required Skills & Qualifications:
- Technical Core:
- Expert in PySpark/Scala Spark internals and Delta Lake features (Z-Ordering Liquid Clustering CDF Time Travel).
- Hands-on experience with advanced streaming patterns (watermarking checkpoints handling late data).
- Data Modeling & Languages:
- Advanced SQL skills for complex transformations.
- Mastery in handling and generating complex nested JSON objects in Spark.
- Experience with Medallion Architecture (Bronze Silver Gold layers) and data contracts.
- Full Stack & DevOps:
- Proficient in automating deployments (CI/CD Git-based workflows).
- Capable of setting up observability/monitoring (Databricks SQL Alerts Grafana).
- Soft Skills:
- Strong architectural thinking to balance cost and latency in streaming solutions.
- Client-focused mindset understanding API consumption needs for JSON Gold layer.
Other Requirements:
- Must be able to work in a hybrid setup and be located in TX or NJ.
- Recent hands-on experience in the telecom industry is required.
