Senior Python Engineer, DataHub Ingestion Framework
Senior Python Engineer, DataHub Ingestion Framework
Employer Active
Job Alert
You will be updated with latest job alerts via email
Share
Job Alert
You will be updated with latest job alerts via email
$ 225000 - 300000
1 Vacancy
Job Description
DataHub built by Acryl Data is an AI & Data Context Platform adopted by over 3000 enterprises including Apple CVS Health Netflix and Visa. Innovated jointly with a thriving opensource community of 13000 members DataHubs metadata graph provides indepth context of AI and data assets with bestinclass scalability and extensibility.
The companys enterprise SaaS offering DataHub Cloud delivers a fully managed solution with AIpowered discovery observability and governance capabilities. Organizations rely on DataHub solutions to accelerate timetovalue from their data investments ensure AI system reliability and implement unified governance enabling AI & data to work together and bring order to data chaos.
The Challenge
As AI and data products become businesscritical enterprises face a metadata crisis:
- No unified way to track the complex data supply chain feeding AI systems
- Engineering teams struggling with data discovery lineage and governance
- Organizations needing machinescale metadata management not just humanbrowsable catalogs
Why This Matters
This is where infrastructure meets impact. The metadata layer youll build will directly power the next generation of AI systems at massive scale. Your code will determine how safely and effectively thousands of organizations deploy AI affecting millions of users worldwide.
The Role
Were looking for an exceptional Python engineer to lead development of DataHubs ingestion framework the core that connects diverse data systems and powers our metadata collection capabilities.
Youll Build
- Scalable faulttolerant ingestion systems for enterprisescale metadata
- Clean intuitive APIs for our connector ecosystem
- Eventdriven architectures for realtime metadata processing
- Schema mapping between diverse systems and DataHubs unified model
- Versioning systems for AI assets (training data model weights embeddings)
You Have
- 4 years building productiongrade distributed systems
- Advanced Python expertise with a focus on API design
- Experience with highscale data processing or integration frameworks
- Strong systems knowledge and distributed architecture experience
- A track record of solving complex technical challenges
Bonus Points
- Experience with DataHub or similar metadata/ETL frameworks (Airflow Airbyte dbt)
- Opensource contributions
- Earlystage startup experience
Location and Compensation
Bay Area (hybrid 3 days in Palo Alto office)
Salary Range: $225000 to $300000
Benefits
- Competitive salary
- Equity
- Medical dental and vision insurance (99% coverage for employees 65% coverage for dependents; USAbased employees)
- Carrot Fertility Program (USAbased employees)
- Remote friendly
- Work from home and monthly coworking space budget
Required Experience:
Senior IC
Employment Type
Full Time
