Principal Engineering Lead — Content & Data Pipeline

We believe in bold ideas diverse perspectives and the drive to transform knowledge into impact. Here your curiosity fuels progress your voice shapes innovation and your ambition helps redefine whats possible within science and learning. We are a culture that obsesses over impact challenges and drives whats next to power infinite possibilities for our customers colleagues and society at large.

About the Role:

About the role

This role leads Wileys Content & Data Pipeline function the platform that takes scholarly content (journals articles books) from a wide mix of sources through ingestion processing and enrichment and delivers it to internal AI systems RAG pipelines and external customers under the right entitlements. It is a hybrid role: you set technical direction and stay hands-on with architecture and code where it matters and you also manage and grow a team of engineers who own the day-to-day pipeline work.

The role sits alongside an existing dev lead who runs the current content ingestion and delivery pipeline; you take end-to-end ownership above that expanding the platform to cover new source types new delivery surfaces (especially internal AI / RAG and licensed external delivery) and higher standards of quality freshness and observability.

The mandate spans the full content lifecycle ingestion transformation enrichment metadata management rights-aware delivery and vendor / supplier integration across scholarly publications and books. Stakeholders include product business licensing partners and vendors and part of the job is turning their needs into a coherent prioritized platform roadmap.

We are looking for someone who leads by doing and by coaching who codes reviews unblocks and mentors while also owning delivery and stakeholder alignment for the team.

What youll do

Technical direction and architecture

• Own end-to-end architecture for the content and data pipeline: ingestion from vendors and public sources transformation and enrichment metadata management and delivery to internal AI / RAG systems and external customers.

• Run and operate the pipeline in production and manage the content lifecycle corrections retractions version updates metadata deposits and registry syndication and rights and entitlements enforced at every delivery channels.

• Set standards for pipeline reliability data quality freshness monitoring testing observability and CI/CD across the teams services.

• Codify architectural decisions in ADRs; drive convergence on shared patterns across ingestion processing and delivery.

• Stay hands-on write code run architecture reviews prototype and unblock the team where needed.

Team leadership and delivery

• Manage and coach a team of engineers spanning content ingestion processing and delivery.

• Partner closely with the existing dev lead on the current pipeline; align on ownership boundaries and coordinate roadmap execution.

• Own delivery commitments scope sequencing dependencies and honest tradeoff conversations with stakeholders when reality does not match the plan.

• Run engineering practices that scale: code review discipline on-call rotation incident response and postmortems quality gates in CI.

Stakeholder partnership

• Partner with product and business teams on capability priorities what content the platform ingests next what delivery surfaces get built what quality standards matter.

• Own vendor and licensing-partner relationships for content sources and delivery pipes technical integration data contracts escalation renewal input.

• Translate stakeholder needs (editorial ops product licensing) into a coherent platform roadmap; push back honestly when requests do not fit the platforms direction.

• Communicate technical strategy and tradeoffs upward and outward not just to peer engineering leaders but to product business and executive audiences.

What youll own

• The Content & Data Pipeline platform end-to-end ingestion through delivery including architecture standards delivery SLAs and data quality guarantees to internal and external consumers.

• The Content & Data Pipeline team build and delivery

• Vendor and licensing-partner technical relationships for content sources and delivery pipes in scope.

• The technical roadmap for expanding the platform to new sources new delivery surfaces and higher engineering bars codified in ADRs and communicated to stakeholders.

Must-have skills

Leadership

• Player-coach engineering leadership at scale you have led an engineering team while remaining genuinely hands-on with architecture and code.

• Cross-team influence with product business vendor and executive stakeholders. Able to communicate technical strategy in language each audience actually uses.

• Vendor and partner relationship management technical integration contract execution renewal input escalation.

• Honest tradeoff communication you make scope quality and timeline conversations concrete rather than optimistic.

Technical depth

• Deep production experience designing and operating content or data pipelines end-to-end ingestion processing delivery at scale.

• AWS at production depth S3 Lambda ECS or EKS SQS / SNS EventBridge Step Functions RDS DynamoDB. Real IAM and cost fluency.

• Python and SQL at production depth.

• Workflow orchestration in production Airflow Step Functions Dagster or equivalent with real experience in idempotency backfills retries dependency management.

• Content transformation experience XML processing schema-driven transformation structured content formats.

• Data quality practice: schema validation defensive parsing incremental correctness checks freshness monitoring alerting on drift.

• Distributed systems literacy: idempotency retries duplicate handling partial failure replay.

• Real production experience with API-based and bulk ingestion from external sources auth pagination rate limiting silent schema changes.

• Testing and CI/CD discipline for pipelines unit integration and data-quality tests in CI; safe deployment patterns.

• Observability practice logs metrics lineage and the discipline of knowing whether the pipeline is actually healthy vs. just running.

• Active use of AI coding agents in your day-to-day workflow always with review.

Hugely preferred: content pipeline experience

Scholarly publishing journals articles books is our domain and our first preference. But the core skill is building pipelines for content not just data and that experience carries over from any content-heavy environment.

• You have built or run pipelines that ingest process and deliver content in structured formats XML HTML EPUB PDF where structure and metadata matter.

• You have dealt with content that gets corrected updated or pulled after it ships and built pipelines that handle that cleanly.

• You have delivered the same content to different consumers in different shapes feeds APIs search AI systems.

• You have worked with content that has usage rights or access restrictions attached.

• You have partnered with editorial production or content-operations teams and earned their trust.

Scholarly-specific knowledge JATS DOIs Crossref retraction handling is a strong plus. We can teach the acronyms to someone who already has the pipeline instincts.

Nice to have

• Scholarly publishing specifics JATS / BITS XML DOI infrastructure Crossref deposits ORCID / ROR identity MeSH and similar controlled vocabularies.

• BigQuery / GCP experience useful for the secondary consumer surface especially as our AI teams standardise there.

• Specialised publishing tooling XSLT at scale existing content-management systems or custom XML processing frameworks.

• Streaming ingestion Kafka Kinesis Pub/Sub for near-real-time content updates.

• Vector search and embedding pipelines useful for the internal AI / RAG delivery surface.

• Prior partnership with ML AI or agent teams as a downstream consumer of your platform.

• Data catalog and lineage tooling: DataHub Amundsen or equivalent.

• Existing relationships in the scholarly publishing ecosystem vendors standards bodies or peer engineering leaders at similar organisations.

How we work

• Player-coach leadership. We lead by doing and by coaching. Purely delegating leaders and purely heads-down principal engineers are both mismatches for this role.

• Content as a first-class engineering concern. Scholarly content has production standards versioning semantics and rights implications that generic data pipelines do not and we engineer to them.

• End-to-end ownership. We own ingestion through delivery. We do not throw work over the wall between ingestion processing and delivery teams; we design across them.

• Stakeholder partnership. Product business vendors and executives are partners not audiences. We share evidence and tradeoffs openly and we push back honestly when needed.

• Learning agility. The scholarly content and AI landscapes both move quickly. We hire leaders who show up curious about new territory rather than defensive of what they already know.

• Quality is a first-class engineering concern. Data-quality bugs are engineering bugs not the upstream sources problem.

• Agentic engineering is a strong expectation including at leadership levels. We use AI coding agents actively always with review and we set the standards our teams follow.

• Production ownership. Engineers including engineering leaders own the platforms they build including on-call incident response and using observability tools to diagnose their own systems.

We power infinite possibilities.

For more than 200 years weve transformed knowledge into discoveries that shape the world. Today our global team of innovators creators and experts is driving whats next in science education and publishingcreating impact that reaches everywhere.

Were not just observers of progress. Were the ones accelerating scientific breakthroughs advancing learning and sparking innovation that redefines entire fields and improves lives.

Here your talent matters. Your ideas have room to grow. And your work creates breakthroughs that can change everything.

Wiley is an equal opportunity/affirmative action employer. We evaluate all qualified applicants and treat all qualified applicants and employees without regard to race color religion sex sexual orientation gender identity or expression national origin disability protected veteran status genetic information or based on any individuals status in any group or class protected by applicable federal state or local laws. Wiley is also committed to providing reasonable accommodation to applicants and employees with disabilities. Applicants who require accommodation to participate in the job application process may contact for assistance.

We are proud that our workplace promotes continual learning and internal mobility. We offer meeting-free Friday afternoons allowing more time for heads down work and professional development and through a robust body of employee programing we facilitate a wide range of opportunities to foster community learn and grow.

We are committed to fair transparent pay and we strive to provide competitive compensation in addition to a comprehensive benefits package. The range below represents Wileys good faith and reasonable estimate of the base pay for this role at the time of posting roles in the United Kingdom Canada USA Austria Czechia Denmark France Greece Italy Netherlands Romania or Spain. It is anticipated that most qualified candidates will fall within the range however the ultimate salary offered for this role may be higher or lower and will be set based on a variety of non-discriminatory factors including but not limited to geographic location skills and competencies.

When applying please attach your resume/CV to be considered.

Salary Range: