Data & AI Engineer

Must Have Technical / Functional Skills

• Strong core data engineering competencies with expertise in data concepts and data modelling
• Experience with Big Data platforms and scalable data pipeline design
• Hands-on experience with Microsoft Fabric Data Agents and Azure AI Services
• Knowledge of AI/ML integration within enterprise data platforms
• Strong analytical and problem-solving capabilities
• Expertise in performance tuning monitoring and optimization
• Proficiency in PySpark for large-scale data processing
• Ecommerce domain knowledge with customer behavior analytics experience
• Experience with Adobe Analytics Adobe Customer Journey Analytics (CJA) and clickstream data analysis

Roles & Responsibilities

Experimentation Data Enablement (Silver Layer Ownership)

• Design build and maintain curated Silver-layer datasets in Microsoft Fabric for experimentation reporting and analytics
• Collaborate with BI/Data Reporting teams to define dimensions metrics and joins such as visitor/session variant campaign geo device channel funnel steps and conversion events
• Develop reusable and standardized data products including tables and views for dashboards scorecards and ad hoc reporting
• Ensure Silver-layer datasets are clean conformed deduplicated and aligned with agreed business definitions

Data Gap Analysis & Assessment

• Conduct regular gap assessments across experimentation requirements existing Silver-layer data and upstream telemetry/source systems
• Identify missing fields inconsistent definitions data latency issues and join-key mismatches
• Document business impact severity remediation plans timelines and dependencies for identified issues
• Recommend improvements in data models including facts/dimensions surrogate keys grain definition and conformance rules

Gold Layer Requirements & Stakeholder Management

• Lead workshops with experimentation BI measurement and engineering teams to define Gold-layer reporting requirements
• Define KPI calculations attribution rules scorecard structures segmentation requirements governance standards and refresh SLAs
• Prepare functional and technical documentation including source-to-target mappings data dictionaries validation rules and acceptance criteria
• Ensure alignment on single-source-of-truth definitions across CJA Power BI and scorecards

Data Pipeline Engineering

• Build and maintain robust pipelines using Microsoft Fabric Pipelines and Azure Data Factory (ADF)
• Work with 1DS telemetry pipelines or equivalent systems to ensure accurate event and attribute flow into Fabric
• Implement orchestration incremental loads monitoring and error-handling mechanisms to meet reporting timelines

Data Validation & Reconciliation

• Perform reconciliation between Silver/Gold datasets and Customer Journey Analytics (CJA)
• Validate event counts session/user logic experiment attribution conversions and time-window consistency
• Build automated checks for missing data duplicate events schema drift and metric anomalies
• Coordinate issue resolution with telemetry tagging product engineering and reporting teams

Experimentation Lifecycle Support

• Ensure datasets are ready for pre-launch checks measurement scorecard generation health checks and post-test analysis
• Curate experiment metadata including test IDs allocation details start/end dates KPI metrics and slicing dimensions
• Support consistent and reliable experimentation scorecard generation

AI Agent Design & Development

• Design and develop AI-powered agents using Fabric Data Agents Copilot and Azure OpenAI
• Enable automation for scorecard creation narrative summaries self-service analytics anomaly investigation and metric definition assistance
• Define agent scope personas grounding datasets RBAC/security models and evaluation metrics
• Partner with experimentation and reporting teams for pilot implementation feedback gathering and production rollout

Documentation Governance & Operational Excellence

• Maintain detailed documentation for datasets transformation logic metric definitions pipelines validation rules and operational runbooks
• Establish standards for naming conventions semantic consistency versioning backward compatibility and performance optimization
• Provide operational support including monitoring troubleshooting incident management and continuous improvement initiatives