Applied Researcher On-Device Multimodal Reasoning

The Video Computer Vision (VCV) organization is an applied research and engineering team developing real-time on-device Computer Vision and Machine Perception technologies across Apple products. Within VCV our team builds next-generation multimodal AI systems that combine on-device multimodal encoders large language models and foundation models to create intelligent systems capable of understanding reasoning and acting across language vision audio and tools. Our work is deeply integrated into the Apple ecosystem partnering across hardware software and ML teams to deliver real-time scalable and privacy-preserving experiences reaching millions of users.

We are seeking an Applied Researcher with deep expertise in multimodal reasoning at small model scale making vision-language models in the smaller regime (under 10B parameters down to sub-1B) think plan and act reliably under strict compute memory and latency this role you will own the reasoning side of the on-device multimodal stack: designing compact VLMs that reason over images video and 3D scene content; compressing and distilling the reasoning process itself; and engineering the decoding and inference path that makes multi-step reasoning affordable on an Apple device. This role offers the unique opportunity to define what on-device intelligence looks like for hundreds of millions of users. Youll push the boundaries of what small models can achieve enabling real-time multimodal understanding and multi-step reasoning without reliance on cloud connectivity. Youll collaborate with hardware teams compiler engineers and ML researchers to unlock capabilities that few organizations can deliver at Apples scale and quality role spans multiple dimensions of efficient on-device reasoning including VLM architecture and connector design reasoning post-training (SFT/RL) chain-of-thought compression speculative and structured decoding visual token reduction quantization and distillation and hardware-aware inference optimization. nnA core focus of this role is efficient reasoning: compressed and latent chain-of-thought reasoning distillation from frontier teachers adaptive test-time compute (knowing when and how long to think) speculative and structured decoding KV-cache compression and visual-token efficiency. A second focus is reasoning over real-time visual perception experts. Rather than consuming pixels alone the VLM should be able to invoke and reason over the outputs of specialist on-device vision models feed-forward 3D scene and geometry estimators (VGGT-style reconstruction depth camera pose) human body and hand mesh/pose recovery object detectors localizers and trackers and fuse those structured metric outputs into its reasoning about the scene. This raises real research questions: how to represent geometry body parameters and detections compactly in a token-budgeted context; how to schedule which experts run at which frame rate within a real-time budget; and how to train a small model to invoke trust and cross-check them. Efficiency is treated as a first-class metric here: reasoning quality is measured at a fixed latency memory and power budget.

Design train and post-train compact vision-language models (under 10B including sub-1B) that perform multi-step visual reasoning grounded visual understanding and language generation within on-device resource budgetsnResearch and implement efficient reasoning techniques compressed and latent chain-of-thought reasoning-trace distillation early-exit and budget-aware reasoning adaptive compute allocation and test-time scaling that maximizes reasoning quality per FLOPnOwn the decoding stack for on-device inference: speculative and self-speculative decoding draft models and multi-token prediction structured/constrained generation KV-cache compression and quantization prefill/decode scheduling and streaming latency (TTFT tokens/sec)nBuild reasoning over real-time perception experts: enable a small VLM to invoke and reason over on-device 3D scene reconstruction and geometry (VGGT-style feed-forward reconstruction depth camera pose) human body/hand pose and mesh recovery object detection localization and tracking designing the representations interfaces and training signals that make those outputs usable inside a limited contextnDevelop structured-output fusion and expert scheduling: compact tokenizations for geometry body parameters and detections; policies for which perception models run at which resolution and frame rate; and mechanisms for the reasoner to resolve conflicts between experts and its own visual featuresnApply reasoning-focused post-training: supervised distillation from frontier teachers preference and RL methods (GRPO RLVR STaR rejection sampling) verifier- and reward-guided decoding and process supervision for multimodal and spatially grounded chainsnDrive visual token efficiency and feature compression: token pruning merging and resampling adaptive resolution and frame-rate policies and learned connectors that preserve reasoning accuracy at a fraction of the visual token budgetnDevelop distillation pruning and quantization strategies that preserve multimodal reasoning fidelity at reduced model sizes including QAT and mixed-precision inference across the language decoder vision encoder and perception expertsnExplore hybrid architectures combining stateful components (SSM/Mamba linear attention) with attention-based components to balance long-context visual reasoning long video multi-image extended dialogue against inference efficiency and bounded memorynOptimize reasoning and encoder architectures for Apple silicon including Neural Engine GPU and ANE-aware design patterns; profile and iterate on latency memory footprint thermal behavior and power consumption across Apples device portfolionPartner with our visual representation and video-encoder efforts (self-supervised and joint-embedding pretraining streaming/stateful encoders world models) to make representations reasoning-ready and co-design the encoder-LLM interfacenBuild evaluation frameworks that measure reasoning fidelity visual and spatial grounding hallucination and robustness under device budgets including accuracy-vs-latency and accuracy-vs-power trade-off curvesnCollaborate with hardware compiler and platform teams to co-design model architectures that exploit device-specific acceleration capabilities

MS in Computer Science Machine Learning AI Computer Vision or a related field (or equivalent practical experience)nStrong foundation in deep learning with specific experience in LLM or VLM training post-training or inference optimizationnDemonstrated experience working with multimodal models (vision-language models multimodal LLMs) in resource-constrained environments including hands-on work with reasoning quality decoding or model compressionnProficiency in Python and modern deep learning frameworks (PyTorch preferred) with familiarity with inference and optimization toolchains (quantization distillation pruning e.g. vLLM/SGLang MLX CoreML)

PhD with research in efficient multimodal reasoning LLM reasoning model compression efficient inference/decoding or lightweight VLM architecturesnExperience training or post-training vision-language models end-to-end connector/projector design visual instruction tuning resolution and token-budget trade-offs small-model recipesnHands-on experience with reasoning techniques: chain-of-thought distillation and compression latent/implicit reasoning reward-guided decoding RL for reasoning (GRPO RLVR STaR) or test-time compute allocationnExpertise in decoding and serving optimizations: speculative decoding structured/grammar-constrained generation KV-cache quantization and eviction continuous batching long-context inferencenExperience combining LLMs with real-time perception models 3D reconstruction and geometry (VGGT DUSt3R/MASt3R-style SLAM monocular depth) human pose and body/hand mesh recovery (SMPL-family) detection segmentation or tracking and with spatial or 3D-grounded reasoning and embodied/spatial VQAnExperience deploying LLM or multimodal models on mobile or edge hardware (CoreML MLX TensorRT-LLM or equivalent) with attention to ANE/GPU kernel and memory constraintsnExperience with quantization-aware training mixed-precision inference and knowledge distillation for vision-language modelsnFamiliarity with efficient vision encoders and self-supervised/joint-embedding pretraining (V-JEPA I-JEPA MAE DINO/DINOv2 SigLIP CLIP) Mamba/SSM vision backbones or streaming architectures for real-time video with fixed memory budgetsnInterest in Video-LLMs long-video reasoning and world models for prediction and planningnPublication record in top-tier venues is a plus (NeurIPS ICML ICLR CVPR ECCV ACL MLSys ICRA etc.)