Evaluation and Application of Foundation Models for 6D Pose Estimation in Industrial Environments Using Digital Twins HF

Joining LINEACT at CESI for a research internship would be a fantastic opportunity to contribute to innovative projects while deepening my skills in a cuttingedge environment focused on digital transformation and Industry 4.0.

Works

Details of the tasks :

This M2 internship is part of the FUSION project and its Work Package 3 (WP3 whose goal is to update the digital twin of an industrial environment based on the robot visionbased perception. Foundation models are increasingly used in the literature across a wide range of applications. Also this is the case in 6D pose estimation with Wen et al.s proposal titled FoundationPose which achieves excellent results compared to stateoftheart methods. The approach has been tested on various datasets.

We aim to evaluate its performance in the context of manufacturing industrial environments through training performed using the digital twin of the production workshop.

The tasks assigned to the intern are as follows:

• Implement FoundationPose.
• Generate a synthetic dataset of the production workshop using the digital twin developed in Unity.
• Train FoundationPose on this data and evaluate the algorithms performance.
• Evaluate the algorithms performance on realworld data.
• If applicable adjust the initial dataset by incorporating labeled realworld data into the synthetic dataset and assess the performance of the hybrid dataset.

Project Context :

This recruitment is part of the FUSION project (Framework for Universal Software Integration in Open Robotics) which was selected under the IDmo France 2030 Regionalized Normandie call for projects. The projects partners are Conscience Robotics (lead) OREKA Ingnierie and CESI LINEACT.

The main objective of the FUSION project is to democratize the use of robotics by introducing a paradigm shift that places the user at the center of the system through:

• The introduction of XR for designing robotic missions and teleoperating robots via a digital twin;
• The reuse and sharing of software modules accessible to everyone;
• An innovative robotic perception approach using a semantic map to update the digital twin making robots increasingly autonomous.

The targeted use case focuses on dismantling operations within a nuclear site cell specifically the cutting of contaminated pipelines. Currently these operations are carried out by operators remotely controlling the robotic arm using only cameras installed on the intervention site and mounted on the robotic arm. This significantly complicates teleoperation due to the lack of depth perception. Our proposal aims first to reduce the complexity of robot teleoperation by replacing environment perception through cameras with immersion in a realtimegenerated digital twin of the work area. Secondly the project seeks to teach robots naturally to perform repetitive tasks that require only occasional supervision.

Laboratory Presentation

CESI LINEACT (UR 7527 the Digital Innovation Laboratory for Businesses and Learning in support of Territorial Competitiveness anticipates and supports technological transformations in sectors and services related to industry and construction. CESIs historical ties with businesses are a determining factor in its research activities leading to a focus on applied research in partnership with industry. A humancentered approach coupled with the use of technologies as well as regional networking and links with education have enabled crossdisciplinary research that centers on human needs and uses addressing technological challenges through these contributions.

Its research is organized into two interdisciplinary scientific teams and two application domains:

• Team 1 Learning and Innovating is primarily focused on Cognitive Sciences Social Sciences Management Sciences Education Science and Innovation Sciences. The main scientific objectives are understanding the effects of the environment particularly instrumented situations with technical objects (platforms prototyping workshops immersive systems) on learning creativity and innovation processes.
• Team 2 Engineering and Digital Tools is mainly focused on Digital Sciences and Engineering. Its main scientific objectives include modeling simulation optimization and data analysis of cyberphysical systems. Research also covers decisionsupport tools and studies of human system interactions especially through digital twins coupled with virtual or augmented environments.

These two teams cross and develop their research in the two application domains of Industry of the Future and City of the Future supported by research platforms primarily the Rouen platform dedicated to the Factory of the Future and the Nanterre platform dedicated to the Factory and Building of the Future.

Profile Sought : Masters in Computer Science with a focus on artificial intelligence computer vision.

Scientific and technical skills :

Skills : Artificial Intelligence and Computer Vision

Technical stack :

• Python & C
• C# (optional)
• PyTORCH DOCKER
• UNITY (optional)

Operating Systems : LINUX & WINDOWS

Interpersonal Skills :

• Autonomy initiative curiosity
• Teamwork ability and good interpersonal skills
• Rigorousness

Bonus at 15 of the Social Security hourly ceiling.

Starting date: February 2025

#CESILINEACT