Abschlussarbeiten, Projekte für Studenten und offene Stellen

In potential navigation systems for minimally invasive surgeries, models created by CT and MRT scans are rendered and overlaid transparently to be shown on the endoscopic video stream. This might lead to sorting errors, because occlusions that usually work as visual cues for depth perception are ambiguous due to the transparent rendering. Explore and implement different rendering methods that could help users in the correct interpretation of the augmented reality overlay. (contact)

Multimodal foundation models typically require fine-tuning for specific downstream tasks. However, transferring their knowledge efficiently across diverse tasks, especially those involving heterogeneous data like medical images and text—remains a major challenge. This project explores the Mixture-of-Experts (MoE) architecture as a solution, enabling task-specific specialization while maintaining a shared knowledge base. The candidate will design, implement, and evaluate an MoE model on ~20 downstream tasks spanning Computational Pathology, Radiology, and Natural Language.

Requirements:
- Strong background in deep learning and PyTorch
- Experience with multimodal data (e.g., vision, language, medical imaging)
- Familiarity with distributed training or large-scale model training is a plus
- Self-driven and able to manage a complex research pipeline (contact) 

Visual Question Answering (VQA) in surgical contexts is a challenging task due to the domains inherent data scarcity and the complexity of surgical scenes. While specialized Vision-Language-Models (VLMs) have been proposed, this project focuses on evaluating how effectively general-purpose VLMs can be adapted to the surgical domain. Help us benchmark pre-trained VLMs on surgical VQA tasks. Fine-tune state-of-the-art VLMs on existing datasets and evaluate question-answer biases to check for text-prior contamination. (contact)

Recent imitation-learning-based approaches have shown remarkable results in learning surgical robotic tasks autonomously, when equipped with deep learning models. However, these successes rely on multi-camera setups, likely not feasible in Minimally-Invasive-Surgical (MIS) settings due to space constraints. Leverage Foundation Stereo to create a 3D point cloud of the MIS scene (link) and use it to render multiple virtual views (link), then train a state-of-the-art imitation learning model (Diffusion Policy, Action-Chunking-Transformer) to predict keypoints for the robot path planning in the needle driving task (in the picture).

Required skills:
- Good proficiency in python.
- Previous experience with pytorch and deep neural networks.
- Knowledge of CV concepts like depth estimation, camera calibration, etc. is a plus.
- No prior knowledge of imitation learning or robotics is needed(contact)

Building upon recent advancements in generative models, use conditional flow matching for visuomotor imitation learning of surgical sub-tasks like needle driving or pick-and-place. A recent line of work builds on successes in diffusion models for imitation learning. It has been hinted here (link) that simplicity of flow matching objectives allows favorable performance in stable training and generation quality compared to stochastic denoising diffusion process. Build an imitation learning algorithm using flow matching inspired by this work (link) and benchmark its performance with respect to diffusion policy learning and other simpler frameworks.
Required skills:
- Good proficiency in python and pytorch
- Basic mathematical understanding of deep generative modeling and ODEs/SDEs
- Previous experience in imitation/reinforcement learning is a plus.
- No prior knowledge of robotics is required! (contact)

Remote surgery is the advancing research field of deploying and telemanipulating surgical robotic systems over large networks. Naturally, connections over such distances may induce network latency or even loss of signal. In order to design systems that tolerate these disturbances we can use model predictive control (MPC) to bridge occuring latency gaps locally. To further enhance the compensation mechanism we want to incorporate learned, task specific policies to augment pure non learning based MPC methods.
In particular, in the scope of this Master Thesis you are going to use Reinforcement Learning to augment an extended Kalman Filter for latency compensation. The learned task will be a simple grasping task of a surgical training object inside a simulated environment (Nvidia IsaacSim).

Requirements
- Proficient with Python (mandatory)
- Linux CLI (highly recommended)
- Deep Learning / Pytorch (beneficial but not mandatory)
- MPC methods, specifically Kalman Filters (beneficial but not mandatory)
- Docker (nice to have)
- IsaacSim / Omniverse (nice to have) (contact)