MoSSNet: Monocular Shape Sensing for Continuum Robots

MoSSNet is an eye-to-hand monocular approach to continuum robot shape sensing. Utilizing a deep encoder-decoder network, MoSSNet achieves accurate and real-time shape sensing without fiducial markers, manual segmentation, or camera calibration.

Authors: Chengnan Shentu*, Enxu Li*, Chaojun Chen, Puspita Triana Dewi, David B. Lindell, Jessica Burgner-Kahrs

IEEE Robotics and Automation Letters (RA-L)

https://ieeexplore.ieee.org/abstract/document/10372078

https://arxiv.org/abs/2303.00891

- UPDATE (October 2024) -
We identified a bug in the metric computation function, which impacted the reported
performance. Additionally, we performed further hyperparameter tuning to restore
the models’ performance to levels close to the originally reported results.
The code and model checkpoints have now been corrected.

Requirements and Installation

Our implementation is based on python 3.7 and Cuda 11.1. Tested on Ubuntu 20.04.

git clone https://github.com/ContinuumRoboticsLab/MoSSNet.git
cd MoSSNet
pip install -r requirements.txt

Datasets and Checkpoints

Our dataset is based on a 2 segment tendon-driven continuum robot (TDCR) that is 250mm in length. You can view sample data in /MoSSNet/dataset_sample.

Dataset Name	#Shapes	Zip File Size	Link
MoSS-Sim	50k	1.4G	[OneDrive] [GoogleDrive]
MoSS-Real	17.5k	18.3G	[OneDrive] [GoogleDrive]
Disturbed-Real	5.1k	5.3G	[OneDrive] [GoogleDrive]

*If a link has expired, you can send us an email at [email protected] and we'll fix it ASAP.

Checkpoint Name	Training Set	Zip File Size	Link
checkpoint_mossnet_sim	MoSS-Sim/train	0.15G	[OneDrive] [GoogleDrive]
checkpoint_mossnet_real	MoSS-Real/train	0.15G	[OneDrive] [GoogleDrive]

---

The datasets are already split with 60-15-25 train-validation-test ratio. You can merge the folders and re-split using the commands below.

python generate_splits.py -d /path_to_repo/dataset/MoSS-Sim
python generate_splits_real.py -d /path_to_repo/dataset/MoSS-Real

Training

Example Training command

python trainer.py -c moss_sim [-p logs/moss_sim/model_best.pth.tar]

You can add more config yaml files in ./cfg and replace -c moss_sim with -c xxx after adding xxx.yaml. You can use the -p flag to train from a previous checkpoint.

Example Training command using debug mode

This setting will only train and eval 100 iterations every epoch to debug model

python trainer.py -c moss_sim --debug

Inference

Example inference command

python evaluator.py -l /logs/moss_sim [-c alternative_checkpoint]

The evaluator follows the config file generated in the log folder during training. By default it will evaluate the model on the validation set. You can modify the logs/log_name/cfg.yaml file to evaluate on the test set for example.

Using our checkpoints

Unzip the checkpoints into MoSSNet/logs.

python evaluator.py -l /logs/checkpoint_mossnet_sim
python evaluator.py -l /logs/checkpoint_mossnet_real

We have modified the cfg.yaml files to evaluate on the test set of MoSS-Sim and MoSS-Real respectively. In the cfg.yaml file for the real checkpoint, if you change the eval base path to dataset/Disturbed-Real, the Disturbed-Real test set will be used instead. You should see the same results as Table 2 in our paper (fps will depend on compute hardware).

Citation

If you find our paper and code useful in your research, please consider giving a star and citation :D

  @article{shentu2024moss,
  author={Shentu, Chengnan and Li, Enxu and Chen, Chaojun and Dewi, Puspita T. and Lindell, David B. and Burgner-Kahrs, Jessica},
  journal={IEEE Robotics and Automation Letters}, 
  title={MoSS: Monocular Shape Sensing for Continuum Robots}, 
  year={2024},
  volume={9},
  number={2},
  pages={1524-1531},
  doi={10.1109/LRA.2023.3346271}}