Error while training on SageMaker #2408
Unanswered
ChikkathotadappaAshwin-zz
asked this question in
Help
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
During training on Sagemaker , the training was stopped with displaying below error, but with same code I was able to succeed running earlier, due to unsatisfactory results, I increased my training data and started to train once again
After making several searches, I found which is answered for similar type of issue, please let me know, if this is the same error
UnexpectedStatusException: Error for Training job train-job-fp-od-initial-fp-aug-20210525-exp-3: Failed. Reason: AlgorithmError: ExecuteUserScriptError:
Command "/usr/bin/python3 train_sm_toolkit.py
message from log:
Traceback (most recent call last):
File "train_sm_toolkit.py", line 432, in
train(args)
File "train_sm_toolkit.py", line 288, in train
train_generator, validation_generator = create_generators(args, backbone.preprocess_image)
File "train_sm_toolkit.py", line 256, in create_generators
**common_args
File "/opt/ml/code/keras_retinanet/preprocessing/coco.py", line 46, in init
super(CocoGenerator, self).init(**kwargs)
File "/opt/ml/code/keras_retinanet/preprocessing/generator.py", line 89, in init
self.group_images()
File "/opt/ml/code/keras_retinanet/preprocessing/generator.py", line 291, in group_images
order.sort(key=lambda x: self.image_aspect_ratio(x))
File "/opt/ml/code/keras_retinanet/preprocessing/generator.py", line 291, in
order.sort(key=lambda x: self.image_aspect_ratio(x))
File "/opt/ml/code/keras_retinanet/preprocessing/coco.py", line 124, in image_aspect_ratio
image = self.coco.loadImgs(self.image_ids[image_index])[0]
File "/usr/local/lib/python3.6/dist-packages/pycocotools/coco.py", line 229, in loadImgs
return [self.imgs[id] for id in ids]
File "/usr/local/lib/python3.6/dist-packages/pycocotools/coco.py", line 229, in
return [self.imgs[id] for id in ids]
Beta Was this translation helpful? Give feedback.
All reactions