Hello, Sushant!
For the past few days I have been trying to reproduce the results of the repository.
For that I followed the guide described in README.md but the outcome was different.
Steps:
- Clone the repo in a new directory
- Download IAM database from official site
- Copy
lines.txt file and lines directory to the data directory (13 353 records).
- In the file
DataLoader.py change the following line:
gtText_list = lineSplit[9].split('|')
to this:
gtText_list = lineSplit[8].split('|')
This is required because the 8-th element (not 9-th) contains ground truth labels. For example:
a01-000u-00 ok 154 19 408 746 1661 89 A|MOVE|to|stop|Mr.|Gaitskell|from
- Run the following command from
src_tensorflow2 directory:
python main.py --train
Environment:
Python: 3.7.9
Tensorflow: 2.7.0
Expected behaviour:
CER is expected to descend slowly approximately to the value specified in README.md: 8.32%.
Actual behaviour:
First try:
CER after epoch 1: 28.1%
CER after epoch 2: 21.0%
But from 3rd to at least 12th epoch CER is between 45% and 52%. And it is not going to go down.
Second try.
After 8th epoch:
Train loss: 62.25793147463152
Val loss: 64.84262824781013
Character error rate: 45.535652%
After 21th epoch:
Train loss: 56.68565004330704
Val loss: 66.37841461644028
Character error rate: 44.809107%
Could you describe the correct way to train the model?
Update 2022-06-09
It seems that the problem is reproduced only in src_tensorflow2 directory.
The code in src_tensorflow1 directory (using TF 1.15.5) after third epoch gives CER 19% and loss still going down.
Update 2022-06-10
The code in src_tensorflow1 directory (using TF 1.15.5) doesn't give stable results too.
I tried 3 more times to run the training from scratch. And CER was not decreasing from some epoch.
Hello, Sushant!
For the past few days I have been trying to reproduce the results of the repository.
For that I followed the guide described in README.md but the outcome was different.
Steps:
lines.txtfile andlinesdirectory to thedatadirectory (13 353 records).DataLoader.pychange the following line:gtText_list = lineSplit[9].split('|')to this:
gtText_list = lineSplit[8].split('|')This is required because the 8-th element (not 9-th) contains ground truth labels. For example:
a01-000u-00 ok 154 19 408 746 1661 89 A|MOVE|to|stop|Mr.|Gaitskell|fromsrc_tensorflow2directory:python main.py --trainEnvironment:
Python: 3.7.9
Tensorflow: 2.7.0
Expected behaviour:
CER is expected to descend slowly approximately to the value specified in README.md: 8.32%.
Actual behaviour:
First try:
CER after epoch 1: 28.1%
CER after epoch 2: 21.0%
But from 3rd to at least 12th epoch CER is between 45% and 52%. And it is not going to go down.
Second try.
After 8th epoch:
Train loss: 62.25793147463152
Val loss: 64.84262824781013
Character error rate: 45.535652%
After 21th epoch:
Train loss: 56.68565004330704
Val loss: 66.37841461644028
Character error rate: 44.809107%
Could you describe the correct way to train the model?
Update 2022-06-09
It seems that the problem is reproduced only in
src_tensorflow2directory.The code in
src_tensorflow1directory (using TF 1.15.5) after third epoch gives CER 19% and loss still going down.Update 2022-06-10
The code in
src_tensorflow1directory (using TF 1.15.5) doesn't give stable results too.I tried 3 more times to run the training from scratch. And CER was not decreasing from some epoch.