DeepSpeech/training/deepspeech_training/util
Catalin Voss 6640cf2341
Remote training I/O once more (#3437)
* Redo remote I/O changes once more; this time without messing with taskcluster

* Add bin changes

* Fix merge-induced issue?

* For the interleaved case with multiple collections, unpack audio on the fly

To reproduce the previous failure

rm data/smoke_test/ldc93s1.csv
rm data/smoke_test/ldc93s1.sdb
rm -rf /tmp/ldc93s1_cache_sdb_csv
rm -rf /tmp/ckpt_sdb_csv
rm -rf /tmp/train_sdb_csv

./bin/run-tc-ldc93s1_new_sdb_csv.sh 109 16000
python -u DeepSpeech.py --noshow_progressbar --noearly_stop --train_files ./data/smoke_test/ldc93s1.sdb,./data/smoke_test/ldc93s1.csv --train_batch_size 1 --feature_cache /tmp/ldc93s1_cache_sdb_csv --dev_files ./data/smoke_test/ldc93s1.sdb,./data/smoke_test/ldc93s1.csv --dev_batch_size 1 --test_files ./data/smoke_test/ldc93s1.sdb,./data/smoke_test/ldc93s1.csv --test_batch_size 1 --n_hidden 100 --epochs 109 --max_to_keep 1 --checkpoint_dir /tmp/ckpt_sdb_csv --learning_rate 0.001 --dropout_rate 0.05 --export_dir /tmp/train_sdb_csv --scorer_path data/smoke_test/pruned_lm.scorer --audio_sample_rate 16000

* Attempt to preserve length information with a wrapper around `map()`… this gets pretty python-y

* Call the right `__next__()`

* Properly implement the rest of the map wrappers here……

* Fix trailing whitespace situation and other linter complaints

* Remove data accidentally checked in

* Fix overlay augmentations

* Wavs must be open in rb mode if we're passing in an external file pointer -- this confused me

* Lint whitespace

* Revert "Fix trailing whitespace situation and other linter complaints"

This reverts commit c3c45397a2.

* Fix linter issue but without such an aggressive diff

* Move unpack_maybe into sample_collections

* Use unpack_maybe in place of duplicate lambda

* Fix confusing comment

* Add clarifying comment for on-the-fly unpacking
2020-12-07 13:07:34 +01:00
..
__init__.py Revert "Merge pull request #3237 from lissyx/rename-training-package" 2020-08-26 11:46:08 +02:00
audio.py Remote training I/O once more (#3437) 2020-12-07 13:07:34 +01:00
augmentations.py Remote training I/O once more (#3437) 2020-12-07 13:07:34 +01:00
check_characters.py Remote training I/O once more (#3437) 2020-12-07 13:07:34 +01:00
checkpoints.py Fix lr initialization on reload. 2020-08-27 15:08:32 +02:00
config.py Remote training I/O once more (#3437) 2020-12-07 13:07:34 +01:00
downloader.py Remote training I/O once more (#3437) 2020-12-07 13:07:34 +01:00
evaluate_tools.py Remote training I/O once more (#3437) 2020-12-07 13:07:34 +01:00
feeding.py Revert "Merge pull request #3237 from lissyx/rename-training-package" 2020-08-26 11:46:08 +02:00
flags.py Rename --utf8 flag to --bytes_output_mode to avoid confusion 2020-10-06 18:19:33 +02:00
gpu.py Revert "Merge pull request #3237 from lissyx/rename-training-package" 2020-08-26 11:46:08 +02:00
helpers.py Remote training I/O once more (#3437) 2020-12-07 13:07:34 +01:00
importers.py Revert "Merge pull request #3237 from lissyx/rename-training-package" 2020-08-26 11:46:08 +02:00
io.py Remote training I/O once more (#3437) 2020-12-07 13:07:34 +01:00
logging.py Revert "Merge pull request #3237 from lissyx/rename-training-package" 2020-08-26 11:46:08 +02:00
sample_collections.py Remote training I/O once more (#3437) 2020-12-07 13:07:34 +01:00
stm.py Revert "Merge pull request #3237 from lissyx/rename-training-package" 2020-08-26 11:46:08 +02:00
taskcluster.py Revert "Merge pull request #3420 from CatalinVoss/remote-io" 2020-11-19 16:58:21 +02:00
text.py Revert "Merge pull request #3237 from lissyx/rename-training-package" 2020-08-26 11:46:08 +02:00