Commit Graph

292 Commits

Author SHA1 Message Date
Daniel Tinazzi
4fa8dd38c6
Fixed M-AILABS broken link
I replaced the broken link with the updated one that I found on the same website here:
http://www.caito.de/2019/01/the-m-ailabs-speech-dataset/
2021-11-17 14:20:19 +01:00
Alexandre Lissy
bde1ebc842 Fix #3608: Remove code refs to TaskCluster 2021-04-08 16:28:49 +02:00
Dustin Zubke
6945663698 Fix #3511: split-sets on sample size 2021-02-28 16:09:37 -05:00
Reuben Morais
f2e57467c6 Compare sample durations with an epsilon 2021-01-18 16:20:03 +00:00
Reuben Morais
d4152f6e67 Add support for Ogg/Opus audio files for training 2021-01-18 12:11:31 +00:00
Catalin Voss
6640cf2341
Remote training I/O once more (#3437)
* Redo remote I/O changes once more; this time without messing with taskcluster

* Add bin changes

* Fix merge-induced issue?

* For the interleaved case with multiple collections, unpack audio on the fly

To reproduce the previous failure

rm data/smoke_test/ldc93s1.csv
rm data/smoke_test/ldc93s1.sdb
rm -rf /tmp/ldc93s1_cache_sdb_csv
rm -rf /tmp/ckpt_sdb_csv
rm -rf /tmp/train_sdb_csv

./bin/run-tc-ldc93s1_new_sdb_csv.sh 109 16000
python -u DeepSpeech.py --noshow_progressbar --noearly_stop --train_files ./data/smoke_test/ldc93s1.sdb,./data/smoke_test/ldc93s1.csv --train_batch_size 1 --feature_cache /tmp/ldc93s1_cache_sdb_csv --dev_files ./data/smoke_test/ldc93s1.sdb,./data/smoke_test/ldc93s1.csv --dev_batch_size 1 --test_files ./data/smoke_test/ldc93s1.sdb,./data/smoke_test/ldc93s1.csv --test_batch_size 1 --n_hidden 100 --epochs 109 --max_to_keep 1 --checkpoint_dir /tmp/ckpt_sdb_csv --learning_rate 0.001 --dropout_rate 0.05 --export_dir /tmp/train_sdb_csv --scorer_path data/smoke_test/pruned_lm.scorer --audio_sample_rate 16000

* Attempt to preserve length information with a wrapper around `map()`… this gets pretty python-y

* Call the right `__next__()`

* Properly implement the rest of the map wrappers here……

* Fix trailing whitespace situation and other linter complaints

* Remove data accidentally checked in

* Fix overlay augmentations

* Wavs must be open in rb mode if we're passing in an external file pointer -- this confused me

* Lint whitespace

* Revert "Fix trailing whitespace situation and other linter complaints"

This reverts commit c3c45397a2.

* Fix linter issue but without such an aggressive diff

* Move unpack_maybe into sample_collections

* Use unpack_maybe in place of duplicate lambda

* Fix confusing comment

* Add clarifying comment for on-the-fly unpacking
2020-12-07 13:07:34 +01:00
Alexandre Lissy
c822a6e875 Importer for dataset from Centre de Conférences Pierre Mendès-France
Released by Ministère de l'Economie, des Finances, et de la Relance
2020-11-24 09:49:39 +01:00
Reuben Morais
f5cbda694a Revert "Merge pull request #3424 from mozilla/io-fixes"
This reverts commit ab1288ffde, reversing
changes made to 08d18d7328.
2020-11-19 16:58:01 +02:00
CatalinVoss
9aaa0e406b Make sure to unpack samples now 2020-11-17 14:31:48 -08:00
Liezl P
af7c4e90df Add missing sys import to import_voxforge.py 2020-10-22 23:09:49 -10:00
Anas Abou Allaban
521842deea
Convert channels for CV2 dataset
When running a training session on the CV2 dataset, it is possible to get the following error:

```
ValueError: Mono-channel audio required
```

This makes the [pysox Transformer](https://pysox.readthedocs.io/en/latest/api.html#sox.transform.Transformer.convert) also convert the channels.
2020-10-15 11:22:39 -04:00
Reuben Morais
83a36b7a34 Rename --utf8 flag to --bytes_output_mode to avoid confusion 2020-10-06 18:19:33 +02:00
Reuben Morais
fb4f5b6a84 Add some coverage for training and inference in bytes output mode 2020-10-06 18:19:33 +02:00
Greg Cooke
20ad86c6ab Added os import in front of makedirs 2020-09-16 14:20:59 -04:00
Eren Gölge
b2df360799
fix missing import 'sys' 2020-09-08 10:15:22 +02:00
Reuben Morais
da55cfae86 Revert "Merge pull request #3237 from lissyx/rename-training-package"
This reverts commit 3dcb3743ac, reversing
changes made to 457198c88d.
2020-08-26 11:46:08 +02:00
Reuben Morais
9788811bc5 Revert "Merge pull request #3241 from lissyx/rename-ctcdecoder"
This reverts commit fd4185f141, reversing
changes made to 1a7dd87601.
2020-08-26 11:45:06 +02:00
Tilman Kamp
96f37a403d Resolves #3235 - Support for .tar(.gz) targets in bin/data_set_tool.py 2020-08-13 10:21:45 +02:00
Alexandre Lissy
ccd9241bd0 Rename ctcdecoder python package 2020-08-10 22:45:43 +02:00
Alexandre Lissy
6f84bd1996 Rename deepspeech_training package 2020-08-10 16:58:18 +02:00
Tilman Kamp
a982a61d83 Resolves #3146 - Let build_sdb.py also output CSV files and rename it accordingly 2020-07-21 17:02:01 +02:00
Karthikeyan Singaravelan
0f27c802d9 Use ElementTree instead of deprecated cElementTree. 2020-07-15 12:46:00 +00:00
Tilman Kamp
61bd5dd88d Resolves #3144 - Add augmentation support to build_sdb.py 2020-07-09 11:55:26 +02:00
Tilman Kamp
084da3724d Fix: #3130 - Missing deepspeech_training.util.text.Alphabet 2020-07-07 17:02:24 +02:00
Tilman Kamp
eebf12134e Warp augmentation 2020-06-29 16:22:31 +02:00
Tilman Kamp
4c6245d155
Merge pull request #3055 from tilmankamp/augext
Refactoring of TF based augmentations
2020-06-18 16:13:46 +02:00
Tilman Kamp
5dd08d2f8e Deactivated scorer in graph augmentation test 2020-06-16 16:57:09 +02:00
Tilman Kamp
7a835bee5a Updated training tests 2020-06-16 13:51:07 +02:00
Tilman Kamp
0bec67d74c
Update bin/play.py
Co-authored-by: Reuben Morais <reuben.morais@gmail.com>
2020-06-16 10:10:01 +02:00
Tilman Kamp
d94db7ca43 Refactoring of TF based augmentations 2020-06-10 13:42:45 +02:00
Alexandre Lissy
cfc79799ec Report imported vs total audio time 2020-06-10 13:12:15 +02:00
Reuben Morais
ecd79531c8 Add training test with --metrics_files 2020-06-08 18:06:21 +02:00
RickyChan
a252ae01a0 Fix csv DictWriter parameter 2020-06-05 22:27:19 +09:00
RickyChan
3c83f9f24a Fix csv writer parameter [https://docs.python.org/3/library/csv.html#csv.writer] 2020-06-05 19:00:52 +08:00
Reuben Morais
3d0ec01853 Fix typo from argument reordering 2020-05-27 19:02:55 +02:00
Reuben Morais
45d8f7cd61 Explicitly pass filter context to multiprocessing function 2020-05-25 18:00:08 +02:00
Tilman Kamp
a5303ccca6 Renamed prepare_samples to augment_samples 2020-05-14 16:50:18 +02:00
Tilman Kamp
c5ceee26dd Live audio augmentation 2020-05-12 10:18:21 +02:00
Tilman Kamp
3871cdc67f CV2 importer: Writes additional train-all.csv with all validated samples except speakers and/or transcripts already in dev or test 2020-05-07 15:09:42 +02:00
GaGo
3daca6f209
Update import_swc.py
I tried to use the importer. With the Error path not defined.

I think it's appear after the refactor by @reuben.
Nothing big, but I think an commit worth. :)
2020-04-23 14:45:39 +02:00
Tilman Kamp
e99e06a278 M-AILAB importer: Ensure all samples are 16 kHz 2020-04-20 17:34:58 +02:00
Jim Regan
5a7e4ea348
import_lingua_libre.py: n channels + bitdepth 2020-04-16 22:44:32 +02:00
Reuben Morais
0c6e90868e Split --load into two to avoid unexpected behavior at evaluation time 2020-04-07 14:24:05 +02:00
Reuben Morais
c738d55012 Remove unneeded six.moves import 2020-03-31 13:57:44 +02:00
Reuben Morais
2f68ed1001 Remove unneeded future imports from importers 2020-03-31 13:55:32 +02:00
Reuben Morais
6f0bf3b3a8 Reformat importers with black 2020-03-31 13:43:30 +02:00
Reuben Morais
b7e6b8c3e6 Sort importer imports with isort 2020-03-31 13:43:00 +02:00
Reuben Morais
20b0ab17ea Remove unused GPU usage tools 2020-03-31 13:42:41 +02:00
Reuben Morais
a05baa35c9 Package training code to avoid sys.path hacks 2020-03-25 21:34:50 +01:00
Tilman Kamp
41da7b2870 Fix #2830 - Support for unlabeled samples 2020-03-24 16:53:02 +01:00