Commit Graph

59 Commits

Author SHA1 Message Date
Reuben Morais
f82c77392d Rewrite data/lm/generate_package.py into native_client/generate_scorer_package.cpp 2020-06-30 09:52:44 +02:00
Alexandre Lissy
6c2cbbd725 Fix #3053: Check output stream when producing scorer 2020-06-16 23:28:01 +02:00
Reuben Morais
4356a2764b Add data/lm doc to RTD, and some general doc improvements and fixes 2020-05-24 15:35:10 +02:00
Reuben Morais
a590e3726b Add link to RTD, actually exit on error 2020-05-18 19:04:33 +02:00
Reuben Morais
e8647aa5fa Add missing import in generate_package.py 2020-05-18 18:52:30 +02:00
Reuben Morais
361e216297 Only ignore (expected) missing trie error in generate_package.py 2020-05-18 17:30:49 +02:00
david gauchard
117324e665
Add a --discount_fallback option to generate_lm.py (#2945) 2020-04-28 11:58:41 +02:00
kdavis-mozilla
f3b4943e18 Updated alpha and beta 2020-04-23 07:19:37 +02:00
DanBmh
bfe778482c
Refactor generate_package.py (#2903)
* Improve formatting and paths in LM README

* Improve logging in generate_package.py

Co-authored-by: Daniel <daniel@mail.de>
2020-04-17 21:20:45 +02:00
Reuben Morais
c27387fd98 README tweaks 2020-04-14 13:07:44 +02:00
Daniel
f82a77f249 Update readme. 2020-04-14 13:00:23 +02:00
Daniel
8c73bf6fbf Small fixes. 2020-04-09 16:58:22 +02:00
Daniel
00e4dbe3fd Merge remote-tracking branch 'upstream/master' 2020-04-08 20:27:43 +02:00
Daniel
c29c0beb72 Default to required params. 2020-04-08 20:23:04 +02:00
Daniel
a291e23041 Update readme. 2020-04-04 10:50:28 +02:00
Daniel
e16b72ff28 Use os.join and kenlm parameter usage description. 2020-04-03 17:58:52 +02:00
Daniel
e862cd41db Read from input.txt.gz again. 2020-04-01 17:29:02 +02:00
Daniel
38afe38f0b Implement some change request. 2020-04-01 17:15:52 +02:00
Daniel
b27e0347b1 Add more parameters.
Implement some change request.
2020-04-01 16:54:58 +02:00
Reuben Morais
83d22e591b
Merge pull request #2856 from reuben/training-install
Package training code to avoid sys.path hacks
2020-03-31 15:42:42 +02:00
Reuben Morais
09673581a4 Error early in generate_package.py if no alphabet was specified and not using UTF-8 mode
X-DeepSpeech: NOBUILD
2020-03-30 18:16:56 +02:00
Daniel
f97c79e0e8 Make generate_lm.py language independent. 2020-03-29 12:29:18 +02:00
Reuben Morais
a05baa35c9 Package training code to avoid sys.path hacks 2020-03-25 21:34:50 +01:00
Daniel
f808720b5b Update readme. 2020-03-09 16:34:04 +01:00
Daniel
9c73700ac7 Add error hint and default values for alpha and beta. 2020-03-09 16:26:57 +01:00
Daniel
ef095881ca Fix too many arguments for format string. 2020-03-03 16:58:47 +01:00
Daniel
c6109c30f3 Add some statistics. 2020-03-03 16:49:52 +01:00
Daniel
c9a433486f Add more arguments. Rename file variables. 2020-03-03 16:48:43 +01:00
Daniel
c505a4ec6c Update some comments. 2020-02-27 17:46:16 +01:00
Daniel
15a75c77ff Rewrite generate_lm.py to allow usage with other languages. 2020-02-27 17:18:19 +01:00
Reuben Morais
8dedda7759 Address review comments 2020-02-11 19:44:36 +01:00
Reuben Morais
1d3b3a31a1 Address review comments and update docs 2020-02-11 19:44:36 +01:00
Reuben Morais
3b54f54524 Fix linter errors
X-DeepSpeech: NOBUILD
2020-02-11 19:44:36 +01:00
Reuben Morais
1e2eb96248 Update all API consumers 2020-02-11 19:44:36 +01:00
Reuben Morais
b34723588d Switch to new scorer format 2020-02-11 19:44:29 +01:00
Reuben Morais
16d5632d6f Write default values for alpha and beta into trie header 2020-02-11 19:44:26 +01:00
Reuben Morais
214b50f490 Add generate_package tool to create combined scorer package 2020-02-11 19:44:26 +01:00
Reuben Morais
7c0354483e Stop including vocabulary data in LM.binary. 2020-02-11 19:44:19 +01:00
Reuben Morais
381faaf6b6 Switch to --prune 0 0 1 model and move generation code to a script 2019-11-15 13:28:45 +01:00
Reuben Morais
ad2769f479 Filter LM by removing very rare words 2019-11-13 17:38:40 +01:00
Reuben Morais
f4cdd988df UTF-8 target 2019-11-11 11:36:16 +01:00
Alexandre Lissy
9ce8c24165 Move from Markdown to reStructuredText 2019-10-04 12:07:32 +02:00
Reuben Morais
585d94df7f Update trie files to renenerated versions 2019-06-21 23:24:21 -03:00
Reuben Morais
60fb5ad04c Remove old versions of decoder binary files 2018-11-08 18:35:42 -02:00
Reuben Morais
3cc9b3711d Use ctcdecode in native client 2018-10-25 17:01:08 -03:00
Alexandre Lissy
7d2c0e27f7 Update trie file to v2 2018-10-02 16:21:56 +02:00
Reuben Morais
cb86e7e191 Update language model to a trie-based LM created from the LibriSpeech LM corpus 2018-09-17 11:11:20 -03:00
kdavis-mozilla
a96954b4a9 Quick #1485 fix. For now added @samgd 4-gram LM built from LibriSpeech train-* 2018-08-10 17:17:25 +02:00
kdavis-mozilla
5fb130012a Quick #1485 fix, for now added OpenSLR SLR11 3-gram 2018-08-09 23:47:12 +02:00
Alexandre Lissy
650f7f851d Revert "Fixes #1236 (Switch KenLM to trie based language model)"
This reverts commit e34c52fcb9.
2018-07-20 18:56:57 +02:00