Reuben Morais
30de5153bc
Fix regressions in bytes output mode
2020-07-04 11:27:33 +02:00
Reuben Morais
d548222518
Return raw scores in confidence value
2020-05-27 16:48:57 +02:00
Reuben Morais
33760a6bcd
Delay beam expansion until a non-blank label has probability >0.1%
2020-04-17 14:33:40 +02:00
Reuben Morais
69bd032605
Improve API naming around Metadata objects
2020-03-17 14:47:58 +01:00
dabinat
e0c42f01a4
Moved result limiting to ModelState instead of CTC decoder
2020-03-17 14:47:58 +01:00
dabinat
32c969c184
Expose multiple transcriptions through the API
2020-03-17 14:47:58 +01:00
Reuben Morais
8dedda7759
Address review comments
2020-02-11 19:44:36 +01:00
Reuben Morais
1d3b3a31a1
Address review comments and update docs
2020-02-11 19:44:36 +01:00
Reuben Morais
be2229ef29
Refactor Scorer so model/trie package can be created by an external tool
2020-02-11 19:44:26 +01:00
Alexandre Lissy
f44d2ddeb9
Don't OOV_SCORE on empty prefix
...
Fixes #2579
2020-01-02 12:39:12 +01:00
Reuben Morais
d2eb305b73
Address review comment and add missing check for presence of scorer
2019-11-12 21:56:42 +01:00
Reuben Morais
0e6952c3a8
Avoid reconstructing strings twice on decode
2019-11-11 12:53:05 +01:00
Reuben Morais
c1b1a59423
Score prefix as soon as a grapheme is formed rather than 1 byte later
2019-11-11 12:52:48 +01:00
Reuben Morais
f4cdd988df
UTF-8 target
2019-11-11 11:36:16 +01:00
Reuben Morais
44a605c8b7
Merge pull request #2435 from mozilla/uplift-utf8-fixes
...
Uplift general fixes from UTF-8 work
2019-10-25 09:09:48 +00:00
Alexandre Lissy
ef3f8004ce
Use std::shared_ptr instead of raw pointer for dictionary_
...
Fixes #2403
2019-10-18 10:15:59 +02:00
Reuben Morais
abc687b3b9
Specify BOS for final scoring at decode if applicable
2019-10-14 21:03:14 +02:00
Reuben Morais
3015237e8d
Replace incomplete sorts with partial sorts
2019-10-14 21:03:14 +02:00
Reuben Morais
6dba6d4a95
Don't explicitly score the BOS token, and avoid copies when scoring sentences
2019-09-26 14:08:32 +02:00
Reuben Morais
4d882a8aec
Simplify decoder impl by making it object oriented, avoid pointers where possible
2019-08-23 12:19:32 +02:00
Reuben Morais
e3bf5d3cc6
Only update time step of leaf prefixes
...
The intention of this check is to improve the accuracy of the timings by recording the time step where the character saw its highest probability rather than the first time step where it was seen. The problem happens when updating the time step of a prefix that already has children. In that case, if any of the children have a time step that is earlier than `new_timestep`, it'll break the linearity of the timings. My fix is to simply check that the prefix we're updating is a leaf.
For example, say during decoding we have the following beams (format is `(char | time)`, tree node id below, nodes with same id are the same object):
```
1. (-1 | 0 ) -> ('s' | 10) -> ('h' | 13) -> ('e' | 14)
A B C D
2. (-1 | 0 ) -> ('s' | 10) -> ('h' | 14)
A B E
```
And the prefix list is [B, C, D, E]. Currently, if we process character 'h' in time step 15 with a probability higher than both C and E, we update both nodes to have time step 15, which breaks linearity in beam 1. With my fix, we only update node E, which is a leaf. In my tests this does fix the problem, but since we don't have any known good quality data to verify against, it's hard to know if it has other side effects.
2019-08-20 12:03:59 +02:00
Alexandre Lissy
c3a2b6ea48
Ensure proper removal of prefix
...
Fixes #2230
2019-07-10 13:27:18 +02:00
Reuben Morais
31afc6811f
Switch to ConstFst from VectorFst and mmap trie file when reading
2019-06-21 22:21:19 -03:00
Reuben Morais
95a54bfdb8
Modify copies rather than live prefix scores in decoder_decode
2019-06-19 19:39:57 -03:00
Reuben Morais
d5d55958da
Clean up formatting in ctc_beam_saerch_decoder.cpp
2019-06-19 19:39:57 -03:00
Reuben Morais
3ceb05e8b9
Keep absolute per-stream time step in DecoderState
2019-06-11 07:58:16 -03:00
Reuben Morais
a46288e1c8
Only create Output structure for beams that are returned in the API
2019-06-01 15:02:41 -03:00
dabinat
d9a269412e
CTC beam search streaming decoder (+6 squashed commits)
...
Squashed commits:
[2941b47] Fixed nits
[700572e] Restored old CTC decoder API
[5aaf75d] Fixed nits
[969d71a] Added a destructor for DecoderState
[af0be6e] Removed accumulated_logits
[9dcb7b4] CTC beam search streaming decoder
2019-05-21 20:54:19 -07:00
Reuben Morais
5bf6e63f1b
Record character timing offset at peak probability of character
...
Uplifted from 173dddbe32
2019-02-17 22:41:18 -03:00
Reuben Morais
a7144e1d38
Address review comments
2018-11-07 23:16:13 -02:00
Reuben Morais
440893c58d
Simplify ctcdecode API signatures to avoid nested STL structures
...
Flatten structure to avoid nested STL structures which are awkward to
wrap with SWIG and slower at runtime.
2018-11-03 07:48:02 -03:00
Reuben Morais
23497fde41
Use unique_ptr for Scorer members
2018-10-30 09:21:30 -03:00
Reuben Morais
770d742cd5
Adjust ctcdecode to use our Alphabet code, support using trie files
2018-10-25 17:01:05 -03:00
Reuben Morais
c34fc5b7ac
Import parlance/ctcdecode into repository
2018-10-25 17:00:48 -03:00
