mirror of
https://github.com/mozilla/DeepSpeech.git
synced 2025-10-26 11:19:39 +00:00
71 lines
3.2 KiB
Markdown
71 lines
3.2 KiB
Markdown
Overview of the process for publishing WER
|
|
==========================================
|
|
|
|
The tracking of WER is made using the following workflow:
|
|
* a dedicated user on the learning machine periodically runs training jobs (cron
|
|
job, or manual runs)
|
|
* this produces, mostly, js/hyper.js containig a concatenated version of all
|
|
previous runs
|
|
* util/website.py contains code that will connect to a SSH server, using SFTP
|
|
* this will publish 'index.html' and its dependencies
|
|
|
|
# Setup of the dedicated user:
|
|
|
|
* Create a standard user
|
|
* Either rely on system's tensorflow or populate a virtualenv
|
|
* Using system tensorflow or a virtualenv might require setting the PYTHONPATH
|
|
env variable (done for system wide tensorflow installation in the example
|
|
below).
|
|
* Install PIP dependencies:
|
|
* jupyter
|
|
* BeautifulSoup4
|
|
* GitPython
|
|
* pysftp
|
|
* xdg
|
|
* requests
|
|
* Construct cron job:
|
|
```
|
|
SHELL=/bin/bash
|
|
PATH=/usr/local/bin:/usr/bin/:/bin
|
|
# Run WER every 15 mins
|
|
*/5 * * * * (mkdir -p $HOME/wer && cd $HOME/wer && source /usr/local/tensorflow-env/bin/activate && /usr/bin/curl -H "Cache-Control: no-cache" -L https://raw.githubusercontent.com/mozilla/DeepSpeech/website/util/automation.py | ds_website_username="u" ds_website_privkey="$HOME/.ssh/k" ds_website_server_fqdn="host.tld" ds_website_server_root="www/" ds_wer_automation="./bin/run-wer-automation.sh" python ; cd) 2>$HOME/.deepspeech_wer.err.log 1>$HOME/.deepspeech_wer.out.log
|
|
```
|
|
* Cron task will take care of:
|
|
* checking if any there were any new merges
|
|
* perform a clone of the git repo and checkout those merges
|
|
* schedule sequential execution against those merges
|
|
* notebook is configured to automatically perform merging and upload if
|
|
the proper environment variables are configured, effectively updating the
|
|
website on each iteration from the above process
|
|
* saving of the hyper.json files produced
|
|
* wiping the cloned git repo
|
|
* A 'lock' file will be created in ~/.cache/deepspeech_wer/ to ensure we do not
|
|
trigger multiple execution at the same time. Unexpected exception might leave
|
|
a stale lock file
|
|
* A 'last_sha1' in the same directory will be used to keep track of what has
|
|
been done last
|
|
* Previous runs' logs will be saved to ~/.local/share/deepspeech_wer/
|
|
* For debugging purpose, `~/.deepspeech_wer.err.log` and `~/.deepspeech_wer.out.log`
|
|
will collect stderr/stdout
|
|
* Expose those environment variable (please refer to util/website.py to have
|
|
more details on each) (cron above does it):
|
|
* ds_website_username
|
|
* ds_website_privkey
|
|
* ds_website_server_fqdn
|
|
* ds_website_server_port
|
|
* ds_website_server_root
|
|
|
|
# Setup of web-facing server:
|
|
|
|
* Ensure existing webroot
|
|
* Generate a SSH key, and upload public key to web-facing server
|
|
* Connect at least one time manually from the training machine to the web-facing
|
|
server to accept the server host key and populate known_hosts file (pay
|
|
attention to the FQDN)
|
|
* Make sure that server is configured with proper DirectoryIndex (Apache, or
|
|
equivalent directive for others), whether system-wide or locally (with a
|
|
.htaccess for example).
|
|
* Bootstrap with empty index.htm (and populate .htaccess if needed)
|
|
* That should be all. Upon any big changes with the HTML codebase, make sure to
|
|
cleanup the mess.
|