From bfc0de9332604f619e7d9f6dc03b9a4b288102de Mon Sep 17 00:00:00 2001 From: Marcel Klehr Date: Fri, 12 Apr 2024 10:51:33 +0200 Subject: [PATCH 1/5] enh(admin_manual): Add docs for recognize Signed-off-by: Marcel Klehr --- admin_manual/ai/app_recognize.rst | 141 ++++++++++++++++++++++++++++++ admin_manual/ai/index.rst | 1 + 2 files changed, 142 insertions(+) create mode 100644 admin_manual/ai/app_recognize.rst diff --git a/admin_manual/ai/app_recognize.rst b/admin_manual/ai/app_recognize.rst new file mode 100644 index 000000000..7a9771251 --- /dev/null +++ b/admin_manual/ai/app_recognize.rst @@ -0,0 +1,141 @@ +====================================== +App: Recognize +====================================== + +.. _ai-app-recognize: + +The *recognize* app provides media tagging and face recognition functionality for the photos app. *Recognize* can group similar faces on user's photos ("face recognition"); it can add fitting tags to photos detecting landscapes, food, vehicles, buildings animals and other objects, as well as known landmarks and monuments; it can recognize music genres in user's audio files and adds tags for those; it can recognize human actions on user's video files and add tags for them. It specifically runs only open source models and does so entirely on-premises. Nextcloud can provide customer support upon request, please talk to your account manager for the possibilities. + +Front-end +--------- + +Tagged files will appear in the Photos app under the "Tags" section as well as in the normal Files app. Face recognition results will appear under the "People" section in the Photos app. + +Requirements +------------ + +* Nextcloud AIO is not supported but will likely work at sub optimal speed +* Minimal Nextcloud version: 26 +* at least ~4GB of RAM dedicated for recognize +* x86 CPU +* GNU lib C +* Background Jobs must be executed via cron +* Using GPU processing is supported, but not required; be prepared for slow performance unless you are using GPU +* We currently only support NVIDIA GPUs +* For GPU support you need to install + + * NVIDIA® GPU drivers version 450.80.02 or higher. + * CUDA® Toolkit 11.x + * cuDNN SDK 8.x + +* GPU Sizing + + * The models used by recognize should run on most semi modern GPUs + +* CPU Sizing + + * If you don't have a GPU, this app will utilize your CPU cores + * The more cores you have and the more powerful the CPU the better, we recommend 10-20 cores + * In the app settings you can set the number of cores to use + +Space usage +~~~~~~~~~~~ + + * ~1.5GB for all models in total + +Installation +------------ + +1. Install the *recognize* app via the "Apps" page in Nextcloud, or by executing + + occ app:enable recognize + +2. Execute the following command on your server terminal of each node that runs background jobs + + occ recognize:download-models + +3. Go to your Nextcloud Administration settings and open the *recognize* admin settings page +4. Enable all modes of operation that you want the app to undertake +5. Enable GPU mode if you have a GPU that you want to use; if you want to use CPU only, you can set the number of cores to use here +6. Execute the following command on your server terminal to stop background processing of existing files + + occ recognize:clear-background-jobs + +7. Execute the following command on your server terminal to process all existing files in bulk (This may take a long time, depending on how many files you have on your instance): + + occ recognize:classify + +8. Execute the following command on your server terminal to calculate face clusters from faces found in all existing files. Run this repeatedly until no more clusters are found. + + occ recognize:cluster-faces + +9. All new files from this point on will be automatically processed in background tasks without manual intervention. + + +Scaling +------- + +It is possible to scale this app by adding multiple "backgroun" nodes to your cluster that will only process background jobs by executing cron.php. + +App store +--------- + +You can also find the app in our app store, where you can write a review: ``_ + +Repository +---------- + +You can find the app's code repository on GitHub where you can report bugs and contribute fixes and features: ``_ + +Nextcloud customers should file bugs directly with our Support system. + +Known Limitations +----------------- + +* Make sure to test the language model you are using it for whether it meets the use-case's quality requirements +* Machine learning models notoriously have a high energy consumption +* Customer support is available upon request, however we can't solve false or problematic output, most performance issues, or other problems caused by the underlying model. Support is thus limited only to bugs directly caused by the implementation of the app (connectors, API, front-end, AppAPI) + +Ethical AI Rating +----------------- + +Rating for Photo object detection: Green +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Positive: + +* the software for training and inference of this model is open source +* the trained model is freely available, and thus can be run on-premises +* the training data is freely available, making it possible to check or correct for bias or optimise the performance and CO2 usage. + +Rating for Photo face recognition: Green +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Positive: + +* the software for training and inference of this model is open source +* the trained model is freely available, and thus can be run on-premises +* the training data is freely available, making it possible to check or correct for bias or optimise the performance and CO2 usage. + +Rating for Video action recognition: Green +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Positive: + +* the software for training and inferencing of this model is open source +* the trained model is freely available, and thus can be ran on-premises +* the training data is freely available, making it possible to check or correct for bias or optimise the performance and CO2 usage. + +Rating Music genre recognition: Yellow +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Positive: + +* the software for training and inference of this model is open source +* the trained model is freely available, and thus can be run on-premises + +Negative: + +* the training data is not freely available, limiting the ability of external parties to check and correct for bias or optimise the model’s performance and CO2 usage. + +Learn more about the Nextcloud Ethical AI Rating `in our blog `_. diff --git a/admin_manual/ai/index.rst b/admin_manual/ai/index.rst index 865be594c..d86935c8c 100644 --- a/admin_manual/ai/index.rst +++ b/admin_manual/ai/index.rst @@ -10,6 +10,7 @@ Artificial Intelligence app_translate app_llm2 app_stt_whisper2 + app_recognize app_context_chat app_api_and_external_apps ai_as_a_service From 03f78e1102a51bb0c41775d251a67f0fe616aeee Mon Sep 17 00:00:00 2001 From: Marcel Klehr Date: Fri, 12 Apr 2024 11:36:10 +0200 Subject: [PATCH 2/5] Apply suggestions from code review Co-authored-by: Julien Veyssier Signed-off-by: Marcel Klehr --- admin_manual/ai/app_recognize.rst | 46 +++++++++++++++---------------- 1 file changed, 23 insertions(+), 23 deletions(-) diff --git a/admin_manual/ai/app_recognize.rst b/admin_manual/ai/app_recognize.rst index 7a9771251..8746445e6 100644 --- a/admin_manual/ai/app_recognize.rst +++ b/admin_manual/ai/app_recognize.rst @@ -15,14 +15,14 @@ Requirements ------------ * Nextcloud AIO is not supported but will likely work at sub optimal speed -* Minimal Nextcloud version: 26 -* at least ~4GB of RAM dedicated for recognize +* Minimum supported Nextcloud version: 26 +* At least ~4GB of RAM dedicated for recognize * x86 CPU * GNU lib C * Background Jobs must be executed via cron -* Using GPU processing is supported, but not required; be prepared for slow performance unless you are using GPU +* Using GPU processing is supported, but not required; slow performance is expected if you are not using a GPU * We currently only support NVIDIA GPUs -* For GPU support you need to install +* For GPU support you need to install: * NVIDIA® GPU drivers version 450.80.02 or higher. * CUDA® Toolkit 11.x @@ -38,7 +38,7 @@ Requirements * The more cores you have and the more powerful the CPU the better, we recommend 10-20 cores * In the app settings you can set the number of cores to use -Space usage +Disk space usage ~~~~~~~~~~~ * ~1.5GB for all models in total @@ -50,14 +50,14 @@ Installation occ app:enable recognize -2. Execute the following command on your server terminal of each node that runs background jobs +2. Execute the following command on your server terminal of each node that runs background jobs: occ recognize:download-models 3. Go to your Nextcloud Administration settings and open the *recognize* admin settings page 4. Enable all modes of operation that you want the app to undertake 5. Enable GPU mode if you have a GPU that you want to use; if you want to use CPU only, you can set the number of cores to use here -6. Execute the following command on your server terminal to stop background processing of existing files +6. Execute the following command on your server terminal to stop background processing of existing files: occ recognize:clear-background-jobs @@ -65,17 +65,17 @@ Installation occ recognize:classify -8. Execute the following command on your server terminal to calculate face clusters from faces found in all existing files. Run this repeatedly until no more clusters are found. +8. Execute the following command on your server terminal to calculate face clusters from faces found in all existing files (Run this repeatedly until no more clusters are found): occ recognize:cluster-faces -9. All new files from this point on will be automatically processed in background tasks without manual intervention. +9. All new files from this point on will be automatically processed in background tasks without manual intervention Scaling ------- -It is possible to scale this app by adding multiple "backgroun" nodes to your cluster that will only process background jobs by executing cron.php. +It is possible to scale this app by adding multiple "background" nodes to your cluster that will only process background jobs by executing cron.php. App store --------- @@ -85,7 +85,7 @@ You can also find the app in our app store, where you can write a review: ``_ +You can find the app's source repository on GitHub where you can report bugs and contribute fixes and features: ``_ Nextcloud customers should file bugs directly with our Support system. @@ -104,38 +104,38 @@ Rating for Photo object detection: Green Positive: -* the software for training and inference of this model is open source -* the trained model is freely available, and thus can be run on-premises -* the training data is freely available, making it possible to check or correct for bias or optimise the performance and CO2 usage. +* The software for training and inference of this model is open source +* The trained model is freely available, and thus can be run on-premises +* The training data is freely available, making it possible to check or correct for bias or optimize the performance and CO2 usage. Rating for Photo face recognition: Green ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Positive: -* the software for training and inference of this model is open source -* the trained model is freely available, and thus can be run on-premises -* the training data is freely available, making it possible to check or correct for bias or optimise the performance and CO2 usage. +* The software for training and inference of this model is open source +* The trained model is freely available, and thus can be run on-premises +* The training data is freely available, making it possible to check or correct for bias or optimize the performance and CO2 usage. Rating for Video action recognition: Green ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Positive: -* the software for training and inferencing of this model is open source -* the trained model is freely available, and thus can be ran on-premises -* the training data is freely available, making it possible to check or correct for bias or optimise the performance and CO2 usage. +* The software for training and inferencing of this model is open source +* The trained model is freely available, and thus can be ran on-premises +* The training data is freely available, making it possible to check or correct for bias or optimize the performance and CO2 usage. Rating Music genre recognition: Yellow ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Positive: -* the software for training and inference of this model is open source -* the trained model is freely available, and thus can be run on-premises +* The software for training and inference of this model is open source +* The trained model is freely available, and thus can be run on-premises Negative: -* the training data is not freely available, limiting the ability of external parties to check and correct for bias or optimise the model’s performance and CO2 usage. +* The training data is not freely available, limiting the ability of external parties to check and correct for bias or optimise the model’s performance and CO2 usage. Learn more about the Nextcloud Ethical AI Rating `in our blog `_. From b7524e3eba9828311df81df833e0fd8e7fab7f0e Mon Sep 17 00:00:00 2001 From: Marcel Klehr Date: Sun, 14 Apr 2024 10:08:26 +0200 Subject: [PATCH 3/5] fix(admin_manual): Fix docs for recognize Signed-off-by: Marcel Klehr --- admin_manual/ai/app_recognize.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/admin_manual/ai/app_recognize.rst b/admin_manual/ai/app_recognize.rst index 8746445e6..162c072b8 100644 --- a/admin_manual/ai/app_recognize.rst +++ b/admin_manual/ai/app_recognize.rst @@ -92,7 +92,7 @@ Nextcloud customers should file bugs directly with our Support system. Known Limitations ----------------- -* Make sure to test the language model you are using it for whether it meets the use-case's quality requirements +* Make sure to test whether the functionality meets the use-case's quality requirements * Machine learning models notoriously have a high energy consumption * Customer support is available upon request, however we can't solve false or problematic output, most performance issues, or other problems caused by the underlying model. Support is thus limited only to bugs directly caused by the implementation of the app (connectors, API, front-end, AppAPI) From c9f7f272f057b060d0dbeaddab8fb8ba9416473f Mon Sep 17 00:00:00 2001 From: Marcel Klehr Date: Sun, 14 Apr 2024 11:16:14 +0200 Subject: [PATCH 4/5] fix(admin_manual): Fix docs for recognize Signed-off-by: Marcel Klehr --- admin_manual/ai/app_recognize.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/admin_manual/ai/app_recognize.rst b/admin_manual/ai/app_recognize.rst index 162c072b8..7fd25ddfe 100644 --- a/admin_manual/ai/app_recognize.rst +++ b/admin_manual/ai/app_recognize.rst @@ -39,7 +39,7 @@ Requirements * In the app settings you can set the number of cores to use Disk space usage -~~~~~~~~~~~ +~~~~~~~~~~~~~~~~ * ~1.5GB for all models in total From fd0c87cd2ddd2d4612b38c97a72bc471d5597165 Mon Sep 17 00:00:00 2001 From: Marcel Klehr Date: Mon, 15 Apr 2024 10:33:31 +0200 Subject: [PATCH 5/5] fix(admin_manual): Fix docs for recognize Signed-off-by: Marcel Klehr --- admin_manual/ai/app_recognize.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/admin_manual/ai/app_recognize.rst b/admin_manual/ai/app_recognize.rst index 7fd25ddfe..a01f646ce 100644 --- a/admin_manual/ai/app_recognize.rst +++ b/admin_manual/ai/app_recognize.rst @@ -16,7 +16,6 @@ Requirements * Nextcloud AIO is not supported but will likely work at sub optimal speed * Minimum supported Nextcloud version: 26 -* At least ~4GB of RAM dedicated for recognize * x86 CPU * GNU lib C * Background Jobs must be executed via cron @@ -30,13 +29,14 @@ Requirements * GPU Sizing - * The models used by recognize should run on most semi modern GPUs + * The models used by recognize require about 1GB of VRAM or less * CPU Sizing * If you don't have a GPU, this app will utilize your CPU cores * The more cores you have and the more powerful the CPU the better, we recommend 10-20 cores * In the app settings you can set the number of cores to use + * At least ~4GB of RAM dedicated for recognize Disk space usage ~~~~~~~~~~~~~~~~