From b3fc70db7fa4eadb7a3117efcbe852f011b552a1 Mon Sep 17 00:00:00 2001 From: Aya Tanikawa <15815271+ayatnkw@users.noreply.github.com> Date: Fri, 3 May 2024 13:57:43 +0200 Subject: [PATCH 1/9] add explanation to N8N_DEFAULT_BINARY_DATA_MODE --- docs/hosting/configuration/environment-variables/binary-data.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hosting/configuration/environment-variables/binary-data.md b/docs/hosting/configuration/environment-variables/binary-data.md index 65ae7ab64..7a71a138e 100644 --- a/docs/hosting/configuration/environment-variables/binary-data.md +++ b/docs/hosting/configuration/environment-variables/binary-data.md @@ -19,4 +19,4 @@ By default, n8n uses memory to store binary data. Enterprise users can choose to | :------- | :---- | :------- | :---------- | | `N8N_AVAILABLE_BINARY_DATA_MODES` | String | `filesystem` | A comma separated list of available binary data modes. | | `N8N_BINARY_DATA_STORAGE_PATH` | String | `N8N_USER_FOLDER/binaryData` | The path where n8n stores binary data. | -| `N8N_DEFAULT_BINARY_DATA_MODE` | String | `default` | The default binary data mode. `default` keeps binary data in memory. Set to `filesystem` to use the filesystem, or `s3` to AWS S3. | +| `N8N_DEFAULT_BINARY_DATA_MODE` | String | `default` | The default binary data mode. `default` keeps binary data in memory. Set to `filesystem` to use the filesystem, or `s3` to AWS S3. Binary data pruning operates on the active binary data mode. For example, if your instance stored data in S3, and you later switched to filesystem mode, n8n only prunes binary data in the filesystem. | From 6c8b63ed26503770f89576a98d3856e73084d7ff Mon Sep 17 00:00:00 2001 From: Aya Tanikawa <15815271+ayatnkw@users.noreply.github.com> Date: Fri, 3 May 2024 14:29:44 +0200 Subject: [PATCH 2/9] change to a note instead of env var description --- .../configuration/environment-variables/binary-data.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/hosting/configuration/environment-variables/binary-data.md b/docs/hosting/configuration/environment-variables/binary-data.md index 7a71a138e..23ce31d92 100644 --- a/docs/hosting/configuration/environment-variables/binary-data.md +++ b/docs/hosting/configuration/environment-variables/binary-data.md @@ -13,10 +13,14 @@ hide: --8<-- "_snippets/self-hosting/file-based-configuration.md" -By default, n8n uses memory to store binary data. Enterprise users can choose to use an external service instead. Refer to [External storage](/hosting/scaling/external-storage/) for more information on using external storage for binary data. +By default, n8n uses memory to store binary data. Enterprise users can choose to use an external service instead. Refer to [External storage](/hosting/scaling/external-storage/) for more information on using external storage for binary data. + +/// note | Binary data pruning +Binary data pruning operates on the active binary data mode. For example, if your instance stored data in S3, and you later switched to filesystem mode, n8n only prunes binary data in the filesystem. This may change in future. +/// | Variable | Type | Default | Description | | :------- | :---- | :------- | :---------- | | `N8N_AVAILABLE_BINARY_DATA_MODES` | String | `filesystem` | A comma separated list of available binary data modes. | | `N8N_BINARY_DATA_STORAGE_PATH` | String | `N8N_USER_FOLDER/binaryData` | The path where n8n stores binary data. | -| `N8N_DEFAULT_BINARY_DATA_MODE` | String | `default` | The default binary data mode. `default` keeps binary data in memory. Set to `filesystem` to use the filesystem, or `s3` to AWS S3. Binary data pruning operates on the active binary data mode. For example, if your instance stored data in S3, and you later switched to filesystem mode, n8n only prunes binary data in the filesystem. | +| `N8N_DEFAULT_BINARY_DATA_MODE` | String | `default` | The default binary data mode. `default` keeps binary data in memory. Set to `filesystem` to use the filesystem, or `s3` to AWS S3.| \ No newline at end of file From 68347230bc8153a0633f3549c5ce0e2ab40e6838 Mon Sep 17 00:00:00 2001 From: Aya Tanikawa <15815271+ayatnkw@users.noreply.github.com> Date: Fri, 3 May 2024 14:34:52 +0200 Subject: [PATCH 3/9] undo change --- docs/hosting/configuration/environment-variables/binary-data.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hosting/configuration/environment-variables/binary-data.md b/docs/hosting/configuration/environment-variables/binary-data.md index 23ce31d92..c0d5119d8 100644 --- a/docs/hosting/configuration/environment-variables/binary-data.md +++ b/docs/hosting/configuration/environment-variables/binary-data.md @@ -23,4 +23,4 @@ Binary data pruning operates on the active binary data mode. For example, if you | :------- | :---- | :------- | :---------- | | `N8N_AVAILABLE_BINARY_DATA_MODES` | String | `filesystem` | A comma separated list of available binary data modes. | | `N8N_BINARY_DATA_STORAGE_PATH` | String | `N8N_USER_FOLDER/binaryData` | The path where n8n stores binary data. | -| `N8N_DEFAULT_BINARY_DATA_MODE` | String | `default` | The default binary data mode. `default` keeps binary data in memory. Set to `filesystem` to use the filesystem, or `s3` to AWS S3.| \ No newline at end of file +| `N8N_DEFAULT_BINARY_DATA_MODE` | String | `default` | The default binary data mode. `default` keeps binary data in memory. Set to `filesystem` to use the filesystem, or `s3` to AWS S3. | \ No newline at end of file From 33e34217fedcc77338c0520ac62245e21044efeb Mon Sep 17 00:00:00 2001 From: Deborah Date: Tue, 7 May 2024 08:54:35 +0100 Subject: [PATCH 4/9] Update binary-data.md --- .../configuration/environment-variables/binary-data.md | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/docs/hosting/configuration/environment-variables/binary-data.md b/docs/hosting/configuration/environment-variables/binary-data.md index c0d5119d8..f0088e92e 100644 --- a/docs/hosting/configuration/environment-variables/binary-data.md +++ b/docs/hosting/configuration/environment-variables/binary-data.md @@ -15,12 +15,9 @@ hide: By default, n8n uses memory to store binary data. Enterprise users can choose to use an external service instead. Refer to [External storage](/hosting/scaling/external-storage/) for more information on using external storage for binary data. -/// note | Binary data pruning -Binary data pruning operates on the active binary data mode. For example, if your instance stored data in S3, and you later switched to filesystem mode, n8n only prunes binary data in the filesystem. This may change in future. -/// | Variable | Type | Default | Description | | :------- | :---- | :------- | :---------- | | `N8N_AVAILABLE_BINARY_DATA_MODES` | String | `filesystem` | A comma separated list of available binary data modes. | | `N8N_BINARY_DATA_STORAGE_PATH` | String | `N8N_USER_FOLDER/binaryData` | The path where n8n stores binary data. | -| `N8N_DEFAULT_BINARY_DATA_MODE` | String | `default` | The default binary data mode. `default` keeps binary data in memory. Set to `filesystem` to use the filesystem, or `s3` to AWS S3. | \ No newline at end of file +| `N8N_DEFAULT_BINARY_DATA_MODE` | String | `default` | The default binary data mode. `default` keeps binary data in memory. Set to `filesystem` to use the filesystem, or `s3` to AWS S3. Note that binary data pruning operates on the active binary data mode. For example, if your instance stored data in S3, and you later switched to filesystem mode, n8n only prunes binary data in the filesystem. This may change in future. | From 3235f7e21a3ede0799a2f629de6dac3b577e195d Mon Sep 17 00:00:00 2001 From: Aya Tanikawa <15815271+ayatnkw@users.noreply.github.com> Date: Wed, 8 May 2024 09:19:08 +0200 Subject: [PATCH 5/9] change page order in Scaling and performance --- mkdocs.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mkdocs.yml b/mkdocs.yml index 8e0430d48..8075197d6 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -1190,8 +1190,8 @@ nav: - Scaling and performance: - Overview: hosting/scaling/overview.md - Performance and benchmarking: hosting/scaling/performance-benchmarking.md - - Execution data: hosting/scaling/execution-data.md - Configuring queue mode: hosting/scaling/queue-mode.md + - Execution data: hosting/scaling/execution-data.md - Binary data filesystem mode: hosting/scaling/binary-data.md - External storage for binary data: hosting/scaling/external-storage.md - Memory-related errors: hosting/scaling/memory-errors.md From 82c57dfcacbb32dc5d854ac94d658bcc7149f0f8 Mon Sep 17 00:00:00 2001 From: Aya Tanikawa <15815271+ayatnkw@users.noreply.github.com> Date: Wed, 8 May 2024 09:27:20 +0200 Subject: [PATCH 6/9] adhere to vale linter --- docs/hosting/scaling/execution-data.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/hosting/scaling/execution-data.md b/docs/hosting/scaling/execution-data.md index d4b5817e8..312f7810d 100644 --- a/docs/hosting/scaling/execution-data.md +++ b/docs/hosting/scaling/execution-data.md @@ -4,7 +4,7 @@ contentType: howto # Execution data -Depending on your executions settings and volume, your n8n database can quickly grow in size and eventually run out of storage. +Depending on your executions settings and volume, your n8n database can grow in size and run out of storage. To avoid this, n8n recommends that you don't save unnecessary data, and enable pruning of old executions data. @@ -57,7 +57,7 @@ n8n: ## Enable data pruning -You can enable data pruning to automatically delete finished executions after a given time period. If you don't set `EXECUTIONS_DATA_MAX_AGE`, 336 hours (14 days) is the default. +You can enable data pruning to automatically delete finished executions after a given time. If you don't set `EXECUTIONS_DATA_MAX_AGE`, 336 hours (14 days) is the default. You can choose to prune finished executions data before the time set in `EXECUTIONS_DATA_MAX_AGE`, using `EXECUTIONS_DATA_PRUNE_MAX_COUNT`. This sets a maximum number of executions to store in the database. Once you reach the limit, n8n starts to delete the oldest execution records. This can help with database performance issues, especially if you use SQLite. The database size can still exceed the limit you set: old executions that haven't finished running don't get deleted, even if they would otherwise be subject to deletion. From b29a3439b385749a4d649d1ef9dcced939163522 Mon Sep 17 00:00:00 2001 From: Aya Tanikawa <15815271+ayatnkw@users.noreply.github.com> Date: Wed, 8 May 2024 11:37:26 +0200 Subject: [PATCH 7/9] add section on binary data pruning --- docs/hosting/scaling/binary-data.md | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/docs/hosting/scaling/binary-data.md b/docs/hosting/scaling/binary-data.md index feae905bf..faf089932 100644 --- a/docs/hosting/scaling/binary-data.md +++ b/docs/hosting/scaling/binary-data.md @@ -4,10 +4,20 @@ description: How to handle large files without degrading n8n's performance. contentType: howto --- -# Binary data filesystem mode +# Binary data -Binary data is any file-type data, such as image files or documents. +Binary data is any file-type data, such as image files or documents generated or processed during the execution of a workflow. -When handling binary data, n8n keeps the data in memory. This can cause crashes when working with large files. +## Enable filesystem mode + +When handling binary data, n8n keeps the data in memory by default. This can cause crashes when working with large files. To avoid this, change the `N8N_DEFAULT_BINARY_DATA_MODE` [environment variable](/hosting/configuration/environment-variables/binary-data) to `filesystem`. This causes n8n to save data to disk, instead of using memory. + +If you're using queue mode, keep this to `default`. n8n doesn't support filesystem mode with queue mode. Queue mode typically involves a distributed setup across multiple machines, each with its own storage. And a worker won't be able to access binary data from another worker. + +## Binary data pruning + +n8n executes binary data pruning as part of execution data pruning. Refer to [Execution data | Enable data pruning](/hosting/scaling/execution-data/#enable-data-pruning) for details. + +If you configure multiple binary data modes, binary data pruning operates on the active binary data mode. For example, if your instance stored data in S3, and you later switched to filesystem mode, n8n only prunes binary data in the filesystem. Refer to [External storage](/hosting/scaling/external-storage/#usage) for details. From 04edcfd0c39ef5544a62378d5e69365459db3b68 Mon Sep 17 00:00:00 2001 From: Aya Tanikawa <15815271+ayatnkw@users.noreply.github.com> Date: Wed, 8 May 2024 13:45:03 +0200 Subject: [PATCH 8/9] add data pruning details --- _snippets/binary-data-pruning.md | 3 +++ docs/hosting/scaling/binary-data.md | 2 +- docs/hosting/scaling/execution-data.md | 2 ++ docs/hosting/scaling/external-storage.md | 4 +--- mkdocs.yml | 2 +- 5 files changed, 8 insertions(+), 5 deletions(-) create mode 100644 _snippets/binary-data-pruning.md diff --git a/_snippets/binary-data-pruning.md b/_snippets/binary-data-pruning.md new file mode 100644 index 000000000..e539aec3e --- /dev/null +++ b/_snippets/binary-data-pruning.md @@ -0,0 +1,3 @@ +/// note | Binary data pruning +Binary data pruning operates on the active binary data mode. For example, if your instance stored data in S3, and you later switched to filesystem mode, n8n only prunes binary data in the filesystem. This may change in future. +/// \ No newline at end of file diff --git a/docs/hosting/scaling/binary-data.md b/docs/hosting/scaling/binary-data.md index faf089932..7899e21d3 100644 --- a/docs/hosting/scaling/binary-data.md +++ b/docs/hosting/scaling/binary-data.md @@ -14,7 +14,7 @@ When handling binary data, n8n keeps the data in memory by default. This can cau To avoid this, change the `N8N_DEFAULT_BINARY_DATA_MODE` [environment variable](/hosting/configuration/environment-variables/binary-data) to `filesystem`. This causes n8n to save data to disk, instead of using memory. -If you're using queue mode, keep this to `default`. n8n doesn't support filesystem mode with queue mode. Queue mode typically involves a distributed setup across multiple machines, each with its own storage. And a worker won't be able to access binary data from another worker. +If you're using queue mode, keep this to `default`. n8n doesn't support filesystem mode with queue mode. ## Binary data pruning diff --git a/docs/hosting/scaling/execution-data.md b/docs/hosting/scaling/execution-data.md index 312f7810d..7e40d0279 100644 --- a/docs/hosting/scaling/execution-data.md +++ b/docs/hosting/scaling/execution-data.md @@ -95,3 +95,5 @@ n8n: /// note | SQLite If you run n8n using the default SQLite database, the disk space of any pruned data isn't automatically freed up but rather reused for future executions data. To free up this space configure the `DB_SQLITE_VACUUM_ON_STARTUP` [environment variable](/hosting/configuration/environment-variables/database/#sqlite) or manually run the [VACUUM](https://www.sqlite.org/lang_vacuum.html){:target=_blank .external-link} operation. /// + +--8<-- "_snippets/binary-data-pruning.md" \ No newline at end of file diff --git a/docs/hosting/scaling/external-storage.md b/docs/hosting/scaling/external-storage.md index 778086b17..d4f443d47 100644 --- a/docs/hosting/scaling/external-storage.md +++ b/docs/hosting/scaling/external-storage.md @@ -75,6 +75,4 @@ n8n continues to read older binary data stored in the filesystem from the filesy If you store binary data in S3 and later switch to filesystem mode, the instance continues to read any data stored in S3, as long as `s3` remains listed in `N8N_AVAILABLE_BINARY_DATA_MODES` and your S3 credentials remain valid. -/// note | Binary data pruning -Binary data pruning operates on the active binary data mode. For example, if your instance stored data in S3, and you later switched to filesystem mode, n8n only prunes binary data in the filesystem. This may change in future. -/// +--8<-- "_snippets/binary-data-pruning.md" diff --git a/mkdocs.yml b/mkdocs.yml index 8075197d6..b197a4b14 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -1192,7 +1192,7 @@ nav: - Performance and benchmarking: hosting/scaling/performance-benchmarking.md - Configuring queue mode: hosting/scaling/queue-mode.md - Execution data: hosting/scaling/execution-data.md - - Binary data filesystem mode: hosting/scaling/binary-data.md + - Binary data: hosting/scaling/binary-data.md - External storage for binary data: hosting/scaling/external-storage.md - Memory-related errors: hosting/scaling/memory-errors.md - Architecture: From 8d51c268c940bdb92edcdf450b87adcaa8db045b Mon Sep 17 00:00:00 2001 From: Aya Tanikawa <15815271+ayatnkw@users.noreply.github.com> Date: Wed, 8 May 2024 16:44:17 +0200 Subject: [PATCH 9/9] move snippet location --- _snippets/{ => self-hosting/scaling}/binary-data-pruning.md | 0 docs/hosting/scaling/execution-data.md | 2 +- docs/hosting/scaling/external-storage.md | 2 +- 3 files changed, 2 insertions(+), 2 deletions(-) rename _snippets/{ => self-hosting/scaling}/binary-data-pruning.md (100%) diff --git a/_snippets/binary-data-pruning.md b/_snippets/self-hosting/scaling/binary-data-pruning.md similarity index 100% rename from _snippets/binary-data-pruning.md rename to _snippets/self-hosting/scaling/binary-data-pruning.md diff --git a/docs/hosting/scaling/execution-data.md b/docs/hosting/scaling/execution-data.md index 7e40d0279..ac55ba37b 100644 --- a/docs/hosting/scaling/execution-data.md +++ b/docs/hosting/scaling/execution-data.md @@ -96,4 +96,4 @@ n8n: If you run n8n using the default SQLite database, the disk space of any pruned data isn't automatically freed up but rather reused for future executions data. To free up this space configure the `DB_SQLITE_VACUUM_ON_STARTUP` [environment variable](/hosting/configuration/environment-variables/database/#sqlite) or manually run the [VACUUM](https://www.sqlite.org/lang_vacuum.html){:target=_blank .external-link} operation. /// ---8<-- "_snippets/binary-data-pruning.md" \ No newline at end of file +--8<-- "_snippets/self-hosting/scaling/binary-data-pruning.md" \ No newline at end of file diff --git a/docs/hosting/scaling/external-storage.md b/docs/hosting/scaling/external-storage.md index d4f443d47..7c2dc6f29 100644 --- a/docs/hosting/scaling/external-storage.md +++ b/docs/hosting/scaling/external-storage.md @@ -75,4 +75,4 @@ n8n continues to read older binary data stored in the filesystem from the filesy If you store binary data in S3 and later switch to filesystem mode, the instance continues to read any data stored in S3, as long as `s3` remains listed in `N8N_AVAILABLE_BINARY_DATA_MODES` and your S3 credentials remain valid. ---8<-- "_snippets/binary-data-pruning.md" +--8<-- "_snippets/self-hosting/scaling/binary-data-pruning.md"