diff --git a/_snippets/self-hosting/scaling/binary-data-pruning.md b/_snippets/self-hosting/scaling/binary-data-pruning.md new file mode 100644 index 000000000..e539aec3e --- /dev/null +++ b/_snippets/self-hosting/scaling/binary-data-pruning.md @@ -0,0 +1,3 @@ +/// note | Binary data pruning +Binary data pruning operates on the active binary data mode. For example, if your instance stored data in S3, and you later switched to filesystem mode, n8n only prunes binary data in the filesystem. This may change in future. +/// \ No newline at end of file diff --git a/docs/hosting/configuration/environment-variables/binary-data.md b/docs/hosting/configuration/environment-variables/binary-data.md index 65ae7ab64..f0088e92e 100644 --- a/docs/hosting/configuration/environment-variables/binary-data.md +++ b/docs/hosting/configuration/environment-variables/binary-data.md @@ -13,10 +13,11 @@ hide: --8<-- "_snippets/self-hosting/file-based-configuration.md" -By default, n8n uses memory to store binary data. Enterprise users can choose to use an external service instead. Refer to [External storage](/hosting/scaling/external-storage/) for more information on using external storage for binary data. +By default, n8n uses memory to store binary data. Enterprise users can choose to use an external service instead. Refer to [External storage](/hosting/scaling/external-storage/) for more information on using external storage for binary data. + | Variable | Type | Default | Description | | :------- | :---- | :------- | :---------- | | `N8N_AVAILABLE_BINARY_DATA_MODES` | String | `filesystem` | A comma separated list of available binary data modes. | | `N8N_BINARY_DATA_STORAGE_PATH` | String | `N8N_USER_FOLDER/binaryData` | The path where n8n stores binary data. | -| `N8N_DEFAULT_BINARY_DATA_MODE` | String | `default` | The default binary data mode. `default` keeps binary data in memory. Set to `filesystem` to use the filesystem, or `s3` to AWS S3. | +| `N8N_DEFAULT_BINARY_DATA_MODE` | String | `default` | The default binary data mode. `default` keeps binary data in memory. Set to `filesystem` to use the filesystem, or `s3` to AWS S3. Note that binary data pruning operates on the active binary data mode. For example, if your instance stored data in S3, and you later switched to filesystem mode, n8n only prunes binary data in the filesystem. This may change in future. | diff --git a/docs/hosting/scaling/binary-data.md b/docs/hosting/scaling/binary-data.md index feae905bf..7899e21d3 100644 --- a/docs/hosting/scaling/binary-data.md +++ b/docs/hosting/scaling/binary-data.md @@ -4,10 +4,20 @@ description: How to handle large files without degrading n8n's performance. contentType: howto --- -# Binary data filesystem mode +# Binary data -Binary data is any file-type data, such as image files or documents. +Binary data is any file-type data, such as image files or documents generated or processed during the execution of a workflow. -When handling binary data, n8n keeps the data in memory. This can cause crashes when working with large files. +## Enable filesystem mode + +When handling binary data, n8n keeps the data in memory by default. This can cause crashes when working with large files. To avoid this, change the `N8N_DEFAULT_BINARY_DATA_MODE` [environment variable](/hosting/configuration/environment-variables/binary-data) to `filesystem`. This causes n8n to save data to disk, instead of using memory. + +If you're using queue mode, keep this to `default`. n8n doesn't support filesystem mode with queue mode. + +## Binary data pruning + +n8n executes binary data pruning as part of execution data pruning. Refer to [Execution data | Enable data pruning](/hosting/scaling/execution-data/#enable-data-pruning) for details. + +If you configure multiple binary data modes, binary data pruning operates on the active binary data mode. For example, if your instance stored data in S3, and you later switched to filesystem mode, n8n only prunes binary data in the filesystem. Refer to [External storage](/hosting/scaling/external-storage/#usage) for details. diff --git a/docs/hosting/scaling/execution-data.md b/docs/hosting/scaling/execution-data.md index d4b5817e8..ac55ba37b 100644 --- a/docs/hosting/scaling/execution-data.md +++ b/docs/hosting/scaling/execution-data.md @@ -4,7 +4,7 @@ contentType: howto # Execution data -Depending on your executions settings and volume, your n8n database can quickly grow in size and eventually run out of storage. +Depending on your executions settings and volume, your n8n database can grow in size and run out of storage. To avoid this, n8n recommends that you don't save unnecessary data, and enable pruning of old executions data. @@ -57,7 +57,7 @@ n8n: ## Enable data pruning -You can enable data pruning to automatically delete finished executions after a given time period. If you don't set `EXECUTIONS_DATA_MAX_AGE`, 336 hours (14 days) is the default. +You can enable data pruning to automatically delete finished executions after a given time. If you don't set `EXECUTIONS_DATA_MAX_AGE`, 336 hours (14 days) is the default. You can choose to prune finished executions data before the time set in `EXECUTIONS_DATA_MAX_AGE`, using `EXECUTIONS_DATA_PRUNE_MAX_COUNT`. This sets a maximum number of executions to store in the database. Once you reach the limit, n8n starts to delete the oldest execution records. This can help with database performance issues, especially if you use SQLite. The database size can still exceed the limit you set: old executions that haven't finished running don't get deleted, even if they would otherwise be subject to deletion. @@ -95,3 +95,5 @@ n8n: /// note | SQLite If you run n8n using the default SQLite database, the disk space of any pruned data isn't automatically freed up but rather reused for future executions data. To free up this space configure the `DB_SQLITE_VACUUM_ON_STARTUP` [environment variable](/hosting/configuration/environment-variables/database/#sqlite) or manually run the [VACUUM](https://www.sqlite.org/lang_vacuum.html){:target=_blank .external-link} operation. /// + +--8<-- "_snippets/self-hosting/scaling/binary-data-pruning.md" \ No newline at end of file diff --git a/docs/hosting/scaling/external-storage.md b/docs/hosting/scaling/external-storage.md index 778086b17..7c2dc6f29 100644 --- a/docs/hosting/scaling/external-storage.md +++ b/docs/hosting/scaling/external-storage.md @@ -75,6 +75,4 @@ n8n continues to read older binary data stored in the filesystem from the filesy If you store binary data in S3 and later switch to filesystem mode, the instance continues to read any data stored in S3, as long as `s3` remains listed in `N8N_AVAILABLE_BINARY_DATA_MODES` and your S3 credentials remain valid. -/// note | Binary data pruning -Binary data pruning operates on the active binary data mode. For example, if your instance stored data in S3, and you later switched to filesystem mode, n8n only prunes binary data in the filesystem. This may change in future. -/// +--8<-- "_snippets/self-hosting/scaling/binary-data-pruning.md" diff --git a/mkdocs.yml b/mkdocs.yml index 9656e4d4b..d27525512 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -1190,9 +1190,9 @@ nav: - Scaling and performance: - Overview: hosting/scaling/overview.md - Performance and benchmarking: hosting/scaling/performance-benchmarking.md - - Execution data: hosting/scaling/execution-data.md - Configuring queue mode: hosting/scaling/queue-mode.md - - Binary data filesystem mode: hosting/scaling/binary-data.md + - Execution data: hosting/scaling/execution-data.md + - Binary data: hosting/scaling/binary-data.md - External storage for binary data: hosting/scaling/external-storage.md - Memory-related errors: hosting/scaling/memory-errors.md - Architecture: