Merge pull request #2083 from n8n-io/DOC-831-clarify-data-pruning-explanation

Doc 831 Clarify explanation around data pruning and env vars related to external storage
This commit is contained in:
Deborah 2024-05-08 15:47:16 +01:00 committed by GitHub
commit d034686ff1
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
6 changed files with 26 additions and 12 deletions

View File

@ -0,0 +1,3 @@
/// note | Binary data pruning
Binary data pruning operates on the active binary data mode. For example, if your instance stored data in S3, and you later switched to filesystem mode, n8n only prunes binary data in the filesystem. This may change in future.
///

View File

@ -13,10 +13,11 @@ hide:
--8<-- "_snippets/self-hosting/file-based-configuration.md"
By default, n8n uses memory to store binary data. Enterprise users can choose to use an external service instead. Refer to [External storage](/hosting/scaling/external-storage/) for more information on using external storage for binary data.
By default, n8n uses memory to store binary data. Enterprise users can choose to use an external service instead. Refer to [External storage](/hosting/scaling/external-storage/) for more information on using external storage for binary data.
| Variable | Type | Default | Description |
| :------- | :---- | :------- | :---------- |
| `N8N_AVAILABLE_BINARY_DATA_MODES` | String | `filesystem` | A comma separated list of available binary data modes. |
| `N8N_BINARY_DATA_STORAGE_PATH` | String | `N8N_USER_FOLDER/binaryData` | The path where n8n stores binary data. |
| `N8N_DEFAULT_BINARY_DATA_MODE` | String | `default` | The default binary data mode. `default` keeps binary data in memory. Set to `filesystem` to use the filesystem, or `s3` to AWS S3. |
| `N8N_DEFAULT_BINARY_DATA_MODE` | String | `default` | The default binary data mode. `default` keeps binary data in memory. Set to `filesystem` to use the filesystem, or `s3` to AWS S3. Note that binary data pruning operates on the active binary data mode. For example, if your instance stored data in S3, and you later switched to filesystem mode, n8n only prunes binary data in the filesystem. This may change in future. |

View File

@ -4,10 +4,20 @@ description: How to handle large files without degrading n8n's performance.
contentType: howto
---
# Binary data filesystem mode
# Binary data
Binary data is any file-type data, such as image files or documents.
Binary data is any file-type data, such as image files or documents generated or processed during the execution of a workflow.
When handling binary data, n8n keeps the data in memory. This can cause crashes when working with large files.
## Enable filesystem mode
When handling binary data, n8n keeps the data in memory by default. This can cause crashes when working with large files.
To avoid this, change the `N8N_DEFAULT_BINARY_DATA_MODE` [environment variable](/hosting/configuration/environment-variables/binary-data) to `filesystem`. This causes n8n to save data to disk, instead of using memory.
If you're using queue mode, keep this to `default`. n8n doesn't support filesystem mode with queue mode.
## Binary data pruning
n8n executes binary data pruning as part of execution data pruning. Refer to [Execution data | Enable data pruning](/hosting/scaling/execution-data/#enable-data-pruning) for details.
If you configure multiple binary data modes, binary data pruning operates on the active binary data mode. For example, if your instance stored data in S3, and you later switched to filesystem mode, n8n only prunes binary data in the filesystem. Refer to [External storage](/hosting/scaling/external-storage/#usage) for details.

View File

@ -4,7 +4,7 @@ contentType: howto
# Execution data
Depending on your executions settings and volume, your n8n database can quickly grow in size and eventually run out of storage.
Depending on your executions settings and volume, your n8n database can grow in size and run out of storage.
To avoid this, n8n recommends that you don't save unnecessary data, and enable pruning of old executions data.
@ -57,7 +57,7 @@ n8n:
## Enable data pruning
You can enable data pruning to automatically delete finished executions after a given time period. If you don't set `EXECUTIONS_DATA_MAX_AGE`, 336 hours (14 days) is the default.
You can enable data pruning to automatically delete finished executions after a given time. If you don't set `EXECUTIONS_DATA_MAX_AGE`, 336 hours (14 days) is the default.
You can choose to prune finished executions data before the time set in `EXECUTIONS_DATA_MAX_AGE`, using `EXECUTIONS_DATA_PRUNE_MAX_COUNT`. This sets a maximum number of executions to store in the database. Once you reach the limit, n8n starts to delete the oldest execution records. This can help with database performance issues, especially if you use SQLite. The database size can still exceed the limit you set: old executions that haven't finished running don't get deleted, even if they would otherwise be subject to deletion.
@ -95,3 +95,5 @@ n8n:
/// note | SQLite
If you run n8n using the default SQLite database, the disk space of any pruned data isn't automatically freed up but rather reused for future executions data. To free up this space configure the `DB_SQLITE_VACUUM_ON_STARTUP` [environment variable](/hosting/configuration/environment-variables/database/#sqlite) or manually run the [VACUUM](https://www.sqlite.org/lang_vacuum.html){:target=_blank .external-link} operation.
///
--8<-- "_snippets/self-hosting/scaling/binary-data-pruning.md"

View File

@ -75,6 +75,4 @@ n8n continues to read older binary data stored in the filesystem from the filesy
If you store binary data in S3 and later switch to filesystem mode, the instance continues to read any data stored in S3, as long as `s3` remains listed in `N8N_AVAILABLE_BINARY_DATA_MODES` and your S3 credentials remain valid.
/// note | Binary data pruning
Binary data pruning operates on the active binary data mode. For example, if your instance stored data in S3, and you later switched to filesystem mode, n8n only prunes binary data in the filesystem. This may change in future.
///
--8<-- "_snippets/self-hosting/scaling/binary-data-pruning.md"

View File

@ -1190,9 +1190,9 @@ nav:
- Scaling and performance:
- Overview: hosting/scaling/overview.md
- Performance and benchmarking: hosting/scaling/performance-benchmarking.md
- Execution data: hosting/scaling/execution-data.md
- Configuring queue mode: hosting/scaling/queue-mode.md
- Binary data filesystem mode: hosting/scaling/binary-data.md
- Execution data: hosting/scaling/execution-data.md
- Binary data: hosting/scaling/binary-data.md
- External storage for binary data: hosting/scaling/external-storage.md
- Memory-related errors: hosting/scaling/memory-errors.md
- Architecture: