From 1d6a7331081d70534ebdf99ffdff44c0b4b98b28 Mon Sep 17 00:00:00 2001 From: Deborah Barnard Date: Thu, 28 Sep 2023 11:13:15 +0100 Subject: [PATCH 01/16] add section --- docs/hosting/external-storage/index.md | 16 ++++++++++++++++ mkdocs.yml | 1 + 2 files changed, 17 insertions(+) create mode 100644 docs/hosting/external-storage/index.md diff --git a/docs/hosting/external-storage/index.md b/docs/hosting/external-storage/index.md new file mode 100644 index 000000000..7226fe628 --- /dev/null +++ b/docs/hosting/external-storage/index.md @@ -0,0 +1,16 @@ +--- +description: [TODO: add short description] +contentType: overview +--- + +# External storage + +!!! info "Feature availability" + * Available on Self-hosted Enterprise plans + * If you want access to this feature on Cloud Enterprise, contact us + + + +[[% import "_macros/section-toc.html" as sectionToc %]] + +[[ sectionToc.sectionToc(page) ]] diff --git a/mkdocs.yml b/mkdocs.yml index 7d63f797e..2b4b9fbe9 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -557,6 +557,7 @@ nav: - Isolating n8n: hosting/environment-variables/configuration-examples/isolation.md - Supported databases and settings: hosting/supported-databases-settings.md - User management: hosting/user-management-self-hosted.md + - External storage: hosting/external-storage/index.md - Security audit: hosting/security-audit.md - Logging and monitoring: - hosting/logging-monitoring/index.md From 684faa59aa76b94b241a5d086facc1c7bff52731 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Iv=C3=A1n=20Ovejero?= Date: Thu, 28 Sep 2023 12:43:00 +0200 Subject: [PATCH 02/16] Initial draft --- docs/hosting/external-storage/index.md | 4 +- docs/hosting/external-storage/s3.md | 60 ++++++++++++++++++++++++++ 2 files changed, 62 insertions(+), 2 deletions(-) create mode 100644 docs/hosting/external-storage/s3.md diff --git a/docs/hosting/external-storage/index.md b/docs/hosting/external-storage/index.md index 7226fe628..b9c588c2e 100644 --- a/docs/hosting/external-storage/index.md +++ b/docs/hosting/external-storage/index.md @@ -1,5 +1,5 @@ --- -description: [TODO: add short description] +description: External storage for your n8n instance. contentType: overview --- @@ -9,7 +9,7 @@ contentType: overview * Available on Self-hosted Enterprise plans * If you want access to this feature on Cloud Enterprise, contact us - +n8n can externally store binary data produced by workflow executions. This feature is useful to avoid relying on the filesystem for storing large amounts of binary data. In future, n8n will support external storage for other kinds of data in addition to binary data. [[% import "_macros/section-toc.html" as sectionToc %]] diff --git a/docs/hosting/external-storage/s3.md b/docs/hosting/external-storage/s3.md new file mode 100644 index 000000000..86d17741a --- /dev/null +++ b/docs/hosting/external-storage/s3.md @@ -0,0 +1,60 @@ +--- +contentType: howto +--- + +# Storing n8n's binary data in S3 + +n8n can use [AWS S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) as an external store for binary data produced by workflow executions. Other S3-compatible services like Cloudflare R2 and Backblaze B2 may work, but are not officially supported. + +!!! note "Enterprise-tier feature" + You will need an [Enterprise license key](/enterprise-key/) for external storage. If your license key has expired and you remain on S3 mode, the instance will be able to read from, but not write to, the S3 bucket. + +## Usage + +Binary data will be written to your n8n S3 bucket in this format: + +``` +workflows/{workflowId}/executions/{executionId}/binary_data/{binaryFileId} +``` + +To inspect binary data in the n8n S3 bucket, you can use the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) with your access key ID and secret access key. You can also access the S3 section in the AWS Management Console with the details from step 7. You can query the AWS S3 API using n8n's AWS S3 node. + +If your instance stored data in S3 and was later switched to filesystem mode, the instance will continue to read any data that was stored in S3, as long as `s3` remains listed in `N8N_AVAILABLE_BINARY_DATA_MODES` and as long as your S3 credentials remain valid. + +At this time, binary data pruning is based on the active binary data mode. For example, if your instance stored data in S3 and was later switched to filesystem mode, only binary data in the filesystem will be pruned. This may change in future. + + +## Setup + +First, create and configure a bucket following the [official AWS documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html). You can use the following policy, replacing `` with the name of the bucket you created. + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "VisualEditor0", + "Effect": "Allow", + "Action": ["s3:*"], + "Resource": ["arn:aws:s3:::", "arn:aws:s3:::/*"] + } + ] +} +``` + +Once you finish creating the bucket, you will have a host, bucket name and region, and an access key ID and secret access key. You will need to set all of these in n8n's environment. + +```sh +export N8N_EXTERNAL_STORAGE_S3_HOST=... +export N8N_EXTERNAL_STORAGE_S3_BUCKET_NAME=... +export N8N_EXTERNAL_STORAGE_S3_BUCKET_REGION=... +export N8N_EXTERNAL_STORAGE_S3_ACCOUNT_ID=... +export N8N_EXTERNAL_STORAGE_S3_SECRET_KEY=... +``` + +Finally, instruct n8n to store binary data in S3 and start the server. + +```sh +export N8N_AVAILABLE_BINARY_DATA_MODES=filesystem,s3 +export N8N_DEFAULT_BINARY_DATA_MODE=s3 +``` From e8e924246645741e8d5a22fd85dc6552b73df58a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Iv=C3=A1n=20Ovejero?= Date: Thu, 28 Sep 2023 12:46:27 +0200 Subject: [PATCH 03/16] Cleanup --- docs/hosting/external-storage/s3.md | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/docs/hosting/external-storage/s3.md b/docs/hosting/external-storage/s3.md index 86d17741a..298a2512a 100644 --- a/docs/hosting/external-storage/s3.md +++ b/docs/hosting/external-storage/s3.md @@ -17,13 +17,10 @@ Binary data will be written to your n8n S3 bucket in this format: workflows/{workflowId}/executions/{executionId}/binary_data/{binaryFileId} ``` -To inspect binary data in the n8n S3 bucket, you can use the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) with your access key ID and secret access key. You can also access the S3 section in the AWS Management Console with the details from step 7. You can query the AWS S3 API using n8n's AWS S3 node. - If your instance stored data in S3 and was later switched to filesystem mode, the instance will continue to read any data that was stored in S3, as long as `s3` remains listed in `N8N_AVAILABLE_BINARY_DATA_MODES` and as long as your S3 credentials remain valid. At this time, binary data pruning is based on the active binary data mode. For example, if your instance stored data in S3 and was later switched to filesystem mode, only binary data in the filesystem will be pruned. This may change in future. - ## Setup First, create and configure a bucket following the [official AWS documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html). You can use the following policy, replacing `` with the name of the bucket you created. @@ -42,7 +39,7 @@ First, create and configure a bucket following the [official AWS documentation]( } ``` -Once you finish creating the bucket, you will have a host, bucket name and region, and an access key ID and secret access key. You will need to set all of these in n8n's environment. +Once you finish creating the bucket, you will have a host, bucket name and region, and an access key ID and secret access key. You will need to set them in n8n's environment. ```sh export N8N_EXTERNAL_STORAGE_S3_HOST=... From 1f675b35b3430fd80154ec6a76dc0d71298893d5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Iv=C3=A1n=20Ovejero?= Date: Thu, 28 Sep 2023 12:49:23 +0200 Subject: [PATCH 04/16] `info` instead of `note` --- docs/hosting/external-storage/s3.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hosting/external-storage/s3.md b/docs/hosting/external-storage/s3.md index 298a2512a..07a0a74d7 100644 --- a/docs/hosting/external-storage/s3.md +++ b/docs/hosting/external-storage/s3.md @@ -6,7 +6,7 @@ contentType: howto n8n can use [AWS S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) as an external store for binary data produced by workflow executions. Other S3-compatible services like Cloudflare R2 and Backblaze B2 may work, but are not officially supported. -!!! note "Enterprise-tier feature" +!!! info "Enterprise-tier feature" You will need an [Enterprise license key](/enterprise-key/) for external storage. If your license key has expired and you remain on S3 mode, the instance will be able to read from, but not write to, the S3 bucket. ## Usage From edd6d92cd520e703656d225e4fe4387fb3ee24f5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Iv=C3=A1n=20Ovejero?= Date: Thu, 28 Sep 2023 12:59:30 +0200 Subject: [PATCH 05/16] Restructure a bit --- docs/hosting/external-storage/s3.md | 29 ++++++++++++++++------------- 1 file changed, 16 insertions(+), 13 deletions(-) diff --git a/docs/hosting/external-storage/s3.md b/docs/hosting/external-storage/s3.md index 07a0a74d7..69297143f 100644 --- a/docs/hosting/external-storage/s3.md +++ b/docs/hosting/external-storage/s3.md @@ -7,19 +7,7 @@ contentType: howto n8n can use [AWS S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) as an external store for binary data produced by workflow executions. Other S3-compatible services like Cloudflare R2 and Backblaze B2 may work, but are not officially supported. !!! info "Enterprise-tier feature" - You will need an [Enterprise license key](/enterprise-key/) for external storage. If your license key has expired and you remain on S3 mode, the instance will be able to read from, but not write to, the S3 bucket. - -## Usage - -Binary data will be written to your n8n S3 bucket in this format: - -``` -workflows/{workflowId}/executions/{executionId}/binary_data/{binaryFileId} -``` - -If your instance stored data in S3 and was later switched to filesystem mode, the instance will continue to read any data that was stored in S3, as long as `s3` remains listed in `N8N_AVAILABLE_BINARY_DATA_MODES` and as long as your S3 credentials remain valid. - -At this time, binary data pruning is based on the active binary data mode. For example, if your instance stored data in S3 and was later switched to filesystem mode, only binary data in the filesystem will be pruned. This may change in future. + You will need an [Enterprise license key](/enterprise-key/) for external storage. If your license key expires and you remain on S3 mode, the instance will be able to read from, but not write to, the S3 bucket. ## Setup @@ -55,3 +43,18 @@ Finally, instruct n8n to store binary data in S3 and start the server. export N8N_AVAILABLE_BINARY_DATA_MODES=filesystem,s3 export N8N_DEFAULT_BINARY_DATA_MODE=s3 ``` + +## Usage + +Once S3 is enabled, any new binary data will be written to, and read from, the S3 bucket. Binary data will be written to your S3 bucket in this format: + +``` +workflows/{workflowId}/executions/{executionId}/binary_data/{binaryFileId} +``` + +Older binary data that was stored in the filesystem will continue to be read from the filesystem, if `filesystem` remains listed in `N8N_AVAILABLE_BINARY_DATA_MODES`. + +If you store binary data in S3 and was later switch to filesystem mode, the instance will continue to read any data that was stored in S3, if `s3` remains listed in `N8N_AVAILABLE_BINARY_DATA_MODES` and if your S3 credentials remain valid. + +!!! note "Binary data pruning" + At this time, binary data pruning is based on the active binary data mode. For example, if your instance stored data in S3 and was later switched to filesystem mode, only binary data in the filesystem will be pruned. This may change in future. From eff2dcd6f1818e736be30e081516b20993743e33 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Iv=C3=A1n=20Ovejero?= Date: Thu, 28 Sep 2023 13:03:35 +0200 Subject: [PATCH 06/16] Rephrase --- docs/hosting/external-storage/s3.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hosting/external-storage/s3.md b/docs/hosting/external-storage/s3.md index 69297143f..f248a6459 100644 --- a/docs/hosting/external-storage/s3.md +++ b/docs/hosting/external-storage/s3.md @@ -54,7 +54,7 @@ workflows/{workflowId}/executions/{executionId}/binary_data/{binaryFileId} Older binary data that was stored in the filesystem will continue to be read from the filesystem, if `filesystem` remains listed in `N8N_AVAILABLE_BINARY_DATA_MODES`. -If you store binary data in S3 and was later switch to filesystem mode, the instance will continue to read any data that was stored in S3, if `s3` remains listed in `N8N_AVAILABLE_BINARY_DATA_MODES` and if your S3 credentials remain valid. +If you store binary data in S3 and later switch to filesystem mode, the instance will continue to read any data that was stored in S3, as long as `s3` remains listed in `N8N_AVAILABLE_BINARY_DATA_MODES` and your S3 credentials remain valid. !!! note "Binary data pruning" At this time, binary data pruning is based on the active binary data mode. For example, if your instance stored data in S3 and was later switched to filesystem mode, only binary data in the filesystem will be pruned. This may change in future. From 148836ae126d32c78ce22b4778c2bd30e47231fc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Iv=C3=A1n=20Ovejero?= Date: Thu, 28 Sep 2023 14:20:22 +0200 Subject: [PATCH 07/16] Add example --- docs/hosting/external-storage/s3.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/hosting/external-storage/s3.md b/docs/hosting/external-storage/s3.md index f248a6459..7b4f8a379 100644 --- a/docs/hosting/external-storage/s3.md +++ b/docs/hosting/external-storage/s3.md @@ -30,9 +30,9 @@ First, create and configure a bucket following the [official AWS documentation]( Once you finish creating the bucket, you will have a host, bucket name and region, and an access key ID and secret access key. You will need to set them in n8n's environment. ```sh -export N8N_EXTERNAL_STORAGE_S3_HOST=... -export N8N_EXTERNAL_STORAGE_S3_BUCKET_NAME=... -export N8N_EXTERNAL_STORAGE_S3_BUCKET_REGION=... +export N8N_EXTERNAL_STORAGE_S3_HOST=... # example: s3.us-east-1.amazonaws.com +export N8N_EXTERNAL_STORAGE_S3_BUCKET_NAME=... +export N8N_EXTERNAL_STORAGE_S3_BUCKET_REGION=... export N8N_EXTERNAL_STORAGE_S3_ACCOUNT_ID=... export N8N_EXTERNAL_STORAGE_S3_SECRET_KEY=... ``` From 81e0811bdfa0421b8ac461bae10aa69f56a8ec2f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Iv=C3=A1n=20Ovejero?= Date: Thu, 28 Sep 2023 14:21:06 +0200 Subject: [PATCH 08/16] Remove whitespace --- docs/hosting/external-storage/s3.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/hosting/external-storage/s3.md b/docs/hosting/external-storage/s3.md index 7b4f8a379..eb0d86efd 100644 --- a/docs/hosting/external-storage/s3.md +++ b/docs/hosting/external-storage/s3.md @@ -31,8 +31,8 @@ Once you finish creating the bucket, you will have a host, bucket name and regio ```sh export N8N_EXTERNAL_STORAGE_S3_HOST=... # example: s3.us-east-1.amazonaws.com -export N8N_EXTERNAL_STORAGE_S3_BUCKET_NAME=... -export N8N_EXTERNAL_STORAGE_S3_BUCKET_REGION=... +export N8N_EXTERNAL_STORAGE_S3_BUCKET_NAME=... +export N8N_EXTERNAL_STORAGE_S3_BUCKET_REGION=... export N8N_EXTERNAL_STORAGE_S3_ACCOUNT_ID=... export N8N_EXTERNAL_STORAGE_S3_SECRET_KEY=... ``` From 8b7ac75e0c90103ba2f90ceaa3acbd3fe05497a4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Iv=C3=A1n=20Ovejero?= Date: Thu, 28 Sep 2023 17:23:50 +0200 Subject: [PATCH 09/16] Add regionless note --- docs/hosting/external-storage/s3.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/hosting/external-storage/s3.md b/docs/hosting/external-storage/s3.md index eb0d86efd..f18957afd 100644 --- a/docs/hosting/external-storage/s3.md +++ b/docs/hosting/external-storage/s3.md @@ -37,6 +37,9 @@ export N8N_EXTERNAL_STORAGE_S3_ACCOUNT_ID=... export N8N_EXTERNAL_STORAGE_S3_SECRET_KEY=... ``` +!!! note "No region" + If your provider does not require a region, e.g. Cloudflare, you can set `N8N_EXTERNAL_STORAGE_S3_BUCKET_REGION` to `'auto'`. + Finally, instruct n8n to store binary data in S3 and start the server. ```sh From 6b0eee49f8dc265b4469ece02c0347e76cbf1b41 Mon Sep 17 00:00:00 2001 From: Deborah Barnard Date: Thu, 28 Sep 2023 16:47:13 +0100 Subject: [PATCH 10/16] change structure - does not need to be full section, just page --- docs/hosting/external-storage.md | 72 ++++++++++++++++++++++++++++++++ mkdocs.yml | 2 +- 2 files changed, 73 insertions(+), 1 deletion(-) create mode 100644 docs/hosting/external-storage.md diff --git a/docs/hosting/external-storage.md b/docs/hosting/external-storage.md new file mode 100644 index 000000000..ee7e0e164 --- /dev/null +++ b/docs/hosting/external-storage.md @@ -0,0 +1,72 @@ +--- +description: External storage for your n8n instance. +contentType: howTo +--- + +# External storage + +!!! info "Feature availability" + * Available on Self-hosted Enterprise plans + * If you want access to this feature on Cloud Enterprise, contact us + +n8n can externally store binary data produced by workflow executions. This feature is useful to avoid relying on the filesystem for storing large amounts of binary data. In future, n8n will support external storage for other kinds of data in addition to binary data. + +## Storing n8n's binary data in S3 + +n8n can use [AWS S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) as an external store for binary data produced by workflow executions. Other S3-compatible services like Cloudflare R2 and Backblaze B2 may work, but are not officially supported. + +!!! info "Enterprise-tier feature" + You will need an [Enterprise license key](/enterprise-key/) for external storage. If your license key expires and you remain on S3 mode, the instance will be able to read from, but not write to, the S3 bucket. + +### Setup + +First, create and configure a bucket following the [official AWS documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html). You can use the following policy, replacing `` with the name of the bucket you created. + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "VisualEditor0", + "Effect": "Allow", + "Action": ["s3:*"], + "Resource": ["arn:aws:s3:::", "arn:aws:s3:::/*"] + } + ] +} +``` + +Once you finish creating the bucket, you will have a host, bucket name and region, and an access key ID and secret access key. You will need to set them in n8n's environment. + +```sh +export N8N_EXTERNAL_STORAGE_S3_HOST=... # example: s3.us-east-1.amazonaws.com +export N8N_EXTERNAL_STORAGE_S3_BUCKET_NAME=... +export N8N_EXTERNAL_STORAGE_S3_BUCKET_REGION=... +export N8N_EXTERNAL_STORAGE_S3_ACCOUNT_ID=... +export N8N_EXTERNAL_STORAGE_S3_SECRET_KEY=... +``` + +!!! note "No region" + If your provider does not require a region, e.g. Cloudflare, you can set `N8N_EXTERNAL_STORAGE_S3_BUCKET_REGION` to `'auto'`. + +Finally, instruct n8n to store binary data in S3 and start the server. + +```sh +export N8N_AVAILABLE_BINARY_DATA_MODES=filesystem,s3 +export N8N_DEFAULT_BINARY_DATA_MODE=s3 +``` + +### Usage + +Once S3 is enabled, any new binary data will be written to, and read from, the S3 bucket. Binary data will be written to your S3 bucket in this format: + +``` +workflows/{workflowId}/executions/{executionId}/binary_data/{binaryFileId} +``` + +Older binary data that was stored in the filesystem will continue to be read from the filesystem, if `filesystem` remains listed in `N8N_AVAILABLE_BINARY_DATA_MODES`. + +If you store binary data in S3 and later switch to filesystem mode, the instance will continue to read any data that was stored in S3, as long as `s3` remains listed in `N8N_AVAILABLE_BINARY_DATA_MODES` and your S3 credentials remain valid. + +!!! note "Binary data pruning" + At this time, binary data pruning is based on the active binary data mode. For example, if your instance stored data in S3 and was later switched to filesystem mode, only binary data in the filesystem will be pruned. This may change in future. diff --git a/mkdocs.yml b/mkdocs.yml index 2b4b9fbe9..395ca2791 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -557,7 +557,7 @@ nav: - Isolating n8n: hosting/environment-variables/configuration-examples/isolation.md - Supported databases and settings: hosting/supported-databases-settings.md - User management: hosting/user-management-self-hosted.md - - External storage: hosting/external-storage/index.md + - External storage: hosting/external-storage.md - Security audit: hosting/security-audit.md - Logging and monitoring: - hosting/logging-monitoring/index.md From 462787a637d022361eaf2f56d3540ac5ccb2154a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Iv=C3=A1n=20Ovejero?= Date: Mon, 2 Oct 2023 11:23:44 +0200 Subject: [PATCH 11/16] Rename envs --- docs/hosting/external-storage.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/hosting/external-storage.md b/docs/hosting/external-storage.md index ee7e0e164..53aadd9ee 100644 --- a/docs/hosting/external-storage.md +++ b/docs/hosting/external-storage.md @@ -42,8 +42,8 @@ Once you finish creating the bucket, you will have a host, bucket name and regio export N8N_EXTERNAL_STORAGE_S3_HOST=... # example: s3.us-east-1.amazonaws.com export N8N_EXTERNAL_STORAGE_S3_BUCKET_NAME=... export N8N_EXTERNAL_STORAGE_S3_BUCKET_REGION=... -export N8N_EXTERNAL_STORAGE_S3_ACCOUNT_ID=... -export N8N_EXTERNAL_STORAGE_S3_SECRET_KEY=... +export N8N_EXTERNAL_STORAGE_S3_ACCESS_KEY=... +export N8N_EXTERNAL_STORAGE_S3_ACCESS_SECRET=... ``` !!! note "No region" From 4630df7eb95747fe0d516ffbfe48ba8efeda8d72 Mon Sep 17 00:00:00 2001 From: Deborah Barnard Date: Mon, 2 Oct 2023 14:45:14 +0100 Subject: [PATCH 12/16] style edit & add env vars to reference doc --- .../environment-variables.md | 18 +++++++++++- docs/hosting/external-storage.md | 28 +++++++++++-------- styles/Vocab/default/accept.txt | 1 + 3 files changed, 34 insertions(+), 13 deletions(-) diff --git a/docs/hosting/environment-variables/environment-variables.md b/docs/hosting/environment-variables/environment-variables.md index bac5a1c33..afa3a911b 100644 --- a/docs/hosting/environment-variables/environment-variables.md +++ b/docs/hosting/environment-variables/environment-variables.md @@ -99,12 +99,15 @@ Enabling overwrites for credentials allows you to set default values for credent ## Binary data +By default, n8n uses the server filesystem to store binary data. Enterprise users can choose to use an external service instead. Refer to [External storage](/hosting/external-storage/) for more information on using external storage for binary data. + | Variable | Type | Default | Description | | :------- | :---- | :------- | :---------- | | `N8N_AVAILABLE_BINARY_DATA_MODES` | String | `filesystem` | A comma separated list of available binary data modes. | | `N8N_BINARY_DATA_STORAGE_PATH` | String | `N8N_USE_FOLDER/binaryData` | The path where n8n stores binary data. | | `N8N_BINARY_DATA_TTL` | Number | `60` | Time to live (in minutes) for binary data of unsaved executions. | -| `N8N_DEFAULT_BINARY_DATA_MODE` | String | `default` | The default binary data mode. `default` keeps binary data in memory. Set to `filesystem` to use the filesystem. | +| `N8N_DEFAULT_BINARY_DATA_MODE` | String | `default` | The default binary data mode. `default` keeps binary data in memory. Set to `filesystem` to use the filesystem, or `s3` to AWS S3. | + ## User management SMTP, and two-factor authentication @@ -200,6 +203,19 @@ Refer to [Log streaming](/log-streaming/) for more information on this feature. | `N8N_EVENTBUS_LOGWRITER_MAXFILESIZEINKB` | Number | `10240` | Maximum size (in kilo-bytes) of an event log file before a new one is started. | | `N8N_EVENTBUS_LOGWRITER_LOGBASENAME` | String | `n8nEventLog` | Basename of the event log file. | +## External data storage + +Refer to [External storage](/hosting/external-storage/) for more information on using external storage for binary data. + +| Variable | Type | Default | Description | +| :------- | :---- | :------- | :---------- | +| `N8N_EXTERNAL_STORAGE_S3_HOST` | String | - | Host of the n8n bucket in S3-compatible external storage. For example, `s3.us-east-1.amazonaws.com` | +| `N8N_EXTERNAL_STORAGE_S3_BUCKET_NAME` | String | - | Name of the n8n bucket in S3-compatible external storage. | +| `N8N_EXTERNAL_STORAGE_S3_BUCKET_REGION` | String | - | Region of the n8n bucket in S3-compatible external storage. For example, `us-east-1`| +| `N8N_EXTERNAL_STORAGE_S3_ACCESS_KEY` | String | - | Access key in S3-compatible external storage | +| `N8N_EXTERNAL_STORAGE_S3_ACCESS_SECRET` | String | - | Access secret in S3-compatible external storage. | + + ## Nodes | Variable | Type | Default | Description | diff --git a/docs/hosting/external-storage.md b/docs/hosting/external-storage.md index 53aadd9ee..625596857 100644 --- a/docs/hosting/external-storage.md +++ b/docs/hosting/external-storage.md @@ -1,5 +1,5 @@ --- -description: External storage for your n8n instance. +description: External storage of binary data for your n8n instance. contentType: howTo --- @@ -7,20 +7,22 @@ contentType: howTo !!! info "Feature availability" * Available on Self-hosted Enterprise plans - * If you want access to this feature on Cloud Enterprise, contact us + * If you want access to this feature on Cloud Enterprise, [contact n8n](https://n8n-community.typeform.com/to/y9X2YuGa){:target=_blank .external-link}. -n8n can externally store binary data produced by workflow executions. This feature is useful to avoid relying on the filesystem for storing large amounts of binary data. In future, n8n will support external storage for other kinds of data in addition to binary data. +n8n can store binary data produced by workflow executions externally. This feature is useful to avoid relying on the filesystem for storing large amounts of binary data. + +n8n will introduce external storage for other data types in the future. ## Storing n8n's binary data in S3 -n8n can use [AWS S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) as an external store for binary data produced by workflow executions. Other S3-compatible services like Cloudflare R2 and Backblaze B2 may work, but are not officially supported. +n8n supports [AWS S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html){:target=_blank .external-link} as an external store for binary data produced by workflow executions. You can use other S3-compatible services like Cloudflare R2 and Backblaze B2, but n8n doesn't officially support these. !!! info "Enterprise-tier feature" You will need an [Enterprise license key](/enterprise-key/) for external storage. If your license key expires and you remain on S3 mode, the instance will be able to read from, but not write to, the S3 bucket. ### Setup -First, create and configure a bucket following the [official AWS documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html). You can use the following policy, replacing `` with the name of the bucket you created. +Create and configure a bucket following the [AWS documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html){:target=_blank .external-link}. You can use the following policy, replacing `` with the name of the bucket you created: ```json { @@ -36,7 +38,7 @@ First, create and configure a bucket following the [official AWS documentation]( } ``` -Once you finish creating the bucket, you will have a host, bucket name and region, and an access key ID and secret access key. You will need to set them in n8n's environment. +Once you finish creating the bucket, you will have a host, bucket name and region, and an access key ID and secret access key. You need to set them in n8n's environment: ```sh export N8N_EXTERNAL_STORAGE_S3_HOST=... # example: s3.us-east-1.amazonaws.com @@ -47,26 +49,28 @@ export N8N_EXTERNAL_STORAGE_S3_ACCESS_SECRET=... ``` !!! note "No region" - If your provider does not require a region, e.g. Cloudflare, you can set `N8N_EXTERNAL_STORAGE_S3_BUCKET_REGION` to `'auto'`. + If your provider doesn't require a region, you can set `N8N_EXTERNAL_STORAGE_S3_BUCKET_REGION` to `'auto'`. -Finally, instruct n8n to store binary data in S3 and start the server. +Tell n8n to store binary data in S3: ```sh export N8N_AVAILABLE_BINARY_DATA_MODES=filesystem,s3 export N8N_DEFAULT_BINARY_DATA_MODE=s3 ``` +Restart the server to load the new configuration. + ### Usage -Once S3 is enabled, any new binary data will be written to, and read from, the S3 bucket. Binary data will be written to your S3 bucket in this format: +After you enable S3, n8n writes and reads any new binary data to and from the S3 bucket. n8n writes binary data to your S3 bucket in this format: ``` workflows/{workflowId}/executions/{executionId}/binary_data/{binaryFileId} ``` -Older binary data that was stored in the filesystem will continue to be read from the filesystem, if `filesystem` remains listed in `N8N_AVAILABLE_BINARY_DATA_MODES`. +n8n continues to read older binary data stored in the filesystem from the filesystem, if `filesystem` remains listed as an option in `N8N_AVAILABLE_BINARY_DATA_MODES`. -If you store binary data in S3 and later switch to filesystem mode, the instance will continue to read any data that was stored in S3, as long as `s3` remains listed in `N8N_AVAILABLE_BINARY_DATA_MODES` and your S3 credentials remain valid. +If you store binary data in S3 and later switch to filesystem mode, the instance continues to read any data stored in S3, as long as `s3` remains listed in `N8N_AVAILABLE_BINARY_DATA_MODES` and your S3 credentials remain valid. !!! note "Binary data pruning" - At this time, binary data pruning is based on the active binary data mode. For example, if your instance stored data in S3 and was later switched to filesystem mode, only binary data in the filesystem will be pruned. This may change in future. + Binary data pruning operates on the active binary data mode. For example, if your instance stored data in S3, and you later switched to filesystem mode, n8n only prunes binary data in the filesystem. This may change in future. diff --git a/styles/Vocab/default/accept.txt b/styles/Vocab/default/accept.txt index e0382d1df..0f748e198 100644 --- a/styles/Vocab/default/accept.txt +++ b/styles/Vocab/default/accept.txt @@ -2,6 +2,7 @@ Adalo Asana Authentik Axios +Backblaze backend Beeminder Boolean From ba971933e1c26d9a24aa810d0f60b70aca3ef031 Mon Sep 17 00:00:00 2001 From: Deborah Barnard Date: Mon, 2 Oct 2023 15:34:51 +0100 Subject: [PATCH 13/16] fix default --- docs/hosting/environment-variables/environment-variables.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hosting/environment-variables/environment-variables.md b/docs/hosting/environment-variables/environment-variables.md index afa3a911b..d59ff34ab 100644 --- a/docs/hosting/environment-variables/environment-variables.md +++ b/docs/hosting/environment-variables/environment-variables.md @@ -99,7 +99,7 @@ Enabling overwrites for credentials allows you to set default values for credent ## Binary data -By default, n8n uses the server filesystem to store binary data. Enterprise users can choose to use an external service instead. Refer to [External storage](/hosting/external-storage/) for more information on using external storage for binary data. +By default, n8n uses memory to store binary data. Enterprise users can choose to use an external service instead. Refer to [External storage](/hosting/external-storage/) for more information on using external storage for binary data. | Variable | Type | Default | Description | | :------- | :---- | :------- | :---------- | From fa718a2c0b1f0710bf0a17c533357e4232468a5d Mon Sep 17 00:00:00 2001 From: Deborah Date: Wed, 11 Oct 2023 12:55:34 +0100 Subject: [PATCH 14/16] Update latest and next version numbers in release notes --- docs/release-notes.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/release-notes.md b/docs/release-notes.md index 13c294453..3ce1cc13c 100644 --- a/docs/release-notes.md +++ b/docs/release-notes.md @@ -33,6 +33,9 @@ n8n uses [semantic versioning](https://semver.org/){:target=_blank .external-lin View the [commits](https://github.com/n8n-io/n8n/compare/n8n@1.10.0...n8n@1.10.1){:target=_blank .external-link} for this version.
**Release date:** 2023-10-11 +!!! note "Next version" + This is the `next` version. n8n recommends using the `latest` version. The `next` version may be unstable. To report issues, use the [forum](https://community.n8n.io/c/questions/12){:target=_blank .external-link}. + This release contains bug fixes. For full release details, refer to [Releases](https://github.com/n8n-io/n8n/releases){:target=_blank .external-link} on GitHub. @@ -45,6 +48,7 @@ View the [commits](https://github.com/n8n-io/n8n/compare/n8n@1.9.2...n8n@1.9.3){ !!! note "Latest version" This is the `latest` version. n8n recommends using the `latest` version. The `next` version may be unstable. To report issues, use the [forum](https://community.n8n.io/c/questions/12){:target=_blank .external-link}. + This release contains bug fixes. For full release details, refer to [Releases](https://github.com/n8n-io/n8n/releases){:target=_blank .external-link} on GitHub. @@ -63,8 +67,6 @@ For full release details, refer to [Releases](https://github.com/n8n-io/n8n/rele View the [commits](https://github.com/n8n-io/n8n/compare/n8n@1.9.1...n8n@1.10.0){:target=_blank .external-link} for this version.
**Release date:** 2023-10-05 -!!! note "Next version" - This is the `next` version. n8n recommends using the `latest` version. The `next` version may be unstable. To report issues, use the [forum](https://community.n8n.io/c/questions/12){:target=_blank .external-link}. This release contains bug fixes and preparatory work for new features. From 445d68ef1cac68ca6aeff7883c6bc6830eb2e159 Mon Sep 17 00:00:00 2001 From: Deborah Date: Wed, 11 Oct 2023 12:55:35 +0100 Subject: [PATCH 15/16] Update latest and next version numbers in snippet --- _snippets/self-hosting/installation/latest-next-version.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_snippets/self-hosting/installation/latest-next-version.md b/_snippets/self-hosting/installation/latest-next-version.md index f0aa95fa7..505bd235e 100644 --- a/_snippets/self-hosting/installation/latest-next-version.md +++ b/_snippets/self-hosting/installation/latest-next-version.md @@ -2,4 +2,4 @@ n8n releases a new minor version most weeks. The `latest` version is for production use. `next` is the most recent release. You should treat `next` as a beta: it may be unstable. To report issues, use the [forum](https://community.n8n.io/c/questions/12){:target=_blank .external-link}. Current `latest`: 1.9.3 - Current `next`: 1.10.0 + Current `next`: 1.10.1 From 19d8a60bad9f0a80735fcca2b30c514e342fa636 Mon Sep 17 00:00:00 2001 From: Deborah Barnard Date: Thu, 12 Oct 2023 09:05:45 +0100 Subject: [PATCH 16/16] remove section --- docs/hosting/external-storage/index.md | 16 ------- docs/hosting/external-storage/s3.md | 63 -------------------------- 2 files changed, 79 deletions(-) delete mode 100644 docs/hosting/external-storage/index.md delete mode 100644 docs/hosting/external-storage/s3.md diff --git a/docs/hosting/external-storage/index.md b/docs/hosting/external-storage/index.md deleted file mode 100644 index b9c588c2e..000000000 --- a/docs/hosting/external-storage/index.md +++ /dev/null @@ -1,16 +0,0 @@ ---- -description: External storage for your n8n instance. -contentType: overview ---- - -# External storage - -!!! info "Feature availability" - * Available on Self-hosted Enterprise plans - * If you want access to this feature on Cloud Enterprise, contact us - -n8n can externally store binary data produced by workflow executions. This feature is useful to avoid relying on the filesystem for storing large amounts of binary data. In future, n8n will support external storage for other kinds of data in addition to binary data. - -[[% import "_macros/section-toc.html" as sectionToc %]] - -[[ sectionToc.sectionToc(page) ]] diff --git a/docs/hosting/external-storage/s3.md b/docs/hosting/external-storage/s3.md deleted file mode 100644 index f18957afd..000000000 --- a/docs/hosting/external-storage/s3.md +++ /dev/null @@ -1,63 +0,0 @@ ---- -contentType: howto ---- - -# Storing n8n's binary data in S3 - -n8n can use [AWS S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) as an external store for binary data produced by workflow executions. Other S3-compatible services like Cloudflare R2 and Backblaze B2 may work, but are not officially supported. - -!!! info "Enterprise-tier feature" - You will need an [Enterprise license key](/enterprise-key/) for external storage. If your license key expires and you remain on S3 mode, the instance will be able to read from, but not write to, the S3 bucket. - -## Setup - -First, create and configure a bucket following the [official AWS documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html). You can use the following policy, replacing `` with the name of the bucket you created. - -```json -{ - "Version": "2012-10-17", - "Statement": [ - { - "Sid": "VisualEditor0", - "Effect": "Allow", - "Action": ["s3:*"], - "Resource": ["arn:aws:s3:::", "arn:aws:s3:::/*"] - } - ] -} -``` - -Once you finish creating the bucket, you will have a host, bucket name and region, and an access key ID and secret access key. You will need to set them in n8n's environment. - -```sh -export N8N_EXTERNAL_STORAGE_S3_HOST=... # example: s3.us-east-1.amazonaws.com -export N8N_EXTERNAL_STORAGE_S3_BUCKET_NAME=... -export N8N_EXTERNAL_STORAGE_S3_BUCKET_REGION=... -export N8N_EXTERNAL_STORAGE_S3_ACCOUNT_ID=... -export N8N_EXTERNAL_STORAGE_S3_SECRET_KEY=... -``` - -!!! note "No region" - If your provider does not require a region, e.g. Cloudflare, you can set `N8N_EXTERNAL_STORAGE_S3_BUCKET_REGION` to `'auto'`. - -Finally, instruct n8n to store binary data in S3 and start the server. - -```sh -export N8N_AVAILABLE_BINARY_DATA_MODES=filesystem,s3 -export N8N_DEFAULT_BINARY_DATA_MODE=s3 -``` - -## Usage - -Once S3 is enabled, any new binary data will be written to, and read from, the S3 bucket. Binary data will be written to your S3 bucket in this format: - -``` -workflows/{workflowId}/executions/{executionId}/binary_data/{binaryFileId} -``` - -Older binary data that was stored in the filesystem will continue to be read from the filesystem, if `filesystem` remains listed in `N8N_AVAILABLE_BINARY_DATA_MODES`. - -If you store binary data in S3 and later switch to filesystem mode, the instance will continue to read any data that was stored in S3, as long as `s3` remains listed in `N8N_AVAILABLE_BINARY_DATA_MODES` and your S3 credentials remain valid. - -!!! note "Binary data pruning" - At this time, binary data pruning is based on the active binary data mode. For example, if your instance stored data in S3 and was later switched to filesystem mode, only binary data in the filesystem will be pruned. This may change in future.