diff --git a/pipeline/outputs/s3.md b/pipeline/outputs/s3.md index 4a3c19708..6843f2eb0 100644 --- a/pipeline/outputs/s3.md +++ b/pipeline/outputs/s3.md @@ -36,84 +36,86 @@ The [Prometheus success/retry/error metrics values](../../administration/monitor ## Configuration parameters | Key | Description | Default | -| --- | ----------- | ------- | -| `alias` | Sets an alias, use for multiple instances of the same output plugin. | _none_ | -| `authorization_endpoint_bearer_token` | Authorization endpoint bearer token. | _none_ | -| `authorization_endpoint_password` | Authorization endpoint basic authentication password. | _none_ | -| `authorization_endpoint_url` | Authorization endpoint URL. | _none_ | -| `authorization_endpoint_username` | Authorization endpoint basic authentication username. | _none_ | -| `auto_retry_requests` | Immediately retry failed requests to AWS services once. This option doesn't affect the normal Fluent Bit retry mechanism with backoff. Instead, it enables an immediate retry with no delay for networking errors, which can help improve throughput during transient network issues. | `true` | -| `blob_database_file` | Absolute path to a database file to be used to store blob files contexts. | _none_ | -| `bucket` | S3 bucket name. | _none_ | -| `canned_acl` | [Predefined Canned ACL policy](https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl) for S3 objects. | _none_ | -| `compression` | Compression type for S3 objects. Supported values: `gzip`, `zstd`, `snappy`. `arrow` and `parquet` are also available if Apache Arrow was enabled at compile time. See [Compression](#compression). | _none_ | -| `content_type` | A standard MIME type for the S3 object, set as the Content-Type HTTP header. | _none_ | -| `endpoint` | Custom endpoint for the S3 API. Endpoints can contain scheme and port. | _none_ | -| `external_id` | Specify an external ID for the STS API. Can be used with the `role_arn` parameter if your role requires an external ID. | _none_ | -| `file_delivery_attempt_limit` | File delivery attempt limit. | `1` | +|-----|-------------|----------| +| `alias` | Sets an alias, use for multiple instances of the same output plugin. | _none_ | +| `authorization_endpoint_bearer_token` | Authorization endpoint bearer token. | _none_ | +| `authorization_endpoint_password` | Authorization endpoint basic authentication password.| _none_ | +| `authorization_endpoint_url` | Authorization endpoint URL. | _none_ | +| `authorization_endpoint_username` | Authorization endpoint basic authentication username.| _none_ | +| `auto_retry_requests` | Immediately retry failed requests to AWS services once. This option doesn't affect the normal Fluent Bit retry mechanism with backoff. Instead, it enables an immediate retry with no delay for networking errors, which can help improve throughput during transient network issues. | `true` | +| `blob_database_file` | Absolute path to a database file to be used to store blob files contexts. | _none_ | +| `bucket` | S3 Bucket name | _none_ | +| `canned_acl` | [Predefined Canned ACL policy](https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl) for S3 objects. | _none_ | +| `compression` | Compression/format for S3 objects. Supported: `gzip` (always available) and `parquet` (requires Arrow build). For `gzip`, the `Content-Encoding` header is set to `gzip`. `parquet` is available **only when Fluent Bit is built with `-DFLB_ARROW=On`** and Arrow GLib/Parquet GLib are installed. Parquet is typically used with `use_put_object On`. | _none_ | +| `content_type` | A standard MIME type for the S3 object, set as the Content-Type HTTP header. | _none_ | +| `endpoint` | Custom endpoint for the S3 API. Endpoints can contain scheme and port. | _none_ | +| `external_id` | Specify an external ID for the STS API. Can be used with the `role_arn` parameter if your role requires an external ID. | _none_ | +| `file_delivery_attempt_limit` | File delivery attempt limit. | `1` | | `format` | Set the record output format. Supported values: `json_lines`, `otlp_json`. When set to `otlp_json`, the `log_key` option isn't supported and only `logs` event chunks are converted. | `json_lines` | -| `host` | IP address or hostname of the target HTTP server. | `127.0.0.1` | -| `json_date_format` | Specify the format of the date. Accepted values: `double`, `epoch`, `epoch_ms`, `iso8601` (2018-05-30T09:39:52.000681Z), `_java_sql_timestamp_` (2018-05-30 09:39:52.000681). | _none_ | -| `json_date_key` | Specify the name of the date key in the output record. To disable the time key, set the value to `false`. | `date` | +| `host` | IP address or hostname of the target HTTP server. | `127.0.0.1` | +| `json_date_format` | Specify the format of the date. Accepted values: `double`, `epoch`, `epoch_ms`, `iso8601` (2018-05-30T09:39:52.000681Z), `_java_sql_timestamp_` (2018-05-30 09:39:52.000681). | _none_ | +| `json_date_key` | Specify the name of the date key in the output record. To disable the time key, set the value to `false`. | `date` | | `log_key` | By default, the whole log record is sent to S3. When specifying a key name with this option, only the value of that key is sent to S3. For example, when using Docker you can specify `log_key log` and only the log message is sent to S3. | _none_ | -| `log_level` | Specifies the log level for output plugin. If not set here, plugin uses global log level in `service` section. | `info` | -| `log_suppress_interval` | Suppresses log messages from output plugin that appear similar within a specified time interval. `0` means no suppression. | `0` | -| `match` | Set a tag pattern to match records that output should process. Exact matches or wildcards (for example `*`). | _none_ | -| `match_regex` | Set a regular expression to match tags for output routing. This allows more flexible matching compared to wildcards. | _none_ | -| `net.connect_timeout` | Set maximum time allowed to establish a connection, this time includes the TLS handshake. | `10s` | -| `net.connect_timeout_log_error` | On connection timeout, specify if it should log an error. When disabled, the timeout is logged as a debug message. | `true` | -| `net.dns.mode` | Select the primary DNS connection type (TCP or UDP). | _none_ | -| `net.dns.prefer_ipv4` | Select the primary DNS resolver type (LEGACY or ASYNC). | _none_ | -| `net.dns.prefer_ipv6` | Prioritize IPv6 DNS results when trying to establish a connection. | _none_ | -| `net.io_timeout` | Set maximum time a connection can stay idle while assigned. | `0s` | -| `net.keepalive_max_recycle` | Set maximum number of times a keepalive connection can be used before it retries. | `2000` | -| `net.max_worker_connections` | Set the maximum number of active TCP connections that can be used per worker thread. | `0` | -| `net.proxy_env_ignore` | Ignore the environment variables `HTTP_PROXY`, `HTTPS_PROXY` and `NO_PROXY` when set. | `false` | -| `net.source_address` | Specify network address to bind for data traffic. | _none_ | -| `net.tcp_keepalive` | Enable or disable Keepalive support. | `off` | -| `net.tcp_keepalive_interval` | Interval between TCP keepalive probes when no response is received on a `keepidle` probe. | `-1` | -| `net.tcp_keepalive_probes` | Number of unacknowledged probes to consider a connection dead. | `-1` | -| `net.tcp_keepalive_time` | Interval between the last data packet sent and the first TCP keepalive probe. | `-1` | -| `part_delivery_attempt_limit` | File part delivery attempt limit. | `1` | -| `part_size` | Size of each part when uploading blob files. | `25M` | -| `port` | TCP port of the target HTTP server. | `80` | -| `preserve_data_ordering` | When an upload request fails, the last received chunk might swap with a later chunk, resulting in data shuffling. This feature prevents shuffling by using a queue logic for uploads. | `true` | -| `profile` | Option to specify an AWS Profile for credentials. | _none_ | -| `region` | The AWS region of your S3 bucket. China regions (`cn-*`), the AWS European Sovereign Cloud regions (`eusc-*`), and Amazon Dedicated Cloud regions (`us-iso-*`, `us-isob-*`, `us-isof-*`, `eu-isoe-*`) are supported; Fluent Bit automatically uses the correct endpoint suffix (`.amazonaws.com.cn`, `.amazonaws.eu`, `.c2s.ic.gov`, `.sc2s.sgov.gov`, `.csp.hci.ic.gov`, or `.cloud.adc-e.uk`) and no custom `endpoint` is required. | `us-east-1` | -| `retry_limit` | Set the maximum number of retries for the S3 plugin's internal retry system. Unlike other output plugins, S3 manages its own buffering and retries internally. If not explicitly set, defaults to `5` to allow sufficient retries for transient S3 errors and avoid wasting partially uploaded multipart data. Accepts an integer or `no_retries` to disable retries entirely. | `5` | -| `role_arn` | ARN of an IAM role to assume (for example, for cross account access). | _none_ | -| `s3_key_format` | Format string for keys in S3. This option supports a UUID, strftime time formatters, a syntax for selecting parts of the Fluent log tag using a syntax inspired by the `rewrite_tag` filter. Add `$UUID` in the format string to insert a random string. Add `$INDEX` in the format string to insert an integer that increments each upload. The `$INDEX` value saves in the `store_dir`. Add `$TAG` in the format string to insert the full log tag. Add `$TAG[0]` to insert the first part of the tag in the S3 key. The tag is split into parts using the characters specified with the `s3_key_format_tag_delimiters` option. Add the extension directly after the last piece of the format string to insert a key suffix. To specify a key suffix in `use_put_object` mode, you must specify `$UUID`. See [S3 Key Format](#s3-key-format-and-tag-delimiters). Time in `s3_key` is the timestamp of the first record in the S3 file. | `/fluent-bit-logs/$TAG/%Y/%m/%d/%H/%M/%S` | -| `s3_key_format_tag_delimiters` | A series of characters which will be used to split the tag into `parts` for use with the s3_key_format option. | `.` | -| `send_content_md5` | Send the Content-MD5 header with `PutObject` and UploadPart requests, as is required when Object Lock is enabled. | `false` | -| `static_file_path` | Disables behavior where UUID string appends to the end of the S3 key name when `$UUID` isn't provided in `s3_key_format`. `$UUID`, time formatters, `$TAG`, and other dynamic key formatters all work as expected while this feature is set to true. | `false` | -| `store_dir` | Directory to locally buffer data before sending. Plugin uses the S3 Multipart upload API to send data in chunks of 5 MB at a time. | `/tmp/fluent-bit/s3` | -| `store_dir_limit_size` | S3 plugin has its own buffering system with files in the `store_dir`. Use the `store_dir_limit_size` to limit the amount of data S3 buffers in the `store_dir` to limit disk usage. If the limit is reached, data will be discarded. Default is 0 which means unlimited. | `0` | -| `storage_class` | Specify the [storage class](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html#AmazonS3-PutObject-request-header-StorageClass) for S3 objects. If this option isn't specified, objects store with the default `STANDARD` storage class. | _none_ | -| `sts_endpoint` | Custom endpoint for the STS API. | _none_ | -| `tls` | Enable or disable TLS/SSL support. | `off` | -| `tls.ca_file` | Absolute path to CA certificate file. | _none_ | -| `tls.ca_path` | Absolute path to scan for certificate files. | _none_ | -| `tls.ciphers` | Specify TLS ciphers up to TLSv1.2. | _none_ | -| `tls.crt_file` | Absolute path to Certificate file. | _none_ | -| `tls.debug` | Set TLS debug level. Accepts `0` (No debug), `1` (Error), `2` (State change), `3` (Informational) and `4` (Verbose). | `1` | -| `tls.key_file` | Absolute path to private Key file. | _none_ | -| `tls.key_passwd` | Optional password for tls.key_file file. | _none_ | -| `tls.max_version` | Specify the maximum version of TLS. | _none_ | -| `tls.min_version` | Specify the minimum version of TLS. | _none_ | -| `tls.verify` | Force certificate validation. | `on` | -| `tls.verify_hostname` | Enable or disable to verify hostname. | `off` | -| `tls.vhost` | Hostname to be used for TLS SNI extension. | _none_ | -| `tls.windows.certstore_name` | Sets the `certstore` name on an output (Windows). | _none_ | -| `tls.windows.use_enterprise_store` | Sets whether using enterprise `certstore` or not on an output (Windows). | _none_ | -| `total_file_size` | Specify file size in S3. Minimum size is `1M`. With `use_put_object On` the maximum size is `1G`. With multipart uploads, the maximum size is `50G`. | `100000000` | -| `upload_chunk_size` | The size of each part for multipart uploads. Default: 5M, Max: 50M, Min: 5M. | `5242880` | +| `log_level` | Specifies the log level for output plugin. If not set here, plugin uses global log level in `service` section. | `info` | +| `log_supress_interval` | Suppresses log messages from output plugin that appear similar within a specified time interval. `0` no suppression. | `0` | +| `match` | Set a tag pattern to match records that output should process. Exact matches or wildcards (for example `*`). | _none_ | +| `match_regex` | Set a regular expression to match tags for output routing. This allows more flexible matching compared to wildcards. | _none_ | +| `net.connect_timeout` | Set maximum time allowed to establish a connection, this time includes the TLS handshake. | `10s` | +| `net.connect_timeout_log_error` | On connection timeout, specify if it should log an error. When disabled, the timeout is logged as a debug message. | `true` | +| `net.dns.mode` | Select the primary DNS connection type (TCP or UDP). | _none_ | +| `net.dns.prefer_ipv4` | Select the primary DNS resolver type (LEGACY or ASYNC). | _none_ | +| `net.dns.prefer_ipv6` | Prioritize IPv6 DNS results when trying to establish a connection. | _none_ | +| `net.io_timeout` | Set maximum time a connection can stay idle while assigned. | `0s` | +| `net.max_worker_connections` | Set the maximum number of active TCP connections that can be used per worker thread. | `0` | +| `net.proxy_env_ignore` | Ignore the environment variables `HTTP_PROXY`, `HTTPS_PROXY` and `NO_PROXY` when set. | `false` | +| `net.source_address` | Specify network address to bind for data traffic. | _none_ | +| `net.tcp_keepalive` | Enable or disable Keepalive support. | `off` | +| `net.keepalive_max_recycle` | Set maximum number of times a keepalive connection can be used before it retries. | `2000` | +| `net.tcp_keepalive_interval` | Interval between TCP keepalive probes when no response is received on a `keepidle` probe. | `-1` | +| `net.tcp_keepalive_probes` | Number of unacknowledged probes to consider a connection dead. | `-1` | +| `net.tcp_keepalive_time` | Interval between the last data packet sent and the first TCP keepalive probe. | `-1` | +| `part_delivery_attempt_limit` | File part delivery attempt limit. | `1` | +| `part_size` | Size of each part when uploading blob files. | `25M` | +| `port` | TCP port of the target HTTP server. | `80` | +| `preserve_data_ordering` | When an upload request fails, the last received chunk might swap with a later chunk, resulting in data shuffling. This feature prevents shuffling by using a queue logic for uploads. | `true` | +| `profile` | Option to specify an AWS Profile for credentials. | _none_ | +| `region` | The AWS region of your S3 bucket. | `us-east-1` | +| `retry_limit` | Set retry limit for output plugin when delivery fails. Integer, `no_limits`, `false`, or `off` to disable, or `no_retries` to disable retries entirely. | `1` | +| `role_arn` | ARN of an IAM role to assume (for example, for cross account access.) | _none_ | +| `s3_key_format` | Format string for keys in S3. This option supports a UUID, strftime time formatters, a syntax for selecting parts of the Fluent log tag using a syntax inspired by the `rewrite_tag` filter. Add `$UUID` in the format string to insert a random string. Add `$INDEX` in the format string to insert an integer that increments each upload. The `$INDEX` value saves in the `store_dir`. Add `$TAG` in the format string to insert the full log tag. Add `$TAG[0]` to insert the first part of the tag in the S3 key. The tag is split into parts using the characters specified with the `s3_key_format_tag_delimiters` option. Add the extension directly after the last piece of the format string to insert a key suffix. To specify a key suffix in `use_put_object` mode, you must specify `$UUID`. See [S3 Key Format](#s3-key-format-and-tag-delimiters). Time in `s3_key` is the timestamp of the first record in the S3 file. | `/fluent-bit-logs/$TAG/%Y/%m/%d/%H/%M/%S` | +| `s3_key_format_tag_delimiters` | A series of characters which will be used to split the tag into `parts` for use with the s3_key_format option. | `.` | +| `send_content_md5` | Send the Content-MD5 header with `PutObject` and UploadPart requests, as is required when Object Lock is enabled. | `false` | +| `sse` | Server-side encryption (`SSE`) for S3 objects. Set to `AES256` for S3-managed keys (`SSE-S3`), `aws:kms` for AWS Key Management Service (`SSE-KMS`), or `aws:kms:dsse` for dual-layer server-side encryption with `KMS` (`DSSE-KMS`). | _none_ | +| `sse_kms_key_id` | AWS `KMS` key Resource Name (ARN) for server-side encryption. Only applicable when `sse` is set to `aws:kms` or `aws:kms:dsse`. If not specified, the default AWS-managed `KMS` key for S3 will be used. | _none_ | +| `static_file_path` | Disables behavior where UUID string appends to the end of the S3 key name when `$UUID` isn't provided in `s3_key_format`. `$UUID`, time formatters, `$TAG`, and other dynamic key formatters all work as expected while this feature is set to true. | `false` | +| `store_dir` | Directory to locally buffer data before sending. Plugin uses the S3 Multipart upload API to send data in chunks of 5 MB at a time. | `/tmp/fluent-bit/s3` | +| `store_dir_limit_size` | S3 plugin has its own buffering system with files in the `store_dir`. Use the `store_dir_limit_size` to limit the amount of data S3 buffers in the `store_dir` to limit disk usage. If the limit is reached, data will be discarded. Default is 0 which means unlimited. | `0` | +| `storage_class` | Specify the [storage class](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html#AmazonS3-PutObject-request-header-StorageClass) for S3 objects. If this option isn't specified, objects store with the default `STANDARD` storage class. | _none_ | +| `sts_endpoint` | Custom endpoint for the STS API. | _none_ | +| `tls` | Enable or disable TLS/SSL support. | `off` | +| `tls.ca_file` | Absolute path to CA certificate file. | _none_ | +| `tls.ca_path` | Absolute path to scan for certificate files. | _none_ | +| `tls.ciphers` | Specify TLS ciphers up to TLSv1.2. | _none_ | +| `tls.crt_file` | Absolute path to Certificate file. | _none_ | +| `tls.debug` | Set TLS debug level. Accepts `0` (No debug), `1`(Error), `2` (State change), `3` (Informational) and `4` (Verbose). | `1` | +| `tls.key_file` | Absolute path to private Key file. | _none_ | +| `tls.key_passwd` | Optional password for tls.key_file file.| _none_ | +| `tls.max_version` | Specify the maximum version of TLS. | _none_ | +| `tls.min_version` | Specify the minimum version of TLS. | _none_ | +| `tls.verify` | Force certificate validation. | `on` | +| `tls.verify_hostname` | Enable or disable to verify hostname. | `off` | +| `tls.vhost` | Hostname to be used for TLS SNI extension. | _none_ | +| `tls.windows.certstore_name` | Sets the `certstore` name on an output (Windows). | _none_ | +| `tls.windows.use_enterprise_store` | Sets whether using enterprise `certstore` or not on an output (Windows). | _none_ | +| `total_file_size` | Specify file size in S3. Minimum size is `1M`. With `use_put_object On` the maximum size is `1G`. With multipart uploads, the maximum size is `50G`.| `100000000` | +| `upload_chunk_size` | The size of each part for multipart uploads. Default: 5M, Max: 50M, Min: 5M. | `5242880` | | `upload_part_freshness_limit` | Maximum lifespan of an uncommitted file part. | `6D` | -| `upload_parts_timeout` | Timeout to upload parts of a blob file. | `10m` | -| `upload_timeout` | When this amount of time elapses, Fluent Bit uploads and creates a new file in S3. Set to `60m` to upload a new file every hour. | `10m` | -| `use_put_object` | Use the S3 `PutObject` API instead of the multipart upload API. When enabled, the key extension is only available when `$UUID` is specified in `s3_key_format`. If `$UUID` isn't included, a random string appends format string and the key extension can't be customized. | `false` | +| `upload_part_freshness_timeout` | Maximum lifespan of an uncommitted file part. | `6D` | +| `upload_parts_timeout` | Timeout to upload parts of a blob file. | `10m` | +| `upload_timeout` | When this amount of time elapses, Fluent Bit uploads and creates a new file in S3. Set to `60m` to upload a new file every hour. | `10m` | +| `use_put_object` | Use the S3 `PutObject` API instead of the multipart upload API. When enabled, the key extension is only available when `$UUID` is specified in `s3_key_format`. If `$UUID` isn't included, a random string appends format string and the key extension can't be customized. | `false` | | `workers` | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `1` | - ## TLS / SSL To skip TLS verification, set `tls.verify` as `false`. For more details about the properties available and general configuration, refer to [TLS/SSL](../../administration/transport-security.md). @@ -590,8 +592,48 @@ pipeline: {% endtab %} {% endtabs %} +An example using `SSE-KMS` encryption: + +{% tabs %} +{% tab title="fluent-bit.yaml" %} + +```yaml +pipeline: + + outputs: + - name: s3 + match: '*' + bucket: your-bucket + region: us-east-1 + store_dir: /home/ec2-user/buffer + total_file_size: 50M + upload_timeout: 10m + sse: aws:kms + sse_kms_key_id: arn:aws:kms:us-east-1:123456789012:key/12345678-1234-1234-1234-123456789012 +``` + +{% endtab %} +{% tab title="fluent-bit.conf" %} + +```text +[OUTPUT] + Name s3 + Match * + bucket your-bucket + region us-east-1 + store_dir /home/ec2-user/buffer + total_file_size 50M + upload_timeout 10m + sse aws:kms + sse_kms_key_id arn:aws:kms:us-east-1:123456789012:key/12345678-1234-1234-1234-123456789012 +``` + +{% endtab %} +{% endtabs %} + ## AWS for Fluent Bit + Amazon distributes a container image with Fluent Bit and plugins. ### GitHub