From b2138ad55127ccf93f3deb4e2859d733bf23eaf3 Mon Sep 17 00:00:00 2001 From: Eliska Romanova Date: Wed, 14 Aug 2024 10:18:56 +0200 Subject: [PATCH] OBSDOCS-270: Reference doc for remote write configuration isn't exhaustive --- ...ring-configuring-remote-write-storage.adoc | 3 +- ...-remote-write-authentication-settings.adoc | 32 ++++----- ...mple-remote-write-queue-configuration.adoc | 68 +++++++++++++++++++ ...accessing-third-party-monitoring-apis.adoc | 2 +- ...on-monitoring-configuration-scenarios.adoc | 2 +- .../configuring-the-monitoring-stack.adoc | 16 +++-- 6 files changed, 96 insertions(+), 27 deletions(-) create mode 100644 modules/monitoring-example-remote-write-queue-configuration.adoc diff --git a/modules/monitoring-configuring-remote-write-storage.adoc b/modules/monitoring-configuring-remote-write-storage.adoc index 1b12b3e506..236b8abdda 100644 --- a/modules/monitoring-configuring-remote-write-storage.adoc +++ b/modules/monitoring-configuring-remote-write-storage.adoc @@ -3,10 +3,9 @@ // * observability/monitoring/configuring-the-monitoring-stack.adoc :_mod-docs-content-type: PROCEDURE -[id="configuring_remote_write_storage_{context}"] +[id="configuring-remote-write-storage_{context}"] = Configuring remote write storage -[role="_abstract"] You can configure remote write storage to enable Prometheus to send ingested metrics to remote systems for long-term storage. Doing so has no impact on how or for how long Prometheus stores metrics. .Prerequisites diff --git a/modules/monitoring-example-remote-write-authentication-settings.adoc b/modules/monitoring-example-remote-write-authentication-settings.adoc index 4ce0fa1b02..f8271a0428 100644 --- a/modules/monitoring-example-remote-write-authentication-settings.adoc +++ b/modules/monitoring-example-remote-write-authentication-settings.adoc @@ -27,8 +27,9 @@ monitoring user-defined projects endif::openshift-dedicated,openshift-rosa[] in the `{namespace-name}` namespace. -.Sample YAML for AWS Signature Version 4 authentication -==== +[id="remote-write-sample-yaml-aws-sigv4_{context}"] +== Sample YAML for AWS Signature Version 4 authentication + The following shows the settings for a `sigv4` secret named `sigv4-credentials` in the `{namespace-name}` namespace. [source,yaml,subs="attributes+"] @@ -77,11 +78,11 @@ data: <4> The key that contains the AWS API secret key in the specified `Secret` object. <5> The name of the AWS profile that is being used to authenticate. <6> The unique identifier for the Amazon Resource Name (ARN) assigned to your role. -==== -.Sample YAML for basic authentication -==== -The following shows sample basic authentication settings for a `Secret` object named `rw-basic-auth` in the `{namespace-name}` namespace: +[id="remote-write-sample-yaml-basic-auth_{context}"] +== Sample YAML for Basic authentication + +The following shows sample Basic authentication settings for a `Secret` object named `rw-basic-auth` in the `{namespace-name}` namespace: [source,yaml,subs="attributes+"] ---- @@ -124,10 +125,10 @@ data: <1> The name of the `Secret` object that contains the authentication credentials. <2> The key that contains the username in the specified `Secret` object. <3> The key that contains the password in the specified `Secret` object. -==== -.Sample YAML for authentication with a bearer token using a `Secret` Object -==== +[id="remote-write-sample-yaml-bearer-token_{context}"] +== Sample YAML for authentication with a bearer token using a `Secret` Object + The following shows bearer token settings for a `Secret` object named `rw-bearer-auth` in the `{namespace-name}` namespace: [source,yaml,subs="attributes+"] @@ -167,10 +168,10 @@ data: <1> The authentication type of the request. The default value is `Bearer`. <2> The name of the `Secret` object that contains the authentication credentials. <3> The key that contains the authentication token in the specified `Secret` object. -==== -.Sample YAML for OAuth 2.0 authentication -==== +[id="remote-write-sample-yaml-oauth-20_{context}"] +== Sample YAML for OAuth 2.0 authentication + The following shows sample OAuth 2.0 settings for a `Secret` object named `oauth2-credentials` in the `{namespace-name}` namespace: [source,yaml,subs="attributes+"] @@ -223,10 +224,10 @@ data: <3> The URL used to fetch a token with the specified `clientId` and `clientSecret`. <4> The OAuth 2.0 scopes for the authorization request. These scopes limit what data the tokens can access. <5> The OAuth 2.0 authorization request parameters required for the authorization server. -==== -.Sample YAML for TLS client authentication -==== +[id="remote-write-sample-yaml-tls_{context}"] +== Sample YAML for TLS client authentication + The following shows sample TLS client settings for a `tls` `Secret` object named `mtls-bundle` in the `{namespace-name}` namespace. [source,yaml,subs="attributes+"] @@ -277,7 +278,6 @@ data: <2> The key in the specified `Secret` object that contains the CA certificate for the endpoint. <3> The key in the specified `Secret` object that contains the client certificate for the endpoint. <4> The key in the specified `Secret` object that contains the client key secret. -==== // Unset the source code block attributes just to be safe. :!namespace-name: diff --git a/modules/monitoring-example-remote-write-queue-configuration.adoc b/modules/monitoring-example-remote-write-queue-configuration.adoc new file mode 100644 index 0000000000..24c8f1d252 --- /dev/null +++ b/modules/monitoring-example-remote-write-queue-configuration.adoc @@ -0,0 +1,68 @@ +// Module included in the following assemblies: +// +// * observability/monitoring/configuring-the-monitoring-stack.adoc + +:_mod-docs-content-type: REFERENCE +[id="example-remote-write-queue-configuration_{context}"] += Example remote write queue configuration + +// Set attributes to distinguish between cluster monitoring examples and user workload monitoring examples. +ifndef::openshift-dedicated,openshift-rosa[] +:configmap-name: cluster-monitoring-config +:namespace-name: openshift-monitoring +:prometheus-instance: prometheusK8s +endif::openshift-dedicated,openshift-rosa[] +ifdef::openshift-dedicated,openshift-rosa[] +:configmap-name: user-workload-monitoring-config +:namespace-name: openshift-user-workload-monitoring +:prometheus-instance: prometheus +endif::openshift-dedicated,openshift-rosa[] + +You can use the `queueConfig` object for remote write to tune the remote write queue parameters. The following example shows the queue parameters with their default values for +ifndef::openshift-dedicated,openshift-rosa[] +default platform monitoring +endif::openshift-dedicated,openshift-rosa[] +ifdef::openshift-dedicated,openshift-rosa[] +monitoring for user-defined projects +endif::openshift-dedicated,openshift-rosa[] +in the `{namespace-name}` namespace. + +.Example configuration of remote write parameters with default values +[source,yaml,subs="attributes+"] +---- +apiVersion: v1 +kind: ConfigMap +metadata: + name: {configmap-name} + namespace: {namespace-name} +data: + config.yaml: | + {prometheus-instance}: + remoteWrite: + - url: "https://remote-write-endpoint.example.com" + + queueConfig: + capacity: 10000 #<1> + minShards: 1 #<2> + maxShards: 50 #<3> + maxSamplesPerSend: 2000 #<4> + batchSendDeadline: 5s #<5> + minBackoff: 30ms #<6> + maxBackoff: 5s #<7> + retryOnRateLimit: false #<8> + sampleAgeLimit: 0s #<9> +---- +<1> The number of samples to buffer per shard before they are dropped from the queue. +<2> The minimum number of shards. +<3> The maximum number of shards. +<4> The maximum number of samples per send. +<5> The maximum time for a sample to wait in buffer. +<6> The initial time to wait before retrying a failed request. The time gets doubled for every retry up to the `maxbackoff` time. +<7> The maximum time to wait before retrying a failed request. +<8> Set this parameter to `true` to retry a request after receiving a 429 status code from the remote write storage. +<9> The samples that are older than the `sampleAgeLimit` limit are dropped from the queue. If the value is undefined or set to `0s`, the parameter is ignored. + +// Unset the source code block attributes just to be safe. +:!namespace-name: +:!prometheus-instance: + diff --git a/observability/monitoring/accessing-third-party-monitoring-apis.adoc b/observability/monitoring/accessing-third-party-monitoring-apis.adoc index c081968bdf..699b37df37 100644 --- a/observability/monitoring/accessing-third-party-monitoring-apis.adoc +++ b/observability/monitoring/accessing-third-party-monitoring-apis.adoc @@ -44,6 +44,6 @@ include::modules/monitoring-resources-reference-for-the-cluster-monitoring-opera ifndef::openshift-dedicated,openshift-rosa[] * xref:../../observability/monitoring/enabling-monitoring-for-user-defined-projects.adoc#enabling-monitoring-for-user-defined-projects_enabling-monitoring-for-user-defined-projects[Enabling monitoring for user-defined projects] endif::openshift-dedicated,openshift-rosa[] -* xref:../../observability/monitoring/configuring-the-monitoring-stack.adoc#configuring_remote_write_storage_configuring-the-monitoring-stack[Configuring remote write storage] +* xref:../../observability/monitoring/configuring-the-monitoring-stack.adoc#configuring-remote-write-storage_configuring-the-monitoring-stack[Configuring remote write storage] * xref:../../observability/monitoring/managing-metrics.adoc#managing-metrics[Managing metrics] * xref:../../observability/monitoring/managing-alerts.adoc#managing-alerts[Managing alerts] diff --git a/observability/monitoring/common-monitoring-configuration-scenarios.adoc b/observability/monitoring/common-monitoring-configuration-scenarios.adoc index 9270673798..3b6714f6cc 100644 --- a/observability/monitoring/common-monitoring-configuration-scenarios.adoc +++ b/observability/monitoring/common-monitoring-configuration-scenarios.adoc @@ -39,7 +39,7 @@ Specify the metrics data retention parameters for Prometheus and Thanos Ruler. * By default, in a newly installed {product-title} system, the monitoring `ClusterOperator` resource reports a `PrometheusDataPersistenceNotConfigured` status message to remind you that storage is not configured. ==== + -* For longer term data retention, xref:../../observability/monitoring/configuring-the-monitoring-stack.adoc#configuring_remote_write_storage_configuring-the-monitoring-stack[configure the remote write feature] to enable Prometheus to send ingested metrics to remote systems for storage. +* For longer term data retention, xref:../../observability/monitoring/configuring-the-monitoring-stack.adoc#configuring-remote-write-storage_configuring-the-monitoring-stack[configure the remote write feature] to enable Prometheus to send ingested metrics to remote systems for storage. + [IMPORTANT] ==== diff --git a/observability/monitoring/configuring-the-monitoring-stack.adoc b/observability/monitoring/configuring-the-monitoring-stack.adoc index 45e3d178e2..61f4881b95 100644 --- a/observability/monitoring/configuring-the-monitoring-stack.adoc +++ b/observability/monitoring/configuring-the-monitoring-stack.adoc @@ -204,16 +204,18 @@ endif::openshift-dedicated,openshift-rosa[] include::modules/monitoring-configuring-remote-write-storage.adoc[leveloffset=+1] include::modules/monitoring-supported-remote-write-authentication-settings.adoc[leveloffset=+2] include::modules/monitoring-example-remote-write-authentication-settings.adoc[leveloffset=+2] +include::modules/monitoring-example-remote-write-queue-configuration.adoc[leveloffset=+2] [role="_additional-resources"] .Additional resources - -* See link:https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage[Setting up remote write compatible endpoints] for steps to create a remote write compatible endpoint (such as Thanos). -* See link:https://prometheus.io/docs/practices/remote_write/#remote-write-tuning[Tuning remote write settings] for information about how to optimize remote write settings for different use cases. +ifndef::openshift-dedicated,openshift-rosa[] +* xref:../../rest_api/monitoring_apis/prometheus-monitoring-coreos-com-v1.adoc#spec-remotewrite-2[Prometheus REST API reference for remote write] +endif::openshift-dedicated,openshift-rosa[] +* link:https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage[Setting up remote write compatible endpoints] (Prometheus documentation) +* link:https://prometheus.io/docs/practices/remote_write/#remote-write-tuning[Tuning remote write settings] (Prometheus documentation) ifndef::openshift-dedicated,openshift-rosa[] // This xref might be relevant for ROSA/OSD if this content is reused: -* See xref:../../nodes/pods/nodes-pods-secrets.adoc#nodes-pods-secrets-about_nodes-pods-secrets[Understanding secrets] for steps to create and configure `Secret` objects in {product-title}. -* See the xref:../../rest_api/monitoring_apis/prometheus-monitoring-coreos-com-v1.adoc#spec-remotewrite-2[Prometheus REST API reference for remote write] for information about additional optional fields. +* xref:../../nodes/pods/nodes-pods-secrets.adoc#nodes-pods-secrets-about_nodes-pods-secrets[Understanding secrets] endif::openshift-dedicated,openshift-rosa[] // Configuring labels for outgoing metrics @@ -223,9 +225,9 @@ include::modules/monitoring-creating-cluster-id-labels-for-metrics.adoc[leveloff [role="_additional-resources"] .Additional resources -* For details about write relabel configuration, see xref:../../observability/monitoring/configuring-the-monitoring-stack.adoc#configuring_remote_write_storage_configuring-the-monitoring-stack[Configuring remote write storage]. +* xref:../../observability/monitoring/configuring-the-monitoring-stack.adoc#configuring-remote-write-storage_configuring-the-monitoring-stack[Configuring remote write storage] ifndef::openshift-dedicated,openshift-rosa[] -* For information about how to get your cluster ID, see xref:../../support/gathering-cluster-data.adoc#support-get-cluster-id_gathering-cluster-data[Obtaining your cluster ID]. +* xref:../../support/gathering-cluster-data.adoc#support-get-cluster-id_gathering-cluster-data[Obtaining your cluster ID] endif::openshift-dedicated,openshift-rosa[] ifndef::openshift-dedicated,openshift-rosa[]