From b35e704e5dc3d33898f41c41b8a88e251512c44c Mon Sep 17 00:00:00 2001 From: Eliska Romanova Date: Mon, 2 Dec 2024 11:33:17 +0100 Subject: [PATCH] OBSDOCS-1478: Ability to modify UWM Prometheus scrape interval --- ...rape-samples-in-user-defined-projects.adoc | 1 + ...als-limits-for-user-defined-projects.adoc} | 57 ++++++++++--------- .../configuring-the-monitoring-stack.adoc | 4 +- .../troubleshooting-monitoring-issues.adoc | 2 +- .../investigating-monitoring-issues.adoc | 2 +- 5 files changed, 35 insertions(+), 31 deletions(-) rename modules/{monitoring-setting-scrape-sample-and-label-limits-for-user-defined-projects.adoc => monitoring-setting-scrape-and-evaluation-intervals-limits-for-user-defined-projects.adoc} (55%) diff --git a/modules/monitoring-limiting-scrape-samples-in-user-defined-projects.adoc b/modules/monitoring-limiting-scrape-samples-in-user-defined-projects.adoc index d98ba26caa..ede5fb096f 100644 --- a/modules/monitoring-limiting-scrape-samples-in-user-defined-projects.adoc +++ b/modules/monitoring-limiting-scrape-samples-in-user-defined-projects.adoc @@ -20,6 +20,7 @@ can use the following measures to control the impact of unbound metrics attribut * Limit the number of samples that can be accepted per target scrape in user-defined projects * Limit the number of scraped labels, the length of label names, and the length of label values +* Configure the intervals between consecutive scrapes and between Prometheus rule evaluations * Create alerts that fire when a scrape sample threshold is reached or when the target cannot be scraped [NOTE] diff --git a/modules/monitoring-setting-scrape-sample-and-label-limits-for-user-defined-projects.adoc b/modules/monitoring-setting-scrape-and-evaluation-intervals-limits-for-user-defined-projects.adoc similarity index 55% rename from modules/monitoring-setting-scrape-sample-and-label-limits-for-user-defined-projects.adoc rename to modules/monitoring-setting-scrape-and-evaluation-intervals-limits-for-user-defined-projects.adoc index ab03b74761..9f05c5570e 100644 --- a/modules/monitoring-setting-scrape-sample-and-label-limits-for-user-defined-projects.adoc +++ b/modules/monitoring-setting-scrape-and-evaluation-intervals-limits-for-user-defined-projects.adoc @@ -3,10 +3,16 @@ // * observability/monitoring/configuring-the-monitoring-stack.adoc :_mod-docs-content-type: PROCEDURE -[id="setting-scrape-sample-and-label-limits-for-user-defined-projects_{context}"] -= Setting scrape sample and label limits for user-defined projects +[id="setting-scrape-and-evaluation-intervals-limits-for-user-defined-projects_{context}"] += Setting scrape intervals, evaluation intervals, and enforced limits for user-defined projects -You can limit the number of samples that can be accepted per target scrape in user-defined projects. You can also limit the number of scraped labels, the length of label names, and the length of label values. +You can set the following scrape and label limits for user-defined projects: + +* Limit the number of samples that can be accepted per target scrape +* Limit the number of scraped labels +* Limit the length of label names and label values + +You can also set an interval between consecutive scrapes and between Prometheus rule evaluations. [WARNING] ==== @@ -34,7 +40,7 @@ endif::openshift-dedicated,openshift-rosa[] $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config ---- -. Add the `enforcedSampleLimit` configuration to `data/config.yaml` to limit the number of samples that can be accepted per target scrape in user-defined projects: +. Add the enforced limit and time interval configurations to `data/config.yaml`: + [source,yaml] ---- @@ -46,31 +52,28 @@ metadata: data: config.yaml: | prometheus: - enforcedSampleLimit: 50000 <1> + enforcedSampleLimit: 50000 # <1> + enforcedLabelLimit: 500 # <2> + enforcedLabelNameLengthLimit: 50 # <3> + enforcedLabelValueLengthLimit: 600 # <4> + scrapeInterval: 1m30s # <5> + evaluationInterval: 1m15s # <6> ---- <1> A value is required if this parameter is specified. This `enforcedSampleLimit` example limits the number of samples that can be accepted per target scrape in user-defined projects to 50,000. - -. Add the `enforcedLabelLimit`, `enforcedLabelNameLengthLimit`, and `enforcedLabelValueLengthLimit` configurations to `data/config.yaml` to limit the number of scraped labels, the length of label names, and the length of label values in user-defined projects: +<2> Specifies the maximum number of labels per scrape. +The default value is `0`, which specifies no limit. +<3> Specifies the maximum character length for a label name. +The default value is `0`, which specifies no limit. +<4> Specifies the maximum character length for a label value. +The default value is `0`, which specifies no limit. +<5> Specifies the interval between consecutive scrapes. The interval must be set between 5 seconds and 5 minutes. +The default value is `30s`. +<6> Specifies the interval between Prometheus rule evaluations. The interval must be set between 5 seconds and 5 minutes. +The default value for Prometheus is `30s`. + -[source,yaml] ----- -apiVersion: v1 -kind: ConfigMap -metadata: - name: user-workload-monitoring-config - namespace: openshift-user-workload-monitoring -data: - config.yaml: | - prometheus: - enforcedLabelLimit: 500 <1> - enforcedLabelNameLengthLimit: 50 <2> - enforcedLabelValueLengthLimit: 600 <3> ----- -<1> Specifies the maximum number of labels per scrape. -The default value is `0`, which specifies no limit. -<2> Specifies the maximum length in characters of a label name. -The default value is `0`, which specifies no limit. -<3> Specifies the maximum length in characters of a label value. -The default value is `0`, which specifies no limit. +[NOTE] +==== +You can also configure the `evaluationInterval` property for Thanos Ruler through the `data/config.yaml/thanosRuler` field. The default value for Thanos Ruler is `15s`. +==== . Save the file to apply the changes. The limits are applied automatically. diff --git a/observability/monitoring/configuring-the-monitoring-stack.adoc b/observability/monitoring/configuring-the-monitoring-stack.adoc index 61f4881b95..4c1a65054d 100644 --- a/observability/monitoring/configuring-the-monitoring-stack.adoc +++ b/observability/monitoring/configuring-the-monitoring-stack.adoc @@ -248,9 +248,9 @@ include::modules/monitoring-choosing-a-metrics-collection-profile.adoc[leveloffs * See xref:../../nodes/clusters/nodes-cluster-enabling-features.adoc[Enabling features using feature gates] for steps to enable Technology Preview features. endif::openshift-dedicated,openshift-rosa[] -// Managing scrape sample limits for user-defined projects +// Managing scrape and evaluation intervals and enforced limits for user-defined projects include::modules/monitoring-limiting-scrape-samples-in-user-defined-projects.adoc[leveloffset=+1] -include::modules/monitoring-setting-scrape-sample-and-label-limits-for-user-defined-projects.adoc[leveloffset=+2] +include::modules/monitoring-setting-scrape-and-evaluation-intervals-limits-for-user-defined-projects.adoc[leveloffset=+2] ifndef::openshift-dedicated,openshift-rosa[] include::modules/monitoring-creating-scrape-sample-alerts.adoc[leveloffset=+2] diff --git a/observability/monitoring/troubleshooting-monitoring-issues.adoc b/observability/monitoring/troubleshooting-monitoring-issues.adoc index d5c21e2ddd..096ec7d33a 100644 --- a/observability/monitoring/troubleshooting-monitoring-issues.adoc +++ b/observability/monitoring/troubleshooting-monitoring-issues.adoc @@ -37,7 +37,7 @@ include::modules/monitoring-determining-why-prometheus-is-consuming-disk-space.a .Additional resources * xref:../../observability/monitoring/accessing-third-party-monitoring-apis.adoc#about-accessing-monitoring-web-service-apis_accessing-monitoring-apis-by-using-the-cli[Accessing monitoring APIs by using the CLI] -* xref:../../observability/monitoring/configuring-the-monitoring-stack.adoc#setting-scrape-sample-and-label-limits-for-user-defined-projects_configuring-the-monitoring-stack[Setting a scrape sample limit for user-defined projects] +* xref:../../observability/monitoring/configuring-the-monitoring-stack.adoc#setting-scrape-and-evaluation-intervals-limits-for-user-defined-projects_configuring-the-monitoring-stack[Setting scrape intervals, evaluation intervals, and scrape limits for user-defined projects] * xref:../../support/getting-support.adoc#support-submitting-a-case_getting-support[Submitting a support case] // Resolving the KubePersistentVolumeFillingUp alert firing for Prometheus diff --git a/support/troubleshooting/investigating-monitoring-issues.adoc b/support/troubleshooting/investigating-monitoring-issues.adoc index adae4f21a1..b3496d3462 100644 --- a/support/troubleshooting/investigating-monitoring-issues.adoc +++ b/support/troubleshooting/investigating-monitoring-issues.adoc @@ -31,7 +31,7 @@ include::modules/monitoring-determining-why-prometheus-is-consuming-disk-space.a [role="_additional-resources"] .Additional resources -* See xref:../../observability/monitoring/configuring-the-monitoring-stack.adoc#setting-scrape-sample-and-label-limits-for-user-defined-projects_configuring-the-monitoring-stack[Setting a scrape sample limit for user-defined projects] for details on how to set a scrape sample limit and create related alerting rules +* xref:../../observability/monitoring/configuring-the-monitoring-stack.adoc#setting-scrape-and-evaluation-intervals-limits-for-user-defined-projects_configuring-the-monitoring-stack[Setting scrape and evaluation intervals and enforced limits for user-defined projects] // Resolving the KubePersistentVolumeFillingUp alert firing for Prometheus include::modules/monitoring-resolving-the-kubepersistentvolumefillingup-alert-firing-for-prometheus.adoc[leveloffset=+1] \ No newline at end of file