This commit avoids the situation where the operator would update the
StatefulSet because the concatenation of configured rules exceeds the
maximum ConfigMap size. To alleviate the issue, the operator now
configures the StatefulSet to always mount 3 "rule" ConfigMaps with
"optional: true". When the operator generates an additional ConfigMap,
it will get mounted automatically into the pod and the config-reloader
sidecar will detect and apply the new configuration. When the
number of "concrete" ConfigMaps decreases, the operation is also
non-disruptive.
The number has been chosen arbitrarily and it could change in the future
but 3 seems a good choice (original reports were complaining about the
number of ConfigMaps fluctuating between 1 and 2).
The change applies both to Prometheus and ThanosRuler.
Closes #5085
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
The Kubernetes version used in the e2e tests is v1.26.3 because v1.27.1
doesn't work yet with kind.
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
* pkg/prometheus/operator.go: Remove context.TODO()
Related to #3158
Signed-off-by: Jayapriya Pai <slashpai9@gmail.com>
* test/framework/*.go: Remove context.TODO()
Propagate context from framework struct instead of using context.TODO()
Related to #3158
Signed-off-by: Jayapriya Pai <slashpai9@gmail.com>
* test/e2e/*.go Remove context.TODO()
Propagate context from Framework struct instead of using context.TODO()
Related to #3158
Signed-off-by: Jayapriya Pai <slashpai9@gmail.com>
Now we can configure the operator to use mTLS RemoteWrite by referencing
the CA, cert and key directly from k8s Secrets/ConfigMaps.
If the key and the cert are both Secrets, they can exist as a single
Secret which contain both 'cert.pem' and 'key.pem' otherwise they can
exist as 2 different Secrets (or a Secret for the key and ConfigMap for
the cert).
Signed-off-by: Yoni Bettan <ybettan@redhat.com>
This commit bumps the kubernetes version dependencies from v1.17
to v1.18.
The kubernetes client-go library introduced breaking
changes in v1.18 by adding context.Context to many of its function
signatures. The kubernetes-sigs/clientgofix tools was used in this
commit to automatically add the necessary context.TODO() arguments
to all client-go usage in the repo.
Fixes #3146
Instead of marshalling all StatefulSet inputs to JSON and hashing that
string, use hashstructure [1] package. In addition do not generate new
rule config maps if underlying rules did not change.
[1] github.com/mitchellh/hashstructure
`k8s.io/api/core/v1.MaxSecretSize` specifies the hard limit Kubernetes
has on the size of config maps [1]. If a set of Prometheus Rule Files
exceeds this limits, they are split onto multiple config maps. If there
is more than one rule config map for a Prometheus, its configuration
secret and its statefulset spec are adjusted accordingly.
The smallest entity to split by is a Prometheus rule file. This patch
does not split an individual rule file, if it exeeds the limit by
itself.
[1] https://github.com/kubernetes/kubernetes/pull/19909/
This patch introduces a new Custom Resource Definition to the
Prometheus Operator - the Rule CRD. It addresses two main
needs:
1. Prometheus (alerting and recording) Rule validation during creation time
via Kubernetes Custom Resource Definition validation.
2. Life-cycle management of Prometheus application Rules alongside the
application itself, inside the applications Kubernetes namespace, not
necessarily the namespace of the scraping Prometheus instance.
A user defines Prometheus alerting and recording Rules via a Kubernetes
Custom Resource Definition. These Custom Resource Definitions can be
fully validated by the Kubernetes API server during creation time via
automatically generated OpenAPI specifications. Instead of the
restriction of a Prometheus instance to only select Rule definitions
inside its own namespace, the Prometheus specification is extended to
also specify namespaces to look for Rule Custom Resource Definitions
outside its own namespace.
---
Dependent technical changes:
- prometheus: Use github.com/jimmidyson/configmap-reload to reload rules
- prometheus: Remove Prometheus Statefulset deletion function. Starting
with K8s >=1.8 this is handled via OwnerReferences.
- prometheus: Do not add rule files checksum to Prometheus configuration
secret
- prometheus: Update StatefulSet only on relevant changes. Instead of
updating the Prometheus StatefulSet on every `sync()` run, only update
it if the input parameters to `makeStatefulSet` change. Enforce this
via a checksum of the parameters which is saved inside the annotations
of the statefulset.
- e2e/prometheus: Check how often resources (Secret, ConfigMap,
Prometheus CRD, Service) are updated to enforce that Prometheus Operator
only updated created resources if necessary.
- contrib/prometheus-config-reloader: Remove logic to retriev K8s
ConfigMaps. These are mounted into the pod right away now.
