This change introduces functional options for all 4 controllers
(previously only Prometheus used them).
It also reduces code duplication regarding API version checks.
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
This change implements a poll-based namespace lister/watcher when the
operator's service account isn't granted the list & watch permissions on
all namespaces. Instead of sending a watch request, the controller will
get every configured namespace every 15 seconds and sends the expected
watch events when a namespace is added/updated/deleted.
It still requires the service account to be granted the get permission
on the configured namespaces.
Closes #3847
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
When the operator was configured to select only a limited number of
namespaces, it would not watch for namespace changes. It means that the
operator may not reconcile when a namespace label is added/removed
(affecting which objects should be selected or not).
This change enables the operator to use a privileged namespace
lister/watcher whenever the service account has the needed permissions.
**IMPORTANT:** it also requires Kubernetes >= 1.22 to be effective but
the operator will degrade to the suboptimal implementation without watch
if this condition isn't met.
Closes #3847
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
Using github.com/go-kit/log instead of github.com/go-kit/kit reduces the
list of things this project depends on. This commit also removes the
wrapper for k8s.io/klog because none of the k8s libraries depend on it
anymore.
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
* pkg/listwatch: don't block when namespaces aren't found
The namespaces listerwatcher shouldn't return an error when some of the
monitored namespaces don't exist. Otherwise the operator will block
while waiting for the informers to sync their cache and it will never
be able to reconcile resources.
Closes #3347
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
* Refactor waitForCacheSync() functions
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
* Add `prometheus_operator_ready` metric
Also add the `PrometheusOperatorNotReady` alert that fires when a
controller hasn't initialized after 5 minutes.
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
* example/mixin: regenerate
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
Service/pod monitors and rules resources watched by the operator
can live in different namespaces than the configmaps and secrets
associated to the Prometheus custom resources.
Before this change, we had only one namespace informer scoped to the
monitor/rule namespaces. Whenever a change of configmap or secret was
detected, the operator couldn't retrieve the associated namespace
because it didn't exist in the monitor/rule namespaces and the
reconciliation wouldn't happen. The same issue applies for Thanos Ruler
and rules.
With this change, separate namespace informers are used when the
monitors/rules and prometheus namespaces aren't identical.
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
This commit bumps the kubernetes version dependencies from v1.17
to v1.18.
The kubernetes client-go library introduced breaking
changes in v1.18 by adding context.Context to many of its function
signatures. The kubernetes-sigs/clientgofix tools was used in this
commit to automatically add the necessary context.TODO() arguments
to all client-go usage in the repo.
Fixes #3146
The original implementation of the denylist-listwatch does not properly
filter out namespace objects as they are not namespaced. Nevertheless,
the denylist-listwatch is applied to the multi-namespace listwatch. This
means that changes to blacklisted namespaces would be handled by
unsuspecting informers.
This commit ensures that namespaces are correctly filtered.
Whenever a multi-listwatch contains only one listwatch, it
should simply use the underlying listwatch without any wrapper. This
improves performance and eliminates unnecessary go routines. This commit
fixes a regression where a denylist-listwatch would wrap underlying
listwatches even if the denylist was empty.
Currently, it is possible to explicitly watch for namespaces with the --namespaces option.
If unset, all namespaces are being watched.
The reverse is not possible today, namely exclude (deny) namespaces from being watched.
This fixes it by introducing the --deny-namespaces option.
It is mutually exclusive with --namespaces.
Currently, listwatch is racy, as revealed by the following error, when
running in k8s:
```
W1128 16:49:11.288643 1 reflector.go:270] github.com/coreos/prometheus-operator/pkg/prometheus/operator.go:403: watch of *v1.ConfigMap ended with: too old resource version: 14464 (19288)
panic: send on closed channel
goroutine 1373 [running]:
github.com/coreos/prometheus-operator/pkg/listwatch.newMultiWatch.func1(0x17f1c00, 0xc00084cab0, 0xc0002bbce0)
/home/sur/go/src/github.com/coreos/prometheus-operator/pkg/listwatch/listwatch.go:181 +0x4f
created by github.com/coreos/prometheus-operator/pkg/listwatch.newMultiWatch
/home/sur/go/src/github.com/coreos/prometheus-operator/pkg/listwatch/listwatch.go:175 +0x298
```
This fixes it by ensuring that the event sender channel is closed,
once all sender dispatcher goroutines exited.
This also fixes a small concurrency issue in the tests and adds the
`-race` parameter to unit tests in the Makefile.
This commit adds multi-namespace support to the Prometheus Operator. PO
now allows all informers to react to changes to multiple, individually
configured, namespaces, i.e. without specifying v1.NamespaceAll. This is
important for cases where the Prometheus Operator does not have
privileges to List all namespaces but should still administer multiple
namespaces. This capability is accomplished by abstracting the
cache.ListerWatchers for the various informers the PO uses.
