mirror of
https://github.com/openshift/openshift-docs.git
synced 2026-02-05 12:46:18 +01:00
128 lines
5.2 KiB
Plaintext
128 lines
5.2 KiB
Plaintext
// Module included in the following assemblies:
|
|
//
|
|
// * observability/logging/config/cluster-logging-systemd
|
|
|
|
:_mod-docs-content-type: PROCEDURE
|
|
[id="cluster-logging-systemd-scaling_{context}"]
|
|
= Configuring systemd-journald for OpenShift Logging
|
|
|
|
As you scale up your project, the default logging environment might need some
|
|
adjustments.
|
|
|
|
For example, if you are missing logs, you might have to increase the rate limits for journald.
|
|
You can adjust the number of messages to retain for a specified period of time to ensure that
|
|
OpenShift Logging does not use excessive resources without dropping logs.
|
|
|
|
You can also determine if you want the logs compressed, how long to retain logs, how or if the logs are stored,
|
|
and other settings.
|
|
|
|
.Procedure
|
|
|
|
. Create a Butane config file, `40-worker-custom-journald.bu`, that includes an `/etc/systemd/journald.conf` file with the required settings.
|
|
+
|
|
[NOTE]
|
|
====
|
|
include::snippets/butane-version.adoc[]
|
|
====
|
|
+
|
|
[source,yaml,subs="attributes+"]
|
|
----
|
|
variant: openshift
|
|
version: {product-version}.0
|
|
metadata:
|
|
name: 40-worker-custom-journald
|
|
labels:
|
|
machineconfiguration.openshift.io/role: "worker"
|
|
storage:
|
|
files:
|
|
- path: /etc/systemd/journald.conf
|
|
mode: 0644 <1>
|
|
overwrite: true
|
|
contents:
|
|
inline: |
|
|
Compress=yes <2>
|
|
ForwardToConsole=no <3>
|
|
ForwardToSyslog=no
|
|
MaxRetentionSec=1month <4>
|
|
RateLimitBurst=10000 <5>
|
|
RateLimitIntervalSec=30s
|
|
Storage=persistent <6>
|
|
SyncIntervalSec=1s <7>
|
|
SystemMaxUse=8G <8>
|
|
SystemKeepFree=20% <9>
|
|
SystemMaxFileSize=10M <10>
|
|
----
|
|
+
|
|
<1> Set the permissions for the `journald.conf` file. It is recommended to set `0644` permissions.
|
|
<2> Specify whether you want logs compressed before they are written to the file system.
|
|
Specify `yes` to compress the message or `no` to not compress. The default is `yes`.
|
|
<3> Configure whether to forward log messages. Defaults to `no` for each. Specify:
|
|
* `ForwardToConsole` to forward logs to the system console.
|
|
* `ForwardToKMsg` to forward logs to the kernel log buffer.
|
|
* `ForwardToSyslog` to forward to a syslog daemon.
|
|
* `ForwardToWall` to forward messages as wall messages to all logged-in users.
|
|
<4> Specify the maximum time to store journal entries. Enter a number to specify seconds. Or
|
|
include a unit: "year", "month", "week", "day", "h" or "m". Enter `0` to disable. The default is `1month`.
|
|
<5> Configure rate limiting. If more logs are received than what is specified in `RateLimitBurst` during the time interval defined by `RateLimitIntervalSec`, all further messages within the interval are dropped until the interval is over. It is recommended to set `RateLimitIntervalSec=30s` and `RateLimitBurst=10000`, which are the defaults.
|
|
<6> Specify how logs are stored. The default is `persistent`:
|
|
* `volatile` to store logs in memory in `/run/log/journal/`. These logs are lost after rebooting.
|
|
* `persistent` to store logs to disk in `/var/log/journal/`. systemd creates the directory if it does not exist.
|
|
* `auto` to store logs in `/var/log/journal/` if the directory exists. If it does not exist, systemd temporarily stores logs in `/run/systemd/journal`.
|
|
* `none` to not store logs. systemd drops all logs.
|
|
<7> Specify the timeout before synchronizing journal files to disk for *ERR*, *WARNING*, *NOTICE*, *INFO*, and *DEBUG* logs.
|
|
systemd immediately syncs after receiving a *CRIT*, *ALERT*, or *EMERG* log. The default is `1s`.
|
|
<8> Specify the maximum size the journal can use. The default is `8G`.
|
|
<9> Specify how much disk space systemd must leave free. The default is `20%`.
|
|
<10> Specify the maximum size for individual journal files stored persistently in `/var/log/journal`. The default is `10M`.
|
|
+
|
|
[NOTE]
|
|
====
|
|
If you are removing the rate limit, you might see increased CPU utilization on the
|
|
system logging daemons as it processes any messages that would have previously
|
|
been throttled.
|
|
====
|
|
+
|
|
For more information on systemd settings, see link:https://www.freedesktop.org/software/systemd/man/journald.conf.html[https://www.freedesktop.org/software/systemd/man/journald.conf.html]. The default settings listed on that page might not apply to {product-title}.
|
|
+
|
|
// Defaults from https://github.com/openshift/openshift-ansible/pull/3753/files#diff-40b7a7231e77d95ca6009dc9bcc0f470R33-R34
|
|
|
|
. Use Butane to generate a `MachineConfig` object file, `40-worker-custom-journald.yaml`, containing the configuration to be delivered to the nodes:
|
|
+
|
|
[source,terminal]
|
|
----
|
|
$ butane 40-worker-custom-journald.bu -o 40-worker-custom-journald.yaml
|
|
----
|
|
|
|
. Apply the machine config. For example:
|
|
+
|
|
[source,terminal]
|
|
----
|
|
$ oc apply -f 40-worker-custom-journald.yaml
|
|
----
|
|
+
|
|
The controller detects the new `MachineConfig` object and generates a new `rendered-worker-<hash>` version.
|
|
|
|
. Monitor the status of the rollout of the new rendered configuration to each node:
|
|
+
|
|
[source,terminal]
|
|
----
|
|
$ oc describe machineconfigpool/worker
|
|
----
|
|
+
|
|
.Example output
|
|
[source,terminal]
|
|
----
|
|
Name: worker
|
|
Namespace:
|
|
Labels: machineconfiguration.openshift.io/mco-built-in=
|
|
Annotations: <none>
|
|
API Version: machineconfiguration.openshift.io/v1
|
|
Kind: MachineConfigPool
|
|
|
|
...
|
|
|
|
Conditions:
|
|
Message:
|
|
Reason: All nodes are updating to rendered-worker-913514517bcea7c93bd446f4830bc64e
|
|
----
|