mirror of
https://github.com/openshift/openshift-docs.git
synced 2026-02-05 12:46:18 +01:00
Fix audit logs
This commit is contained in:
Maciej Szulik
committed by
openshift-cherrypick-robot
openshift-cherrypick-robot
parent
87ff03cae6
commit
357228edc1
@@ -1,183 +0,0 @@
|
||||
// Module included in the following assemblies:
|
||||
//
|
||||
// * nodes/nodes-audit-log.adoc
|
||||
|
||||
[id="nodes-nodes-audit-log-basic-config_{context}"]
|
||||
= Configuring the API Audit Log level
|
||||
|
||||
|
||||
You can configure the audit feature to set log level, retention policy, and the type of events to log.
|
||||
|
||||
//This is a best guess attempt inspired by https://github.com/openshift/api/pull/279
|
||||
|
||||
.Procedure
|
||||
|
||||
. Set the audit log level:
|
||||
|
||||
.. Get the name of the API server Custom Resource (CR):
|
||||
+
|
||||
----
|
||||
$ oc get APIServer
|
||||
|
||||
NAME AGE
|
||||
cluster 18h
|
||||
----
|
||||
|
||||
.. Edit the API server CR:
|
||||
+
|
||||
----
|
||||
$ oc edit APIServer cluster
|
||||
----
|
||||
+
|
||||
[source,yaml]
|
||||
----
|
||||
apiVersion: config.openshift.io/v1
|
||||
kind: APIServer
|
||||
metadata:
|
||||
annotations:
|
||||
release.openshift.io/create-only: "true"
|
||||
creationTimestamp: 2019-05-09T18:56:37Z
|
||||
generation: 2
|
||||
name: cluster
|
||||
resourceVersion: "562502"
|
||||
selfLink: /apis/config.openshift.io/v1/apiservers/cluster
|
||||
uid: 2c4e980b-728c-11e9-967d-0a973adad40e
|
||||
spec:
|
||||
logLevel: "Normal" <1>
|
||||
operatorloglevel: "TraceAll" <2>
|
||||
----
|
||||
<1> Specify the log level for an overall component.
|
||||
<2> Specify the log level for the API server Operator.
|
||||
+
|
||||
You can set the log level for both settings to one of the following. The setting can be different for each setting:
|
||||
+
|
||||
* *Normal*. Normal is the default. Normal working log information, including helpful notices for auditing or common operations. Similar to `glog=2`.
|
||||
* *Debug*. Debug is for troubleshooting problems. A greater quantity of notices than *Normal*, but contain less information than *Trace*. Common operations might be logged. Similar to `glog=4`.
|
||||
* *Trace*. Trace is for troubleshooting problems when *Debug* is not verbose enough. Logs every function call as part of a common operation, including tracing execution of a query. Similar to `glog=6`.
|
||||
* *TraceAll*. TraceAll is troubleshooting at the level of API content/decoding. Contains complete body content. In production clusters, this setting causes performance degradation and results in a significant number of logs. Similar to `glog=8`.
|
||||
|
||||
////
|
||||
. Configure the audit parameters
|
||||
|
||||
.. Edit the auditing configuration file on an API server pod in the `openshift-kube-apiserver` project:
|
||||
+
|
||||
----
|
||||
$ oc -n openshift-kube-apiserver exec <kube-api-server-name> cat /etc/kubernetes/static-pod-resources/configmaps/config/config.yaml |json_reformat
|
||||
----
|
||||
+
|
||||
For example:
|
||||
+
|
||||
----
|
||||
$ oc -n openshift-kube-apiserver exec kube-apiserver-ip-10-0-129-10.ec2.internal cat /etc/kubernetes/static-pod-resources/configmaps/config/config.yaml |json_reformat
|
||||
----
|
||||
|
||||
.. Edit the `auditConfig` stanza as needed:
|
||||
+
|
||||
----
|
||||
"auditConfig": {
|
||||
"auditFilePath": "/var/log/kube-apiserver/audit.log", <1>
|
||||
"enabled": true, <2>
|
||||
"logFormat": "json", <3>
|
||||
"maximumFileSizeMegabytes": 100, <4>
|
||||
"maximumRetainedFiles": 10, <5>
|
||||
"policyConfiguration": { <6>
|
||||
"apiVersion": "audit.k8s.io/v1beta1",
|
||||
"kind": "Policy",
|
||||
"omitStages": [
|
||||
"RequestReceived"
|
||||
],
|
||||
"rules": [
|
||||
{
|
||||
"level": "None",
|
||||
"resources": [
|
||||
{
|
||||
"group": "",
|
||||
"resources": [
|
||||
"events"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"level": "None",
|
||||
"nonResourceURLs": [
|
||||
"/api*",
|
||||
"/version",
|
||||
"/healthz",
|
||||
"/readyz"
|
||||
],
|
||||
"userGroups": [
|
||||
"system:authenticated",
|
||||
"system:unauthenticated"
|
||||
]
|
||||
},
|
||||
{
|
||||
"level": "Metadata",
|
||||
"omitStages": [
|
||||
"RequestReceived"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
|
||||
----
|
||||
<1> File path where the requests should be logged to. If not set, logs are printed to master logs.
|
||||
<2> A boolean to enable or disable audit logs. Default is `true`.
|
||||
<3> Specifies maximum number of days to retain old audit log files based on the time stamp encoded in their filename.
|
||||
<4> Specifies the maximum number of old audit log files to retain.
|
||||
<5> Specifies maximum size in megabytes of the log file before it gets rotated. Defaults to 100MB.
|
||||
<6> Configure the audit policy, described in the next step.
|
||||
|
||||
.. Configure the audit policy setting:
|
||||
+
|
||||
----
|
||||
"policyConfiguration": {
|
||||
"apiVersion": "audit.k8s.io/v1beta1",
|
||||
"kind": "Policy",
|
||||
"omitStages": [
|
||||
"RequestReceived" <1>
|
||||
],
|
||||
"rules": [
|
||||
{
|
||||
"level": "None", <2>
|
||||
"resources": [
|
||||
{
|
||||
"group": "",
|
||||
"resources": [
|
||||
"events"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"level": "None",
|
||||
"nonResourceURLs": [ <3>
|
||||
"/api*",
|
||||
"/version",
|
||||
"/healthz",
|
||||
"/readyz"
|
||||
],
|
||||
"userGroups": [ <4>
|
||||
"system:authenticated",
|
||||
"system:unauthenticated"
|
||||
]
|
||||
},
|
||||
{
|
||||
"level": "Metadata", <5>
|
||||
"omitStages": [
|
||||
"RequestReceived"
|
||||
]
|
||||
----
|
||||
<1> Configure whether to generate audit events for all requests in RequestReceived stage.
|
||||
<2> Configure whether to log API server events.
|
||||
<3> Configure whether to log authenticated requests to certain non-resource URL paths.
|
||||
<4> Configure a list of users the rule applies to.
|
||||
<5> Configure a catch-all rule to log all other requests at the Metadata level. Long-running requests like watches that fall under this rule will not generate an audit event in RequestReceived.
|
||||
+
|
||||
For each rule, set `level` to one of the following:
|
||||
* None. Do not log events that match this rule.
|
||||
* Metadata - Log request metadata: requesting user, timestamp, resource, verb, and so forth. Do not request or response body.
|
||||
* Request - Log event metadata and request body but not response body. This does not apply for non-resource requests.
|
||||
* RequestReceived - Log event metadata, request and response bodies. This does not apply for non-resource requests.
|
||||
////
|
||||
@@ -3,20 +3,20 @@
|
||||
// * nodes/nodes-audit-log.adoc
|
||||
|
||||
[id="nodes-nodes-audit-log-basic-viewing_{context}"]
|
||||
= Viewing the API Audit Log
|
||||
= Viewing the audit log
|
||||
|
||||
You can view the basic audit log.
|
||||
You can view logs for the {product-title} API server or the Kubernetes API server for each master node.
|
||||
|
||||
.Procedure
|
||||
|
||||
To view the basic audit log:
|
||||
To view the audit log:
|
||||
|
||||
. View the {product-title} API server logs
|
||||
|
||||
.. If necessary, get the node IP and name of the log you want to view:
|
||||
.. If necessary, get the node name of the log you want to view:
|
||||
+
|
||||
----
|
||||
$ oc --insecure-skip-tls-verify adm node-logs --role=master --path=openshift-apiserver/
|
||||
$ oc adm node-logs --role=master --path=openshift-apiserver/
|
||||
|
||||
ip-10-0-140-97.ec2.internal audit-2019-04-09T00-12-19.834.log
|
||||
ip-10-0-140-97.ec2.internal audit-2019-04-09T11-13-00.469.log
|
||||
@@ -32,28 +32,28 @@ ip-10-0-170-165.ec2.internal audit.log
|
||||
.. View the {product-title} API server log for a specific master node and timestamp or view all the logs for that master:
|
||||
+
|
||||
----
|
||||
$ oc adm node-logs <node-ip> <log-name> --path=openshift-apiserver/<log-name>
|
||||
$ oc adm node-logs <node-name> --path=openshift-apiserver/<log-name>
|
||||
----
|
||||
+
|
||||
For example:
|
||||
+
|
||||
----
|
||||
$ oc adm node-logs ip-10-0-140-97.ec2.internal audit-2019-04-08T13-09-01.227.log --path=openshift-apiserver/audit-2019-04-08T13-09-01.227.log
|
||||
$ oc adm node-logs ip-10-0-140-97.ec2.internal audit.log --path=openshift-apiserver/audit.log
|
||||
$ oc adm node-logs ip-10-0-140-97.ec2.internal --path=openshift-apiserver/audit-2019-04-08T13-09-01.227.log
|
||||
$ oc adm node-logs ip-10-0-140-97.ec2.internal --path=openshift-apiserver/audit.log
|
||||
----
|
||||
+
|
||||
The output appears similar to the following:
|
||||
+
|
||||
----
|
||||
ip-10-0-140-97.ec2.internal {"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2019-04-09T18:52:03Z"},"level":"Metadata","timestamp":"2019-04-09T18:52:03Z","auditID":"9708b50d-8956-4c87-b9eb-a53ba054c13d","stage":"ResponseComplete","requestURI":"/","verb":"get","user":{"username":"system:anonymous","groups":["system:unauthenticated"]},"sourceIPs":["10.128.0.1"],"userAgent":"Go-http-client/2.0","responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2019-04-09T18:52:03.914638Z","stageTimestamp":"2019-04-09T18:52:03.915080Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"cluster-status-binding\" of ClusterRole \"cluster-status\" to Group \"system:unauthenticated\""}}
|
||||
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"ad209ce1-fec7-4130-8192-c4cc63f1d8cd","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/openshift-kube-controller-manager/configmaps/cert-recovery-controller-lock?timeout=35s","verb":"update","user":{"username":"system:serviceaccount:openshift-kube-controller-manager:localhost-recovery-client","uid":"dd4997e3-d565-4e37-80f8-7fc122ccd785","groups":["system:serviceaccounts","system:serviceaccounts:openshift-kube-controller-manager","system:authenticated"]},"sourceIPs":["::1"],"userAgent":"cluster-kube-controller-manager-operator/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"configmaps","namespace":"openshift-kube-controller-manager","name":"cert-recovery-controller-lock","uid":"5c57190b-6993-425d-8101-8337e48c7548","apiVersion":"v1","resourceVersion":"574307"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2020-04-02T08:27:20.200962Z","stageTimestamp":"2020-04-02T08:27:20.206710Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"system:openshift:operator:kube-controller-manager-recovery\" of ClusterRole \"cluster-admin\" to ServiceAccount \"localhost-recovery-client/openshift-kube-controller-manager\""}}
|
||||
----
|
||||
|
||||
. View the Kubernetes API server logs:
|
||||
|
||||
.. If necessary, get the node IP and name of the log you want to view:
|
||||
.. If necessary, get the node name of the log you want to view:
|
||||
+
|
||||
----
|
||||
$ oc --insecure-skip-tls-verify adm node-logs --role=master --path=kube-apiserver/
|
||||
$ oc adm node-logs --role=master --path=kube-apiserver/
|
||||
|
||||
ip-10-0-140-97.ec2.internal audit-2019-04-09T14-07-27.129.log
|
||||
ip-10-0-140-97.ec2.internal audit-2019-04-09T19-18-32.542.log
|
||||
@@ -69,19 +69,18 @@ ip-10-0-170-165.ec2.internal audit.log
|
||||
.. View the Kubernetes API server log for a specific master node and timestamp or view all the logs for that master:
|
||||
+
|
||||
----
|
||||
$ oc adm node-logs <node-ip> <log-name> --path=kube-apiserver/<log-name>
|
||||
$ oc adm node-logs <node-name> --path=kube-apiserver/<log-name>
|
||||
----
|
||||
+
|
||||
For example:
|
||||
+
|
||||
----
|
||||
$ oc adm node-logs ip-10-0-140-97.ec2.internal audit-2019-04-09T14-07-27.129.log --path=kube-apiserver/audit-2019-04-09T14-07-27.129.log
|
||||
$ oc adm node-logs ip-10-0-170-165.ec2.internal audit.log --path=kube-apiserver/audit.log
|
||||
$ oc adm node-logs ip-10-0-140-97.ec2.internal --path=kube-apiserver/audit-2019-04-09T14-07-27.129.log
|
||||
$ oc adm node-logs ip-10-0-170-165.ec2.internal --path=kube-apiserver/audit.log
|
||||
----
|
||||
+
|
||||
The output appears similar to the following:
|
||||
+
|
||||
----
|
||||
ip-10-0-140-97.ec2.internal {"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2019-04-09T19:56:58Z"},"level":"Metadata","timestamp":"2019-04-09T19:56:58Z","auditID":"6e96c88b-ab6f-44d2-b62e-d1413efd676b","stage":"ResponseComplete","requestURI":"/api/v1/nodes/audit-2019-04-09T14-07-27.129.log","verb":"get","user":{"username":"kube:admin","groups":["system:cluster-admins","system:authenticated"],"extra":{"scopes.authorization.openshift.io":["user:full"]}},"sourceIPs":["10.0.57.93"],"userAgent":"oc/v1.14.6+c4799753c (linux/amd64) kubernetes/ba88cb2","objectRef":{"resource":"nodes","name":"audit-2019-04-09T14-07-27.129.log","apiVersion":"v1"},"responseStatus":{"metadata":{},"status":"Failure","reason":"NotFound","code":404},"requestReceivedTimestamp":"2019-04-09T19:56:58.982157Z","stageTimestamp":"2019-04-09T19:56:58.985300Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"cluster-admins\" of ClusterRole \"cluster-admin\" to Group \"system:cluster-admins\""}}
|
||||
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"ad209ce1-fec7-4130-8192-c4cc63f1d8cd","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/openshift-kube-controller-manager/configmaps/cert-recovery-controller-lock?timeout=35s","verb":"update","user":{"username":"system:serviceaccount:openshift-kube-controller-manager:localhost-recovery-client","uid":"dd4997e3-d565-4e37-80f8-7fc122ccd785","groups":["system:serviceaccounts","system:serviceaccounts:openshift-kube-controller-manager","system:authenticated"]},"sourceIPs":["::1"],"userAgent":"cluster-kube-controller-manager-operator/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"configmaps","namespace":"openshift-kube-controller-manager","name":"cert-recovery-controller-lock","uid":"5c57190b-6993-425d-8101-8337e48c7548","apiVersion":"v1","resourceVersion":"574307"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2020-04-02T08:27:20.200962Z","stageTimestamp":"2020-04-02T08:27:20.206710Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"system:openshift:operator:kube-controller-manager-recovery\" of ClusterRole \"cluster-admin\" to ServiceAccount \"localhost-recovery-client/openshift-kube-controller-manager\""}}
|
||||
----
|
||||
|
||||
|
||||
@@ -3,30 +3,34 @@
|
||||
// * nodes/nodes-audit-log.adoc
|
||||
|
||||
[id="nodes-pods-audit-log-basic_{context}"]
|
||||
= About the API Audit Log
|
||||
= About the API audit log
|
||||
|
||||
Audit works at the API server level, logging all requests coming to the server.
|
||||
Each audit log contains two entries:
|
||||
Audit works at the API server level, logging all requests coming to the server. Each audit log contains the following information:
|
||||
|
||||
. The request line containing:
|
||||
.. A Unique ID allowing to match the response line (see #2)
|
||||
.. The source IP of the request
|
||||
.. The HTTP method being invoked
|
||||
.. The original user invoking the operation
|
||||
.. The impersonated user for the operation (`self` meaning himself)
|
||||
.. The impersonated group for the operation (`lookup` meaning user's group)
|
||||
.. The namespace of the request or <none>
|
||||
.. The URI as requested
|
||||
.Audit log fields
|
||||
[cols="1,2",options="header"]
|
||||
|===
|
||||
|Field |Description
|
||||
|`level` | The audit level at which the event was generated.
|
||||
|`auditID` |A unique audit ID, generated for each request.
|
||||
|`stage` |The stage of the request handling when this event instance was generated.
|
||||
|`requestURI` |The request URI as sent by the client to a server.
|
||||
|`verb` |The Kubernetes verb associated with the request. For non-resource requests, this is the lowercase HTTP method.
|
||||
|`user` |The authenticated user information.
|
||||
|`impersonatedUser` |Optional. The impersonated user information, if the request is impersonating another user.
|
||||
|`sourceIPs` |Optional. The source IPs, from where the request originated and any intermediate proxies.
|
||||
|`userAgent` |Optional. The user agent string reported by the client. Note that the user agent is provided by the client, and must not be trusted.
|
||||
|`objectRef` |Optional. The object reference this request is targeted at. This does not apply for List-type requests, or non-resource requests.
|
||||
|`responseStatus` |Optional. The response status, populated even when the ResponseObject is not a Status type. For successful responses, this will only include the Code. For non-status type error responses, this will be auto-populated with the error Message.
|
||||
|`requestObject` |Optional. The API object from the request, in JSON format. The RequestObject is recorded as is in the request (possibly re-encoded as JSON), prior to version conversion, defaulting, admission or merging. It is an external versioned object type, and might not be a valid object on its own. This is omitted for non-resource requests and is only logged at Request Level and higher.
|
||||
|`responseObject` |Optional. The API object returned in the response, in JSON format. The ResponseObject is recorded after conversion to the external type, and serialized as JSON. This is omitted for non-resource requests and is only logged at Response Level.
|
||||
|`requestReceivedTimestamp` |The time that the request reached the API server.
|
||||
|`stageTimestamp` |The time that the request reached the current audit stage.
|
||||
|`annotations` |Optional. An unstructured key value map stored with an audit event that may be set by plugins invoked in the request serving chain, including authentication, authorization and admission plugins. Note that these annotations are for the audit event, and do not correspond to the `metadata.annotations` of the submitted object. Keys should uniquely identify the informing component to avoid name collisions, for example `podsecuritypolicy.admission.k8s.io/policy`. Values should be short. Annotations are included in the Metadata level.
|
||||
|===
|
||||
|
||||
. The response line containing:
|
||||
.. The unique ID from #1
|
||||
.. The response code
|
||||
|
||||
You can view logs for the master nodes for the {product-title} API server or the Kubernetes API server.
|
||||
|
||||
Example output for the Kubelet API server:
|
||||
Example output for the Kubernetes API server:
|
||||
|
||||
----
|
||||
ip-10-0-140-97.ec2.internal {"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2019-04-09T19:56:58Z"},"level":"Metadata","timestamp":"2019-04-09T19:56:58Z","auditID":"6e96c88b-ab6f-44d2-b62e-d1413efd676b","stage":"ResponseComplete","requestURI":"/api/v1/nodes/audit-2019-04-09T14-07-27.129.log","verb":"get","user":{"username":"kube:admin","groups":["system:cluster-admins","system:authenticated"],"extra":{"scopes.authorization.openshift.io":["user:full"]}},"sourceIPs":["10.0.57.93"],"userAgent":"oc/v1.14.6+c4799753c (linux/amd64) kubernetes/ba88cb2","objectRef":{"resource":"nodes","name":"audit-2019-04-09T14-07-27.129.log","apiVersion":"v1"},"responseStatus":{"metadata":{},"status":"Failure","reason":"NotFound","code":404},"requestReceivedTimestamp":"2019-04-09T19:56:58.982157Z","stageTimestamp":"2019-04-09T19:56:58.985300Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"cluster-admins\" of ClusterRole \"cluster-admin\" to Group \"system:cluster-admins\""}}
|
||||
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"ad209ce1-fec7-4130-8192-c4cc63f1d8cd","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/openshift-kube-controller-manager/configmaps/cert-recovery-controller-lock?timeout=35s","verb":"update","user":{"username":"system:serviceaccount:openshift-kube-controller-manager:localhost-recovery-client","uid":"dd4997e3-d565-4e37-80f8-7fc122ccd785","groups":["system:serviceaccounts","system:serviceaccounts:openshift-kube-controller-manager","system:authenticated"]},"sourceIPs":["::1"],"userAgent":"cluster-kube-controller-manager-operator/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"configmaps","namespace":"openshift-kube-controller-manager","name":"cert-recovery-controller-lock","uid":"5c57190b-6993-425d-8101-8337e48c7548","apiVersion":"v1","resourceVersion":"574307"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2020-04-02T08:27:20.200962Z","stageTimestamp":"2020-04-02T08:27:20.206710Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"system:openshift:operator:kube-controller-manager-recovery\" of ClusterRole \"cluster-admin\" to ServiceAccount \"localhost-recovery-client/openshift-kube-controller-manager\""}}
|
||||
----
|
||||
|
||||
|
||||
@@ -19,7 +19,5 @@ administrators, or other components of the system.
|
||||
|
||||
include::modules/nodes-nodes-audit-log-basic.adoc[leveloffset=+1]
|
||||
|
||||
include::modules/nodes-nodes-audit-log-basic-config.adoc[leveloffset=+1]
|
||||
|
||||
include::modules/nodes-nodes-audit-log-basic-viewing.adoc[leveloffset=+1]
|
||||
|
||||
|
||||
Reference in New Issue
Block a user