openshift-docs/modules/gathering-application-diagnostic-data.adoc

// Module included in the following assemblies:
//
// * support/troubleshooting/troubleshooting-s2i.adoc

:_mod-docs-content-type: PROCEDURE
[id="gathering-application-diagnostic-data_{context}"]
= Gathering application diagnostic data to investigate application failures

Application failures can occur within running application pods. In these situations, you can retrieve diagnostic information with these strategies:

* Review events relating to the application pods.
* Review the logs from the application pods, including application-specific log files that are not collected by the OpenShift Logging framework.
* Test application functionality interactively and run diagnostic tools in an application container.

.Prerequisites

ifndef::openshift-rosa,openshift-dedicated[]
* You have access to the cluster as a user with the `cluster-admin` role.
endif::openshift-rosa,openshift-dedicated[]
ifdef::openshift-rosa,openshift-dedicated[]
* You have access to the cluster as a user with the `dedicated-admin` role.
endif::openshift-rosa,openshift-dedicated[]
* You have installed the OpenShift CLI (`oc`).

.Procedure

. List events relating to a specific application pod. The following example retrieves events for an application pod named `my-app-1-akdlg`:
+
[source,terminal]
----
$ oc describe pod/my-app-1-akdlg
----

. Review logs from an application pod:
+
[source,terminal]
----
$ oc logs -f pod/my-app-1-akdlg
----

. Query specific logs within a running application pod. Logs that are sent to stdout are collected by the OpenShift Logging framework and are included in the output of the preceding command. The following query is only required for logs that are not sent to stdout.
+
.. If an application log can be accessed without root privileges within a pod, concatenate the log file as follows:
+
[source,terminal]
----
$ oc exec my-app-1-akdlg -- cat /var/log/my-application.log
----
+
.. If root access is required to view an application log, you can start a debug container with root privileges and then view the log file from within the container. Start the debug container from the project's `DeploymentConfig` object. Pod users typically run with non-root privileges, but running troubleshooting pods with temporary root privileges can be useful during issue investigation:
+
[source,terminal]
----
$ oc debug dc/my-deployment-configuration --as-root -- cat /var/log/my-application.log
----
+
[NOTE]
====
You can access an interactive shell with root access within the debug pod if you run `oc debug dc/<deployment_configuration> --as-root` without appending `-- <command>`.
====

. Test application functionality interactively and run diagnostic tools, in an application container with an interactive shell.
.. Start an interactive shell on the application container:
+
[source,terminal]
----
$ oc exec -it my-app-1-akdlg /bin/bash
----
+
.. Test application functionality interactively from within the shell. For example, you can run the container's entry point command and observe the results. Then, test changes from the command line directly, before updating the source code and rebuilding the application container through the S2I process.
+
.. Run diagnostic binaries available within the container.
+
[NOTE]
====
Root privileges are required to run some diagnostic binaries. In these situations you can start a debug pod with root access, based on a problematic pod's `DeploymentConfig` object, by running `oc debug dc/<deployment_configuration> --as-root`. Then, you can run diagnostic binaries as root from within the debug pod.
====

ifndef::openshift-rosa,openshift-dedicated[]
. If diagnostic binaries are not available within a container, you can run a host's diagnostic binaries within a container's namespace by using `nsenter`. The following example runs `ip ad` within a container's namespace, using the host`s `ip` binary.
// cannot create resource "namespaces" in API group
.. Enter into a debug session on the target node. This step instantiates a debug pod called `<node_name>-debug`:
+
[source,terminal]
----
$ oc debug node/my-cluster-node
----
+
.. Set `/host` as the root directory within the debug shell. The debug pod mounts the host's root file system in `/host` within the pod. By changing the root directory to `/host`, you can run binaries contained in the host's executable paths:
+
[source,terminal]
----
# chroot /host
----
+
[NOTE]
====
{product-title} {product-version} cluster nodes running {op-system-first} are immutable and rely on Operators to apply cluster changes. Accessing cluster nodes by using SSH is not recommended. However, if the {product-title} API is not available, or the kubelet is not properly functioning on the target node, `oc` operations will be impacted. In such situations, it is possible to access nodes using `ssh core@<node>.<cluster_name>.<base_domain>` instead.
====
+
.. Determine the target container ID:
+
[source,terminal]
----
# crictl ps
----
+
.. Determine the container's process ID. In this example, the target container ID is `a7fe32346b120`:
+
[source,terminal]
----
# crictl inspect a7fe32346b120 --output yaml | grep 'pid:' | awk '{print $2}'
----
+
.. Run `ip ad` within the container's namespace, using the host's `ip` binary. This example uses `31150` as the container's process ID. The `nsenter` command enters the namespace of a target process and runs a command in its namespace. Because the target process in this example is a container's process ID, the `ip ad` command is run in the container's namespace from the host:
+
[source,terminal]
----
# nsenter -n -t 31150 -- ip ad
----
+
[NOTE]
====
Running a host's diagnostic binaries within a container's namespace is only possible if you are using a privileged container such as a debug node.
====
endif::openshift-rosa,openshift-dedicated[]