mirror of
https://github.com/openshift/installer.git
synced 2026-02-05 06:46:36 +01:00
d63debee34
* add "pre-command start", "pre-command end", "post-command start" and "post-command end" phases * fixed issue where the kubelet was not notifying systemd that it had started since it had been moved to a script
95 lines
4.5 KiB
Markdown
95 lines
4.5 KiB
Markdown
#### Bootstrap Service Records ####
|
|
|
|
For the purposes of diagnosing installation failures that occur during bootstrapping, the
|
|
progresses of services running on the bootstrap machines are tracked in json files in the
|
|
/var/log/openshift directory. The progress for each service is tracked in its own file. For
|
|
example, the bootkube service progress is tracked in the /var/log/openshift/bootkube.json
|
|
file. The following progress events are tracked.
|
|
* A service adds an entry when the service starts.
|
|
* A service adds an entry when the service ends. The entry includes the result of the service
|
|
invocation, either success or failure. If the invocation failed, then the entry includes
|
|
the line number of the error and the last three lines from the service's journal log.
|
|
* A service adds an entry when a stage of the service starts. An example of a service stage
|
|
is the cvo-bootstrap stage of the bootkube service. During that stage, the
|
|
cluster-version-operator renders its manifests.
|
|
* A service adds an entry when a stage of the service ends. This is similar to the entry
|
|
added when a service ends, including having a result and error information if applicable.
|
|
* A service adds an entry when a pre- or post-command starts.
|
|
* A service adds en entry when a pre- or post-command ends. This is similar to the entry
|
|
added when a service ends, including having a result and error information if applicable.
|
|
|
|
##### Managing Service Records in a Service #####
|
|
|
|
To track its progress, a service should source the /usr/local/bin/bootstrap-service-record.sh
|
|
script. When a service sources the script, the script will add an entry to the json file for
|
|
the service indicating that the service started. When the service ends, either successfully
|
|
or due to an error, the script will add an entry to the json file for the service indicating that
|
|
the service ended. The script will consider whether the last command executed was successful or
|
|
not in order to determine whether the service was successful.
|
|
|
|
For tracking stages, the service should call functions from the sourced script.
|
|
* The service should call the `record_service_stage_start` function when a stage of the service
|
|
starts. The function takes the name of the stage as its single argument.
|
|
* The service should call the `record_service_stage_success` function when a stage of the service
|
|
ends successfully.
|
|
* The service should call the `record_service_stage_failure` function when a stage of the service
|
|
ends due to a failure. The script will automatically record an entry for a stage failure if the
|
|
service ends during the execution of a stage.
|
|
|
|
###### Pre- and Post-Commands ######
|
|
|
|
If a service has pre- or post-commands that could either run for significant periods or could
|
|
potentially fail, then those commands should add to the json file as well. Such a command should
|
|
source the same /usr/local/bin/bootstrap-service-record.sh script. It should also set either the
|
|
`PRE_COMMAND` or `POST_COMMAND` environment variable with a value that identifies the command.
|
|
For example, kubelet.service has a pre-command of /usr/local/bin/kubelet-pause-image.sh. The
|
|
kubelet-pause-image.sh script sets the `PRE_COMMAND` environment variable to "kubelet-pause-image"
|
|
before sourcing the bootstrap-service-record.sh script. The entries for the pre-command will
|
|
contain a `preCommand` field with the "kubelet-pause-image" value.
|
|
|
|
###### Sample Script #######
|
|
|
|
```shell script
|
|
#!/usr/bin/env bash
|
|
set -euoE pipefail
|
|
|
|
# Source the script to record service entries.
|
|
# This will create en entry for the start of the service.
|
|
. /usr/local/bin/bootstrap-service-record.sh
|
|
|
|
# Record the start of the "first" stage.
|
|
record_service_stage_start "first"
|
|
|
|
# Record the successful end of the "first" stage.
|
|
record_service_stage_success
|
|
|
|
while true
|
|
do
|
|
# Record the start of the "second" stage.
|
|
record_service_stage_start "second-stage"
|
|
|
|
if [ some_check ]
|
|
then
|
|
# Record the successful end of the "second" stage.
|
|
record_service_stage_success
|
|
break
|
|
else
|
|
# Record the failing end of the "second" stage.
|
|
record_service_stage_failure
|
|
fi
|
|
done
|
|
|
|
# Record the start of the third stage.
|
|
record_service_stage_start "third"
|
|
|
|
# If the command fails, then an entry will be recorded for the failing end
|
|
# of the third stage and an entry will be recorded for the failing end of
|
|
# the service.
|
|
some_command_that_may_fail
|
|
|
|
# Record the end of the third stage.
|
|
record_service_stage_success
|
|
|
|
# Since this is the end of the script, an entry will be recorded for the
|
|
# successful end of the service.
|
|
``` |