1
0
mirror of https://github.com/appuio/appuio-pruner.git synced 2026-02-05 09:45:36 +01:00

Reimplement the pruner.

This is a complete reimplementation of the APPUiO pruner.

- The custom "cron" reimplementation was dropped in favor of regular
CronJobs
- Images are now also pruned hourly since [0] was fixed over a year ago
and backported back to 3.6

Compatible with OpenShift 3.9 and upwards. Creates 3 CronJobs to clean
up builds, deployments and images hourly.

Resolves: APPU-1308, APPU-1306
Fixes: #1

[0]: https://bugzilla.redhat.com/show_bug.cgi?id=1408676
This commit is contained in:
Manuel Hutter
2018-11-09 16:36:58 +01:00
committed by Manuel Hutter
parent 73fd59d53b
commit c53bf3d02e
10 changed files with 262 additions and 321 deletions

View File

@@ -1,15 +0,0 @@
FROM registry.access.redhat.com/rhel7:latest
LABEL io.k8s.display-name="APPUiO Pruner" \
io.k8s.description="The APPUiO Pruner prunes old builds, deployments and images."
RUN cd /usr/local/bin && \
curl -k -O ${OC_URL:-https://console.appuio.ch/console/extensions/clients/linux/oc} && \
chmod 755 oc
ENV HOME /tmp/
COPY pruner.sh /tmp/
COPY jobs-cleaner.py /tmp/
ENTRYPOINT ["/tmp/pruner.sh"]

View File

@@ -1,54 +1,49 @@
# appuio-pruner
The APPUiO pruner prunes old builds, deployments and registry images from an OpenShift cluster.
# APPUiO Pruner
## Installation
This repository contains an Ansible role to install the **APPUiO object
pruner**, a set of 3 cronjobs which clean up old deployments, builds and images
respectively.
oc new-project appuio-infra
oc new-app https://github.com/appuio/appuio-pruner
If you are running an insecure registry, you need to specify the following environment variable:
## What does it do?
oc new-app https://github.com/appuio/appuio-pruner -e INSECURE_REGISTRY=true
It runs `oc adm prune XXX`.
Give APPUiO Service Account the required permissions:
## Service Account
oc adm policy add-cluster-role-to-user edit system:serviceaccount:appuio-infra:default
oc adm policy add-cluster-role-to-user system:image-pruner system:serviceaccount:appuio-infra:default
The pruner jobs access the OpenShift API using the service account which the job
was started with. The playbook will create the required service account, roles
and role bindings for the jobs to do their job.
## Ansible Role
## Requirements
This repository contains an Ansible role for automatic installation of the APPUiO pruner.
* [OpenShift Container Platform][ocp] 3.9 or later, or
* [OKD] 3.9 or later
### Requirements
## Role variables
One of:
| Name | Default value | Description |
|-------------------------|----------------|-------------|
| appuio_pruner_namespace | `appuio-infra` | Namespace to install the APPUiO pruner into |
| appuio_pruner_image | `"docker.io/appuio/oc:{{ openshift_release }}"` | Image for the pruner job |
| appuio_pruner_schedule | `@hourly` | Schedule in [Cron] format |
* OpenShift Enterprise 3.2
* OpenShift Container Platform 3.3 or later
* OpenShift Origin M5 1.3 or later.
### Role Variables
[ocp]: https://www.openshift.com/
[OKD]: https://www.okd.io/
[Cron]: https://en.wikipedia.org/wiki/Cron
| Name | Default value | Description |
|-----------------|---------------------------------------------------------------|---------------------------------------------------------------------------------------|
| src | *role_src*, https://github.com/appuio/appuio-pruner.git | Source repository of the APPUiO pruner |
| version | *role_version*, master | Version of the pruner to build, i.e. Git ref of repo above |
| deployment_type | *openshift_deployment_type*, openshift-enterprise | OpenShift deployment type (`openshift-enterprise` or `origin`), determines base image |
| oc_url | https://console.appuio.ch/console/extensions/clients/linux/oc | URL of OpenShift client (oc) |
| namespace | appuio-infra | namespace to install pruner into |
| timezone | *appuio_container_timezone*, UTC | Timezone of the container |
In case of multiple default values the first defined value is used.
### Dependencies
## Dependencies
* <https://github.com/appuio/ansible-module-openshift>
### Example Usage
## Example Usage
`playbook.yml`:
```yaml
roles:
- role: appuio-pruner
- role: appuio-pruner
appuio_pruner_namespace: appuio-pruner
```

View File

@@ -1,3 +1,4 @@
---
namespace: appuio-infra
openshift_deployment_type: openshift-enterprise
appuio_pruner_namespace: appuio-infra
appuio_pruner_image: "docker.io/appuio/oc:{{ openshift_release }}"
appuio_pruner_schedule: "@hourly"

193
files/appuio-pruner.yml Normal file
View File

@@ -0,0 +1,193 @@
---
kind: Template
apiVersion: v1
metadata:
name: appuio-pruner
annotations:
description: Install the APPUiO Pruner
version: '2.0.0'
parameters:
- description: Namespace the template is being deployed into
name: NAMESPACE
required: true
- description: Image to use for the pruner jobs
name: IMAGE
required: true
- description: Schedule to use for the cronjobs
name: SCHEDULE
value: "@hourly"
objects:
- apiVersion: v1
kind: ServiceAccount
metadata:
name: pruner
- apiVersion: authorization.openshift.io/v1
kind: ClusterRole
metadata:
annotations:
openshift.io/reconcile-protect: "true"
labels:
app: appuio-pruner
name: appuio-pruner
rules:
- apiGroups:
- build.openshift.io
resources:
- buildconfigs
verbs:
- list
- apiGroups:
- build.openshift.io
resources:
- builds
verbs:
- list
- delete
- apiGroups:
- apps.openshift.io
resources:
- deploymentconfigs
verbs:
- list
- apiGroups: []
resources:
- replicationcontrollers
verbs:
- list
- delete
- apiVersion: authorization.openshift.io/v1
kind: ClusterRoleBinding
metadata:
name: appuio-pruner
roleRef:
name: appuio-pruner
subjects:
- kind: ServiceAccount
name: pruner
namespace: ${NAMESPACE}
userNames:
- 'system:serviceaccount:${NAMESPACE}:pruner'
- apiVersion: authorization.openshift.io/v1
kind: ClusterRoleBinding
metadata:
name: appuio-image-pruner
roleRef:
name: system:image-pruner
subjects:
- kind: ServiceAccount
name: pruner
namespace: ${NAMESPACE}
userNames:
- 'system:serviceaccount:${NAMESPACE}:pruner'
- apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: prune-builds
spec:
schedule: ${SCHEDULE}
failedJobsHistoryLimit: 3
successfulJobsHistoryLimit: 3
concurrencyPolicy: Forbid
startingDeadlineSeconds: 86400
jobTemplate:
spec:
activeDeadlineSeconds: 7200
completions: 1
# Don't retry failed executions
backoffLimit: 0
template:
metadata:
labels:
name: prune-builds
spec:
containers:
- name: pruner
image: ${IMAGE}
imagePullPolicy: IfNotPresent
args:
- /usr/bin/oc
- adm
- prune
- builds
- --orphans
- --confirm
serviceAccountName: pruner
restartPolicy: Never
- apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: prune-deployments
spec:
schedule: ${SCHEDULE}
failedJobsHistoryLimit: 3
successfulJobsHistoryLimit: 3
concurrencyPolicy: Forbid
startingDeadlineSeconds: 86400
jobTemplate:
spec:
activeDeadlineSeconds: 7200
completions: 1
# Don't retry failed executions
backoffLimit: 0
template:
metadata:
labels:
name: prune-deployments
spec:
containers:
- name: pruner
image: ${IMAGE}
imagePullPolicy: IfNotPresent
args:
- /usr/bin/oc
- adm
- prune
- deployments
- --orphans
- --confirm
serviceAccountName: pruner
restartPolicy: Never
- apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: prune-images
spec:
schedule: ${SCHEDULE}
failedJobsHistoryLimit: 3
successfulJobsHistoryLimit: 3
concurrencyPolicy: Forbid
startingDeadlineSeconds: 86400
jobTemplate:
spec:
activeDeadlineSeconds: 7200
completions: 1
# Don't retry failed executions
backoffLimit: 0
template:
metadata:
labels:
name: prune-deployments
spec:
containers:
- name: pruner
image: ${IMAGE}
imagePullPolicy: IfNotPresent
args:
- /usr/bin/oc
- adm
- prune
- images
- --confirm
serviceAccountName: pruner
restartPolicy: Never

View File

@@ -1,116 +0,0 @@
kind: Template
apiVersion: v1
metadata:
name: pruner
objects:
- kind: ImageStream
apiVersion: v1
metadata:
name: appuio-pruner-base
spec:
tags:
- name: latest
from:
kind: DockerImage
name: ${BASE_IMAGE}
importPolicy:
scheduled: true
- kind: ImageStream
apiVersion: v1
metadata:
name: appuio-pruner
creationTimestamp: null
spec:
dockerImageRepository: null
status:
dockerImageRepository: ''
- kind: BuildConfig
apiVersion: v1
metadata:
name: appuio-pruner
creationTimestamp: null
spec:
triggers:
- type: ConfigChange
- type: ImageChange
imageChange: {}
source:
type: Git
git:
uri: '${PRUNER_SOURCE}'
ref: '${PRUNER_VERSION}'
secrets: []
strategy:
type: Docker
dockerStrategy:
from:
kind: ImageStreamTag
name: appuio-pruner-base:latest
env:
- name: OC_URL
value: ${OC_URL}
output:
to:
kind: ImageStreamTag
name: 'appuio-pruner:latest'
resources: {}
postCommit: {}
status:
lastVersion: 0
- kind: DeploymentConfig
apiVersion: v1
metadata:
name: appuio-pruner
creationTimestamp: null
spec:
strategy:
resources: {}
triggers:
- type: ConfigChange
- type: ImageChange
imageChangeParams:
automatic: true
containerNames:
- appuio-pruner
from:
kind: ImageStreamTag
name: 'appuio-pruner:latest'
replicas: 1
test: false
selector:
deploymentconfig: appuio-pruner
template:
metadata:
creationTimestamp: null
labels:
deploymentconfig: appuio-pruner
annotations:
openshift.io/container.appuio-pruner.image.entrypoint: '["/bin/bash"]'
spec:
containers:
- name: appuio-pruner
image: 'appuio-pruner:latest'
env:
- name: TZ
value: ${TIMEZONE}
parameters:
- description: 'APPUiO pruner Git repository'
name: PRUNER_SOURCE
value: https://github.com/appuio/appuio-pruner.git
required: true
- description: 'APPUiO pruner version, i.e. git ref of the specified repository'
name: PRUNER_VERSION
value: master
required: true
- description: Base image for pruner, registry.access.redhat.com/rhel7 or centos:7, defaults to the former
name: BASE_IMAGE
required: true
value: registry.access.redhat.com/rhel7
- description: 'Timezone (TZ) of the container, see https://en.wikipedia.org/wiki/List_of_tz_database_time_zones for a list'
name: TIMEZONE
value: UTC
required: true
- description: 'URL of oc client'
name: OC_URL
value: https://console.appuio.ch/console/extensions/clients/linux/oc
required: true

View File

@@ -1,87 +0,0 @@
#!/usr/bin/python
import os
import subprocess
import json
from StringIO import StringIO
class JobsCleaner:
def __init__(self, keep_number_of_jobs):
if not isinstance(keep_number_of_jobs, int):
raise TypeError("keep_number_of_jobs must be an integer")
self.keep_number_of_jobs = keep_number_of_jobs
p = subprocess.Popen(['oc', 'project', 'default'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
(output, err) = p.communicate()
if err:
print err
def filter_newest_jobs(self, jobs_dict):
for i in range(self.keep_number_of_jobs):
max_val = max(jobs_dict.values())
max_val_key = [key for key, val in jobs_dict.iteritems() if val == max_val]
del jobs_dict[str(max_val_key[0])]
return jobs_dict
def delete_jobs(self, jobs_dict, project):
for job_name in jobs_dict.iterkeys():
p = subprocess.Popen(['oc', 'delete', 'job', str(job_name), '-n', project], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
(output, err) = p.communicate()
if err:
print err
else:
print output
# wrapper method
def clean_jobs(self):
# get all scheduledJobs
p = subprocess.Popen(['oc', 'get', '--all-namespaces', 'scheduledjob', '-o', 'json'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
(output, err) = p.communicate()
scheduled_jobs = json.load(StringIO(output))
# get all jobs
p = subprocess.Popen(['oc', 'get', '--all-namespaces', 'job', '-o', 'json'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
(output, err) = p.communicate()
jobs = json.load(StringIO(output))
for sj in scheduled_jobs['items']:
scheduled_job_name = sj['metadata']['name']
scheduled_job_project = sj['metadata']['namespace']
succeeded_jobs = dict()
failed_jobs = dict()
for j in jobs['items']:
if scheduled_job_project in j['metadata']['namespace']:
if "\"" + scheduled_job_name + "\"" in j['metadata']['annotations'].get('kubernetes.io/created-by', '') and "completionTime" in j['status']:
job_name = j['metadata']['name']
job_completion_time = j['status']['completionTime']
job_succeeded = j['status']['succeeded']
if job_succeeded == 1:
succeeded_jobs[job_name] = job_completion_time
else:
failed_jobs[job_name] = job_completion_time
if len(succeeded_jobs) > self.keep_number_of_jobs:
succeeded_jobs_filtered = self.filter_newest_jobs(succeeded_jobs)
self.delete_jobs(succeeded_jobs_filtered, scheduled_job_project)
if len(failed_jobs) > self.keep_number_of_jobs:
failed_jobs_filtered = self.filter_newest_jobs(failed_jobs)
self.delete_jobs(failed_jobs_filtered, scheduled_job_project)
def main():
cleaner = JobsCleaner(int(os.getenv('JOBS_NUMBER_TO_KEEP', '3')))
cleaner.clean_jobs()
if __name__ == "__main__": main()

View File

@@ -1,16 +1,16 @@
---
galaxy_info:
author: APPUiO Team
description: Role installing APPUiO pruner
description: Role installing the APPUiO Pruner
company: Puzzle ITC and VSHN
license: Apache License, Version 2.0
min_ansible_version: 2.2
min_ansible_version: 2.4
platforms:
- name: EL
versions:
- 7
- name: EL
versions:
- 7
categories:
- cloud
- cloud
dependencies:
- src: git+https://github.com/appuio/ansible-module-openshift.git
version: v1.3.1
- src: git+https://github.com/appuio/ansible-module-openshift.git
version: v1.4.3

6
playbook.yml Normal file
View File

@@ -0,0 +1,6 @@
---
- hosts: localhost
connection: local
gather_facts: false
roles:
- appuio-pruner

View File

@@ -1,42 +0,0 @@
#!/bin/bash
while true; do
date
echo
echo "Pruning old deployments"
oc adm prune deployments --orphans --confirm
echo
echo "Pruning old builds"
oc adm prune builds --orphans --confirm
echo
# Only prune images at 3:15 in the morning to prevent
# triggering bug deleting blobs which are pushed during prune.
#
# https://bugzilla.redhat.com/show_bug.cgi?id=1408676
if [ $(date +%H) -eq 3 ]; then
# Check if insecure registry variable is set
if [ "$INSECURE_REGISTRY" = true ]; then
echo "Pruning old images from insecure registry"
oc adm prune images --confirm --force-insecure
else
echo "Pruning old images from registry"
oc adm prune images --confirm
fi
echo
fi
export JOBS_NUMBER_TO_KEEP=3
echo "Cleaning up jobs"
python /tmp/jobs-cleaner.py
echo "----------------------------------------"
# Wait till it's 15 past for the next time
SLEEP_MINUTES=$(( ( 75 - $(date +%-M) ) % 60 ))
if [ ${SLEEP_MINUTES} -eq 0 ]; then SLEEP_MINUTES=60; fi
sleep ${SLEEP_MINUTES}m
done

View File

@@ -1,23 +1,29 @@
---
- name: Create project for pruner
- name: Create project for APPUiO Pruner
openshift_project:
name: "{{ namespace }}"
name: "{{ appuio_pruner_namespace }}"
- name: Instantiate pruner template
openshift_resource:
namespace: "{{ namespace }}"
template: "{{ role_path }}/files/pruner-template.yml"
app_name: appuio-pruner
arguments:
PRUNER_SOURCE: "{{ src | default(role_src) | default('https://github.com/appuio/appuio-pruner.git') }}"
PRUNER_VERSION: "{{ version | default(role_version) | default('master') }}"
BASE_IMAGE: "{{ ((deployment_type | default(openshift_deployment_type)) == 'origin') | ternary('centos:7', 'registry.access.redhat.com/rhel7') }}"
TIMEZONE: "{{ timezone | default(appuio_container_timezone) | default('UTC') }}"
OC_URL: "{{ oc_url | default('https://console.appuio.ch/console/extensions/clients/linux/oc') }}"
- name: Create temp directory for doing work in
tempfile:
state: directory
register: tempdir
changed_when: false
- name: Configure permissions for pruner service account
openshift_policy:
cluster_roles:
- edit
- system:image-pruner
users: "system:serviceaccount:{{ namespace }}:default"
- vars:
r_template: "{{ tempdir.path }}/appuio-pruner.yml"
block:
- name: Copy template
copy:
src: appuio-pruner.yml
dest: "{{ r_template }}"
changed_when: false
- name: Apply pruner template
openshift_resource:
namespace: "{{ appuio_pruner_namespace }}"
template: "{{ r_template }}"
app_name: appuio-pruner
arguments:
NAMESPACE: "{{ appuio_pruner_namespace }}"
IMAGE: "{{ appuio_pruner_image }}"
SCHEDULE: "{{ appuio_pruner_schedule }}"