mirror of
https://github.com/openshift/openshift-docs.git
synced 2026-02-05 12:46:18 +01:00
Add new Operator / OLM topics
This commit is contained in:
@@ -190,6 +190,12 @@ Name: Operators
|
||||
Dir: operators
|
||||
Distros: openshift-*
|
||||
Topics:
|
||||
- Name: What are Operators?
|
||||
File: olm-what-are-operators
|
||||
- Name: Adding Operators to your cluster
|
||||
File: olm-adding-operators-to-cluster
|
||||
- Name: Creating applications from installed Operators
|
||||
File: olm-creating-apps-from-installed-operators
|
||||
- Name: Getting started with the Operator SDK
|
||||
File: osdk-getting-started
|
||||
- Name: Operators based on Helm charts
|
||||
|
||||
BIN
images/etcd-operator-overview.png
Normal file
BIN
images/etcd-operator-overview.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 83 KiB |
BIN
images/etcd-operator-resources.png
Normal file
BIN
images/etcd-operator-resources.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 59 KiB |
BIN
images/olm-catalog-sources.png
Normal file
BIN
images/olm-catalog-sources.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 100 KiB |
BIN
images/olm-workflow.png
Normal file
BIN
images/olm-workflow.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 28 KiB |
BIN
images/osdk-workflow.png
Normal file
BIN
images/osdk-workflow.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 34 KiB |
@@ -37,7 +37,7 @@ $ cd memcached-operator
|
||||
+
|
||||
[TIP]
|
||||
====
|
||||
See xref:operators-appendices.adoc#operator-project-scaffolding-layout_operator-appendices[Appendices] to
|
||||
See xref:operators-appendices.adoc#olm-operator-project-scaffolding-layout_operator-appendices[Appendices] to
|
||||
learn about the project directory structure created by the previous commands.
|
||||
====
|
||||
|
||||
|
||||
@@ -11,9 +11,9 @@ powered by a Helm chart using tools and libraries provided by the Operator SDK.
|
||||
[TIP]
|
||||
====
|
||||
It is best practice to build a new Operator for each chart. This can allow for
|
||||
more native-behaving Kubernetes APIs (e.g., `oc get Nginx`) and flexibility if
|
||||
you ever want to write a fully-fledged Operator in Go, migrating away from a
|
||||
Helm-based Operator.
|
||||
more native-behaving Kubernetes APIs (for example, `oc get Nginx`) and
|
||||
flexibility if you ever want to write a fully-fledged Operator in Go, migrating
|
||||
away from a Helm-based Operator.
|
||||
====
|
||||
|
||||
.Prerequisites
|
||||
@@ -40,7 +40,7 @@ $ cd nginx-operator
|
||||
[TIP]
|
||||
====
|
||||
See
|
||||
xref:operators-appendices.adoc#operator-project-scaffolding-layout_operator-appendices[Appendices]
|
||||
xref:operators-appendices.adoc#olm-operator-project-scaffolding-layout_operator-appendices[Appendices]
|
||||
to learn about the project directory structure created by the previous commands.
|
||||
====
|
||||
+
|
||||
@@ -92,8 +92,8 @@ logic for each Nginx Custom Resource (CR):
|
||||
* Create a Nginx Service if it does not exist.
|
||||
* Create a Nginx Ingress if it is enabled and does not exist.
|
||||
* Ensure that the Deployment, Service, and optional Ingress match the desired
|
||||
configuration (e.g., replica count, image, service type) as specified by the
|
||||
Nginx CR.
|
||||
configuration (for example, replica count, image, service type) as specified by
|
||||
the Nginx CR.
|
||||
--
|
||||
+
|
||||
By default, the `nginx-operator` watches `Nginx` resource events as shown in the
|
||||
|
||||
77
modules/olm-creating-etcd-cluster-from-operator.adoc
Normal file
77
modules/olm-creating-etcd-cluster-from-operator.adoc
Normal file
@@ -0,0 +1,77 @@
|
||||
// Module included in the following assemblies:
|
||||
//
|
||||
// * operators/olm-creating-apps-from-installed-operators.adoc
|
||||
|
||||
[id='olm-creating-etcd-cluster-from-operator_{context}']
|
||||
= Creating an etcd cluster using an Operator
|
||||
|
||||
This procedure walks through creating a new etcd cluster using the etcd
|
||||
Operator, managed by the Operator Lifecycle Manager (OLM).
|
||||
|
||||
.Prerequisites
|
||||
|
||||
- Access to a {product-title} 4.0 cluster
|
||||
- etcd Operator already installed to the cluster by an administrator
|
||||
|
||||
.Procedure
|
||||
|
||||
. Create a new project in the {product-title} web console for this procedure. This
|
||||
example uses a project called *my-etcd*.
|
||||
|
||||
. Navigate to the *Catalogs -> Installed Operators* page. The Operators that have
|
||||
been installed to the cluster by the cluster administrator and are available for
|
||||
use are shown here as a list of ClusterServiceVersions (CSVs). CSVs are used to
|
||||
launch and manage the software provided by the Operator.
|
||||
+
|
||||
[TIP]
|
||||
====
|
||||
You can get this list from the CLI using:
|
||||
|
||||
----
|
||||
$ oc get csv
|
||||
----
|
||||
====
|
||||
|
||||
. On the *Installed Operators* page, click on the etcd Operator to view more
|
||||
details and available actions:
|
||||
+
|
||||
.etcd Operator overview
|
||||
image::etcd-operator-overview.png[]
|
||||
+
|
||||
As shown under *Provided APIs*, this Operator makes available three new resource
|
||||
types, including one for an *etcd Cluster* (the `EtcdCluster` resource). These
|
||||
objects work similar to the built-in native Kubernetes ones, such as
|
||||
`Deployments` or `ReplicaSets`, but contain logic specific to managing etcd.
|
||||
|
||||
. Create a new etcd cluster:
|
||||
|
||||
.. In the *etcd Cluster* API box, click *Create New*.
|
||||
|
||||
.. The next screen allows you to make any modifications to the minimal starting
|
||||
template of an `EtcdCluster` object, such as the size of the cluster. For now,
|
||||
click *Create* to finalize. This triggers the Operator to start up the Pods,
|
||||
Services, and other components of the new etcd cluster.
|
||||
|
||||
. Click the *Resources* tab to see that your project now contains a number of
|
||||
resources created and configured automatically by the Operator.
|
||||
+
|
||||
.etcd Operator resources
|
||||
image::etcd-operator-resources.png[]
|
||||
+
|
||||
Verify that a Kubernetes service has been created that allows you to access the
|
||||
database from other Pods in your project.
|
||||
|
||||
. All users with the `edit` role in a given project can create, manage, and delete
|
||||
application instances (an etcd cluster, in this example) managed by Operators
|
||||
that have already been created in the project, in a self-service manner, just
|
||||
like a cloud service. If you want to enable additional users with this ability,
|
||||
project administrators can add the role using the following command:
|
||||
+
|
||||
----
|
||||
$ oc policy add-role-to-user edit <user> -n <target_project>
|
||||
----
|
||||
|
||||
You now have an etcd cluster that will react to failures and rebalance data as
|
||||
Pods become unhealthy or are migrated between nodes in the cluster. Most
|
||||
importantly, cluster administrators or developers with proper access can now
|
||||
easily use the database with their applications.
|
||||
42
modules/olm-installing-operator-using-operatorhub.adoc
Normal file
42
modules/olm-installing-operator-using-operatorhub.adoc
Normal file
@@ -0,0 +1,42 @@
|
||||
// Module included in the following assemblies:
|
||||
//
|
||||
// * operators/olm-adding-operators-to-cluster.adoc
|
||||
|
||||
[id='olm-installing-operator-using-operatorhub_{context}']
|
||||
= Installing an Operator using the OperatorHub
|
||||
|
||||
As a cluster administrator, you can install an Operator using the OperatorHub in
|
||||
the {product-title} web console to make it available for all developers using
|
||||
your cluster. This procedure uses the etcd Operator as an example.
|
||||
|
||||
.Prerequisites
|
||||
|
||||
- Access to an {product-title} v4 cluster using an account with `cluster-admin`
|
||||
permissions.
|
||||
|
||||
.Procedure
|
||||
|
||||
. Navigate in the web console to the *Catalog → Operator Hub* page.
|
||||
|
||||
. Click *Show Community Operators*.
|
||||
+
|
||||
[NOTE]
|
||||
====
|
||||
This Operator will be available under the *Red Hat* category in the GA release.
|
||||
====
|
||||
|
||||
. Choose *etcd* from the list of available Operators, and click *Install*.
|
||||
|
||||
. On the *Create Operator Subscription* page, change the *Target* to the
|
||||
*global-operators* Operator Group. This makes the Operator available to all
|
||||
users and projects that use this {product-title} cluster.
|
||||
|
||||
. On the *Catalog → Installed Operators* page, verify that the *etcd*
|
||||
ClusterServiceVersion (CSV) eventually shows up and its *Status* ultimately
|
||||
resolves to *InstallSucceeded*.
|
||||
+
|
||||
If it does not, switch to the *Catalog → Operator Management* page and inspect
|
||||
the *Operator Subscriptions* and *Install Plans* tabs for any failure or errors
|
||||
under *Status*. Then, check the logs in any Pods in the *openshift-operators*
|
||||
project (on the *Workloads → Pods* page) that are reporting issues to
|
||||
troubleshoot further.
|
||||
31
modules/olm-operator-framework.adoc
Normal file
31
modules/olm-operator-framework.adoc
Normal file
@@ -0,0 +1,31 @@
|
||||
// Module included in the following assemblies:
|
||||
//
|
||||
// * operators/what-are-operators.adoc
|
||||
|
||||
[id='olm-operator-framework-{context}']
|
||||
= Operator Framework
|
||||
|
||||
The Operator Framework is a family of tools and capabilities to deliver on the
|
||||
customer experience describe above. It is not just about writing code; testing,
|
||||
delivering, and updating Operators is just as important. The Operator Framework
|
||||
components consiss of open source tools to tackle these problems:
|
||||
|
||||
Operator SDK::
|
||||
Assists Operator authors in bootstrapping, building, testing, and packaging
|
||||
their own Operator based on their expertise without requiring knowledge of
|
||||
Kubernetes API complexities.
|
||||
|
||||
Operator Lifecycle Manager::
|
||||
Controls the installation, upgrade, and RBAC of Operators in a cluster. Deployed
|
||||
by default in {product-title} 4.
|
||||
|
||||
Operator Metering::
|
||||
Collects operational metrics about Operators on the cluster for Day 2 management
|
||||
and aggregating usage metrics.
|
||||
|
||||
Operator Hub::
|
||||
Web console for discovering and installing Operators on your cluster. Deployed
|
||||
by default in {product-title} 4.
|
||||
|
||||
These tools are designed to be composable, so you can use any that are useful to
|
||||
you.
|
||||
27
modules/olm-operator-lifecycle-manager.adoc
Normal file
27
modules/olm-operator-lifecycle-manager.adoc
Normal file
@@ -0,0 +1,27 @@
|
||||
// Module included in the following assemblies:
|
||||
//
|
||||
// * operators/olm-adding-operators-to-cluster.adoc
|
||||
|
||||
[id='olm-operator-lifecycle-manager-{context}']
|
||||
= Understanding the Operator Lifecycle Manager
|
||||
|
||||
In {product-title} 4.0, the _Operator Lifecycle Manager_ (OLM) helps users
|
||||
install, update, and manage the lifecycle of all Operators and their associated
|
||||
services running across their clusters. It is part of the
|
||||
link:https://coreos.com/blog/introducing-operator-framework[Operator Framework],
|
||||
an open source toolkit designed to manage Kubernetes native applications
|
||||
(Operators) in an effective, automated, and scalable way.
|
||||
|
||||
.Operator Lifecycle Manager workflow
|
||||
image::olm-workflow.png[]
|
||||
|
||||
The OLM runs by default in {product-title} 4.0, which aids cluster
|
||||
administrators in installing, upgrading, and granting access to Operators
|
||||
running on their cluster. The {product-title} web console provides management
|
||||
screens for cluster administrators to install Operators, as well as grant
|
||||
specific projects access to use the catalog of Operators available on the
|
||||
cluster.
|
||||
|
||||
For developers, a self-service experience allows provisioning and configuring
|
||||
instances of databases, monitoring, and big data services without having to be
|
||||
subject matter experts, because the Operator has that knowledge baked into it.
|
||||
@@ -2,7 +2,7 @@
|
||||
//
|
||||
// * operators/operators-appendices.adoc
|
||||
|
||||
[id='operator-project-scaffolding-layout_{context}']
|
||||
[id='olm-operator-project-scaffolding-layout_{context}']
|
||||
= Operator project scaffolding layout
|
||||
|
||||
The `operator-sdk` CLI generates a number of packages for each Operator project.
|
||||
28
modules/olm-operatorhub.adoc
Normal file
28
modules/olm-operatorhub.adoc
Normal file
@@ -0,0 +1,28 @@
|
||||
// Module included in the following assemblies:
|
||||
//
|
||||
// * operators/olm-adding-operators-to-cluster.adoc
|
||||
|
||||
[id='olm-operatorhub-{context}']
|
||||
= Understanding the OperatorHub
|
||||
|
||||
The _OperatorHub_ is available via the {product-title} web console and is the
|
||||
interface that cluster administrators use to discover and install Operators.
|
||||
With one click, an Operator can be pulled from their off-cluster source,
|
||||
installed and subscribed on the cluster, and made ready for engineering teams to
|
||||
self-service manage the product across deployment environments using the
|
||||
Operator Lifecycle Manager (OLM).
|
||||
|
||||
Cluster administrators can choose from catalog sources of Operators grouped into
|
||||
the following categories:
|
||||
|
||||
Red Hat Operators::
|
||||
Red Hat products packaged and shipped by Red Hat. Supported by Red Hat.
|
||||
|
||||
Certified Operators::
|
||||
Products from leading independent software vendors (ISVs). Red Hat partners with
|
||||
ISVs to package and ship. Supported by the ISV.
|
||||
|
||||
Community Operators::
|
||||
Optionally-visible software maintained by relevant representatives in the
|
||||
link:https://github.com/operator-framework/community-operators[operator-framework/community-operators]
|
||||
GitHub repository. No official support.
|
||||
38
modules/olm-why-use-operators.adoc
Normal file
38
modules/olm-why-use-operators.adoc
Normal file
@@ -0,0 +1,38 @@
|
||||
// Module included in the following assemblies:
|
||||
//
|
||||
// * operators/what-are-operators.adoc
|
||||
|
||||
[id='olm-why-use-operators-{context}']
|
||||
= Why use Operators?
|
||||
|
||||
Operators provide:
|
||||
|
||||
--
|
||||
- Repeatability of installation and upgrade.
|
||||
- Constant health checks of every system component.
|
||||
- Over-the-air (OTA) updates for OpenShift components and ISV content.
|
||||
- A place to encapsulate knowledge from field engineers and spread it to all
|
||||
users, not just one or two.
|
||||
--
|
||||
|
||||
Why deploy on Kubernetes?::
|
||||
Kubernetes (and by extension, {product-title}) contains all of the primitives
|
||||
needed to build complex distributed systems – secret handling, load balancing,
|
||||
service discovery, autoscaling – that work across on-premise and cloud
|
||||
providers.
|
||||
|
||||
Why manage your app with Kubernetes APIs and `kubectl` tooling?::
|
||||
These APIs are feature rich, have clients for all platforms and plug into the
|
||||
cluster’s access control/auditing. An Operator uses the Kubernetes' extension
|
||||
mechanism, Custom Resource Definitions (CRDs), so your custom object, for
|
||||
example `MongoDB`, looks and acts just like the built-in, native Kubernetes
|
||||
objects.
|
||||
|
||||
How do Operators compare with Service Brokers?::
|
||||
A Service Broker is a step towards programmatic discovery and deployment of an
|
||||
app. However, because it is not a long running process, it cannot execute Day 2
|
||||
operations like upgrade, failover, or scaling. Customizations and
|
||||
parameterization of tunables are provided at install time, versus an Operator
|
||||
that is constantly watching your cluster's current state. Off-cluster services
|
||||
continue to be a good match for a Service Broker, although Operators exist for
|
||||
these as well.
|
||||
@@ -87,7 +87,7 @@ the Operator SDK as a Pod in the cluster.
|
||||
|Location of `kubeconfig` for a cluster. Default: `~/.kube/config`.
|
||||
|
||||
|`--image-pull-policy` (string)
|
||||
|Set test pod image pull policy. Allowed values: `Always` (default), `Never`.
|
||||
|Set test Pod image pull policy. Allowed values: `Always` (default), `Never`.
|
||||
|
||||
|`--namespace` (string)
|
||||
|Namespace to run tests in. Default: `default`.
|
||||
|
||||
@@ -24,10 +24,10 @@ The main function of an Operator is to read from a custom object that represents
|
||||
your application instance and have its desired state match what is running. In
|
||||
the case of a Helm-based Operator, the object's spec field is a list of
|
||||
configuration options that are typically described in Helm's `values.yaml` file.
|
||||
Instead of setting these values with flags using the Helm CLI (e.g., `helm
|
||||
install -f values.yaml`), you can express them within a Custom Resource (CR),
|
||||
which, as a native Kubernetes object, enables the benefits of RBAC applied to it
|
||||
and an audit trail.
|
||||
Instead of setting these values with flags using the Helm CLI (for example, `helm install -f values.yaml`),
|
||||
you can express them within a Custom Resource (CR), which, as a native
|
||||
Kubernetes object, enables the benefits of RBAC applied to it and an audit
|
||||
trail.
|
||||
|
||||
For an example of a simple CR called `Tomcat`:
|
||||
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
//
|
||||
// * operators/osdk-getting-started.adoc
|
||||
|
||||
[id='about-opreator-sdk-{context}']
|
||||
[id='osdk-operator-sdk-{context}']
|
||||
= Understanding the Operator SDK
|
||||
|
||||
The link:https://coreos.com/operators/[Operator Framework] is an open source
|
||||
@@ -35,6 +35,9 @@ The SDK provides the following workflow to develop a new operator:
|
||||
. Define the Operator reconciling logic in a designated handler and use the SDK API to interact with resources.
|
||||
. Use the SDK CLI to build and generate the Operator deployment manifests.
|
||||
|
||||
.Operator SDK workflow
|
||||
image::osdk-workflow.png[]
|
||||
|
||||
At a high level, an Operator using the SDK processes events for watched
|
||||
resources in a user-defined handler and takes actions to reconcile the state of
|
||||
the application.
|
||||
22
operators/olm-adding-operators-to-cluster.adoc
Normal file
22
operators/olm-adding-operators-to-cluster.adoc
Normal file
@@ -0,0 +1,22 @@
|
||||
[id='olm-adding-operators-to-a-cluster']
|
||||
= Adding Operators to a cluster
|
||||
{product-author}
|
||||
{product-version}
|
||||
:data-uri:
|
||||
:icons:
|
||||
:experimental:
|
||||
:toc: macro
|
||||
:toc-title:
|
||||
:prewrap!:
|
||||
:context: olm-adding-operators-to-a-cluster
|
||||
|
||||
toc::[]
|
||||
|
||||
{nbsp} +
|
||||
This guide outlines the basics of the Operator Lifecycle Manager (OLM) and walks
|
||||
through an example of installing and subscribing a cluster to Operators from the
|
||||
OperatorHub.
|
||||
|
||||
include::modules/olm-operator-lifecycle-manager.adoc[leveloffset=+1]
|
||||
include::modules/olm-operatorhub.adoc[leveloffset=+1]
|
||||
include::modules/olm-installing-operator-using-operatorhub.adoc[leveloffset=+1]
|
||||
19
operators/olm-creating-apps-from-installed-operators.adoc
Normal file
19
operators/olm-creating-apps-from-installed-operators.adoc
Normal file
@@ -0,0 +1,19 @@
|
||||
[id='olm-creating-apps-from-installed-operators']
|
||||
= Creating applications from installed Operators
|
||||
{product-author}
|
||||
{product-version}
|
||||
:data-uri:
|
||||
:icons:
|
||||
:experimental:
|
||||
:toc: macro
|
||||
:toc-title:
|
||||
:prewrap!:
|
||||
:context: olm-creating-apps-from-installed-operators
|
||||
|
||||
toc::[]
|
||||
|
||||
{nbsp} +
|
||||
This guide walks through an example of creating applications from an installed
|
||||
Operator using the {product-title} 4.0 web console.
|
||||
|
||||
include::modules/olm-creating-etcd-cluster-from-operator.adoc[leveloffset=+1]
|
||||
39
operators/olm-what-are-operators.adoc
Normal file
39
operators/olm-what-are-operators.adoc
Normal file
@@ -0,0 +1,39 @@
|
||||
[id='olm-what-are-operators']
|
||||
= What are Operators?
|
||||
{product-author}
|
||||
{product-version}
|
||||
:data-uri:
|
||||
:icons:
|
||||
:experimental:
|
||||
:toc: macro
|
||||
:toc-title:
|
||||
:prewrap!:
|
||||
:context: olm-what-are-operators
|
||||
|
||||
toc::[]
|
||||
|
||||
{nbsp} +
|
||||
[.lead]
|
||||
Conceptually, _Operators_ take human operational knowledge and encode it into
|
||||
software that is more easily shared with consumers.
|
||||
|
||||
Operators are pieces of software that ease the operational complexity of running
|
||||
another piece of software. They act like an extension of the software vendor's
|
||||
engineering team, watching over a Kubernetes environment (such as
|
||||
{product-title}) and using its current state to make decisions in real time.
|
||||
Advanced Operators are designed to handle upgrades seamlessly, react to failures
|
||||
automatically, and not take shortcuts, like skipping a software backup process
|
||||
to save time.
|
||||
|
||||
[.lead]
|
||||
More technically, _Operators_ are a method of packaging, deploying, and managing a
|
||||
Kubernetes application.
|
||||
|
||||
A Kubernetes application is an app that is both deployed on Kubernetes and
|
||||
managed using the Kubernetes APIs and `kubectl` or `oc` tooling. To be able to
|
||||
make the most of Kubernetes, you need a set of cohesive APIs to extend in order
|
||||
to service and manage your apps that run on Kubernetes. Think of
|
||||
Operators as the runtime that manages this type of app on Kubernetes.
|
||||
|
||||
include::modules/olm-why-use-operators.adoc[leveloffset=+1]
|
||||
include::modules/olm-operator-framework.adoc[leveloffset=+1]
|
||||
@@ -12,4 +12,4 @@
|
||||
|
||||
toc::[]
|
||||
|
||||
include::modules/operator-project-staffolding-layout.adoc[leveloffset=+2]
|
||||
include::modules/olm-operator-project-staffolding-layout.adoc[leveloffset=+2]
|
||||
|
||||
@@ -13,23 +13,13 @@
|
||||
toc::[]
|
||||
|
||||
{nbsp} +
|
||||
This guide walks through an example of building a simple Memcached Operator
|
||||
(`memcached-operator`) using the `operator-sdk` CLI tool and
|
||||
`controller-runtime` library API. It also shows how to manage the new Operator's
|
||||
lifecycle from installation through updating to a new version.
|
||||
This guide walks through an example of building a simple Memcached Operator and
|
||||
managing its lifecycle on a Kubernetes-based cluster (such as {product-title})
|
||||
from installation to upgrade. This is accomplished using two centerpieces of the
|
||||
Operator Framework: the Operator SDK (the `operator-sdk` CLI tool and
|
||||
`controller-runtime` library API) and the Operator Lifecycle Manager (OLM).
|
||||
|
||||
As an administrator of a Kubernetes-based cluster, you can accomplish this using
|
||||
two centerpieces of the Operator Framework:
|
||||
|
||||
Operator SDK:: Assists developers in bootstrapping and building an Operator
|
||||
based on their expertise without requiring knowledge of Kubernetes API
|
||||
complexities.
|
||||
|
||||
Operator Lifecycle Manager (OLM):: Helps Operator users install, update, and
|
||||
generally manage the lifecycle of all Operators and their associated services
|
||||
running across their clusters.
|
||||
|
||||
include::modules/about-operator-sdk.adoc[leveloffset=+1]
|
||||
include::modules/osdk-operator-sdk.adoc[leveloffset=+1]
|
||||
include::modules/installing-operator-sdk-cli.adoc[leveloffset=+1]
|
||||
include::modules/building-memcached-operator-using-osdk.adoc[leveloffset=+1]
|
||||
include::modules/managing-memcached-operator-using-olm.adoc[leveloffset=+1]
|
||||
|
||||
Reference in New Issue
Block a user