openshift-docs/modules/deployments-ab-testing-lb.adoc

// Module included in the following assemblies:
//
// * applications/deployments/route-based-deployment-strategies.adoc

:_mod-docs-content-type: PROCEDURE
[id="deployments-ab-testing-lb_{context}"]
= Load balancing for A/B testing

The user sets up a route with multiple services. Each service handles a version of the application.

Each service is assigned a `weight` and the portion of requests to each service is the `service_weight` divided by the `sum_of_weights`. The `weight` for each service is distributed to the service's endpoints so that the sum of the endpoint `weights` is the service `weight`.

The route can have up to four services. The `weight` for the service can be between `0` and `256`. When the `weight` is `0`, the service does not participate in load balancing but continues to serve existing persistent connections. When the service `weight` is not `0`, each endpoint has a minimum `weight` of `1`. Because of this, a service with a lot of endpoints can end up with higher `weight` than intended. In this case, reduce the number of pods to get the expected load balance `weight`.

////
See the
xref:../../architecture/networking/routes.adoc#alternateBackends[Alternate
Backends and Weights] section for more information.

The web console allows users to set the weighting and show balance between them:

weighting.png[Visualization of Alternate Back Ends in the Web Console]
////

.Procedure

To set up the A/B environment:

. Create the two applications and give them different names. Each creates a `Deployment` object. The applications are versions of the same program; one is usually the current production version and the other the proposed new version.
.. Create the first application. The following example creates an application called `ab-example-a`:
+
[source,terminal]
----
$ oc new-app openshift/deployment-example --name=ab-example-a
----
+
.. Create the second application:
+
[source,terminal]
----
$ oc new-app openshift/deployment-example:v2 --name=ab-example-b
----
+
Both applications are deployed and services are created.

. Make the application available externally via a route. At this point, you can expose either. It can be convenient to expose the current production version first and later modify the route to add the new version.
+
[source,terminal]
----
$ oc expose svc/ab-example-a
----
+
Browse to the application at `ab-example-a.<project>.<router_domain>` to verify that you see the expected version.

. When you deploy the route, the router balances the traffic according to the `weights` specified for the services. At this point, there is a single service with default `weight=1` so all requests go to it. Adding the other service as an `alternateBackends` and adjusting the `weights` brings the A/B setup to life. This can be done by the `oc set route-backends` command or by editing the route.
+
[NOTE]
====
When using `alternateBackends`, also use the `roundrobin` load balancing strategy to ensure requests are distributed as expected to the services based on weight. `roundrobin` can be set for a route by using a route annotation. See the _Additional resources_ section for more information about route annotations.
====
+
Setting the `oc set route-backend` to `0` means the service does not participate in load balancing, but continues to serve existing persistent connections.
+
[NOTE]
====
Changes to the route just change the portion of traffic to the various services. You might have to scale the deployment to adjust the number of pods to handle the anticipated loads.
====
+
To edit the route, run:
+
[source,terminal]
----
$ oc edit route <route_name>
----
+
.Example output
[source,terminal]
----
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: route-alternate-service
  annotations:
    haproxy.router.openshift.io/balance: roundrobin
# ...
spec:
  host: ab-example.my-project.my-domain
  to:
    kind: Service
    name: ab-example-a
    weight: 10
  alternateBackends:
  - kind: Service
    name: ab-example-b
    weight: 15
# ...
----

[id="deployments-ab-testing-lb-web_{context}"]
== Managing weights of an existing route using the web console

.Procedure

. Navigate to the *Networking* -> *Routes* page.

. Click the Options menu {kebab} next to the route you want to edit and select *Edit Route*.

. Edit the YAML file. Update the `weight` to be an integer between `0` and `256` that specifies the relative weight of the target against other target reference objects. The value `0` suppresses requests to this back end. The default is `100`. Run `oc explain routes.spec.alternateBackends` for more information about the options.

. Click *Save*.

[id="deployments-ab-testing-lb-web-new-route_{context}"]
== Managing weights of an new route using the web console

. Navigate to the *Networking* -> *Routes* page.

. Click *Create Route*.

. Enter the route *Name*.

. Select the *Service*.

. Click *Add Alternate Service*.

. Enter a value for *Weight* and *Alternate Service Weight*. Enter a number between `0` and `255` that depicts relative weight compared with other targets. The default is `100`.

. Select the *Target Port*.

. Click *Create*.

[id="deployments-ab-testing-lb-cli_{context}"]
== Managing weights using the CLI

.Procedure

. To manage the services and corresponding weights load balanced by the route, use the `oc set route-backends` command:
+
[source,terminal]
----
$ oc set route-backends ROUTENAME \
    [--zero|--equal] [--adjust] SERVICE=WEIGHT[%] [...] [options]
----
+
For example, the following sets `ab-example-a` as the primary service with `weight=198` and `ab-example-b` as the first alternate service with a `weight=2`:
+
[source,terminal]
----
$ oc set route-backends ab-example ab-example-a=198 ab-example-b=2
----
+
This means 99% of traffic is sent to service `ab-example-a` and 1% to service `ab-example-b`.
+
This command does not scale the deployment. You might be required to do so to have enough pods to handle the request load.

. Run the command with no flags to verify the current configuration:
+
[source,terminal]
----
$ oc set route-backends ab-example
----
+
.Example output
[source,terminal]
----
NAME                    KIND     TO           WEIGHT
routes/ab-example       Service  ab-example-a 198 (99%)
routes/ab-example       Service  ab-example-b 2   (1%)
----

. To override the default values for the load balancing algorithm, adjust the annotation on the route by setting the algorithm to `roundrobin`. For a route on {product-title}, the default load balancing algorithm is set to `random` or `source` values.

+
To set the algorithm to `roundrobin`, run the command:
+
[source,terminal]
----
$ oc annotate routes/<route-name> haproxy.router.openshift.io/balance=roundrobin
----
+
For Transport Layer Security (TLS) passthrough routes, the default value is `source`. For all other routes, the default is `random`.


. To alter the weight of an individual service relative to itself or to the primary service, use the `--adjust` flag. Specifying a percentage adjusts the service relative to either the primary or the first alternate (if you specify the primary). If there are other backends, their weights are kept proportional to the changed.
+
The following example alters the weight of `ab-example-a` and `ab-example-b` services:
+
[source,terminal]
----
$ oc set route-backends ab-example --adjust ab-example-a=200 ab-example-b=10
----
+
Alternatively, alter the weight of a service by specifying a percentage:
+
[source,terminal]
----
$ oc set route-backends ab-example --adjust ab-example-b=5%
----
+
By specifying `+` before the percentage declaration, you can adjust a weighting relative to the current setting. For example:
+
[source,terminal]
----
$ oc set route-backends ab-example --adjust ab-example-b=+15%
----
+
The `--equal` flag sets the `weight` of all services to `100`:
+
[source,terminal]
----
$ oc set route-backends ab-example --equal
----
+
The `--zero` flag sets the `weight` of all services to `0`. All requests then return with a 503 error.
+
[NOTE]
====
Not all routers may support multiple or weighted backends.
====

[id="deployments-ab-one-service-multi-dc_{context}"]
== One service, multiple `Deployment` objects

.Procedure

. Create a new application, adding a label `ab-example=true` that will be common to all shards:
+
[source,terminal]
----
$ oc new-app openshift/deployment-example --name=ab-example-a --as-deployment-config=true --labels=ab-example=true --env=SUBTITLE\=shardA
----
+
[source,terminal]
----
$ oc delete svc/ab-example-a
----
+
The application is deployed and a service is created. This is the first shard.

. Make the application available via a route, or use the service IP directly:
+
[source,terminal]
----
$ oc expose deployment ab-example-a --name=ab-example --selector=ab-example\=true
----
+
[source,terminal]
----
$ oc expose service ab-example
----

. Browse to the application at `ab-example-<project_name>.<router_domain>` to verify you see the `v1` image.

. Create a second shard based on the same source image and label as the first shard, but with a different tagged version and unique environment variables:
+
[source,terminal]
----
$ oc new-app openshift/deployment-example:v2 \
    --name=ab-example-b --labels=ab-example=true \
    SUBTITLE="shard B" COLOR="red" --as-deployment-config=true
----
+
[source,terminal]
----
$ oc delete svc/ab-example-b
----

. At this point, both sets of pods are being served under the route. However, because both browsers (by leaving a connection open) and the router (by default, through a cookie) attempt to preserve your connection to a back-end server, you might not see both shards being returned to you.
+
To force your browser to one or the other shard:

.. Use the `oc scale` command to reduce replicas of `ab-example-a` to `0`.
+
[source,terminal]
----
$ oc scale dc/ab-example-a --replicas=0
----
+
Refresh your browser to show `v2` and `shard B` (in red).

.. Scale `ab-example-a` to `1` replica and `ab-example-b` to `0`:
+
[source,terminal]
----
$ oc scale dc/ab-example-a --replicas=1; oc scale dc/ab-example-b --replicas=0
----
+
Refresh your browser to show `v1` and `shard A` (in blue).

. If you trigger a deployment on either shard, only the pods in that shard are affected. You can trigger a deployment by changing the `SUBTITLE` environment variable in either `Deployment` object:
+
[source,terminal]
----
$ oc edit dc/ab-example-a
----
+
or
+
[source,terminal]
----
$ oc edit dc/ab-example-b
----