openshift-docs/modules/nw-multus-advanced-annotations.adoc

// Module included in the following assemblies:
//
// * networking/multiple_networks/attaching-pod.adoc

[id="nw-multus-advanced-annotations_{context}"]
= Specifying pod-specific addressing and routing options

When attaching a pod to an additional network, you may want to specify further properties
about that network in a particular pod. This allows you to change some aspects of routing, as well
as specify static IP addresses and MAC addresses. In order to accomplish this, you can use the JSON formatted annotations.

.Prerequisites

* The pod must be in the same namespace as the additional network.
* Install the OpenShift Command-line Interface (`oc`).
* You must log in to the cluster.
ifdef::sriov[]
* You must have the SR-IOV Operator installed and a SriovNetwork CR defined.
endif::sriov[]

.Procedure

To add a pod to an additional network while specifying addressing and/or routing options, complete the following steps:

. Edit the `Pod` resource definition. If you are editing an existing `Pod` resource, run the
following command to edit its definition in the default editor. Replace `<name>`
with the name of the `Pod` resource to edit.
+
[source,terminal]
----
$ oc edit pod <name>
----

. In the `Pod` resource definition, add the `k8s.v1.cni.cncf.io/networks`
parameter to the pod `metadata` mapping. The `k8s.v1.cni.cncf.io/networks`
accepts a JSON string of a list of objects that reference the name of NetworkAttachmentDefinition Custom Resource (CR) names
in addition to specifying additional properties.
+
[source,yaml]
----
metadata:
  annotations:
    k8s.v1.cni.cncf.io/networks: '[<network>[,<network>,...]]' <1>
----
<1> Replace `<network>` with a JSON object as shown in the following examples. The single quotes are required.

. In the following example the annotation specifies which network attachment will have the default route,
using the `default-route` parameter.
+
[source,yaml]
----
apiVersion: v1
kind: Pod
metadata:
  name: example-pod
  annotations:
    k8s.v1.cni.cncf.io/networks: '
    {
      "name": "net1"
    },
    {
      "name": "net2", <1>
      "default-route": ["192.0.2.1"] <2>
    }'
spec:
  containers:
  - name: example-pod
    command: ["/bin/bash", "-c", "sleep 2000000000000"]
    image: centos/tools
----
<1> The `name` key is the name of the additional network to associate
with the pod.
<2> The `default-route` key specifies a value of a gateway for traffic to be routed over if no other
routing entry is present in the routing table. If more than one `default-route` key is specified,
this will cause the pod to fail to become active.

The default route will cause any traffic that is not specified in other routes to be routed to the gateway.

[IMPORTANT]
====
Setting the default route to an interface other than the default network interface for {product-title}
may cause traffic that is anticipated for pod-to-pod traffic to be routed over another interface.
====

To verify the routing properties of a pod, the `oc` command may be used to execute the `ip` command within a pod.

[source,terminal]
----
$ oc exec -it <pod_name> -- ip route
----

[NOTE]
====
You may also reference the pod's `k8s.v1.cni.cncf.io/networks-status` to see which additional network has been
assigned the default route, by the presence of the `default-route` key in the JSON-formatted list of objects.
====


To set a static IP address or MAC address for a pod you can use the JSON formatted annotations. This requires you create networks that specifically allow for this functionality. This can be specified in a rawCNIConfig for the CNO.

. Edit the CNO CR by running the following command:
+
[source,terminal]
----
$ oc edit networks.operator.openshift.io cluster
----

The following YAML describes the configuration parameters for the CNO:

.Cluster Network Operator YAML configuration
[source,yaml]
----
name: <name> <1>
namespace: <namespace> <2>
rawCNIConfig: '{ <3>
  ...
}'
type: Raw
----
<1> Specify a name for the additional network attachment that you are
creating. The name must be unique within the specified `namespace`.

<2> Specify the namespace to create the network attachment in. If
you do not specify a value, then the `default` namespace is used.

<3> Specify the CNI plug-in configuration in JSON format, which
is based on the following template.

The following object describes the configuration parameters for utilizing static MAC address
and IP address using the macvlan CNI plug-in:

.macvlan CNI plug-in JSON configuration object using static IP and MAC address
[source,json]
----
{
  "cniVersion": "0.3.1",
  "plugins": [{ <1>
      "type": "macvlan",
      "capabilities": { "ips": true }, <2>
      "master": "eth0", <3>
      "mode": "bridge",
      "ipam": {
        "type": "static"
      }
    }, {
      "capabilities": { "mac": true }, <4>
      "type": "tuning"
    }]
}
----

<1> The `plugins` field specifies a configuration list of CNI configurations.

<2> The `capabilities` key denotes that a request is being made to enable the static IP functionality of a CNI plug-ins runtime configuration capabilities.

<3> The `master` field is specific to the macvlan plug-in.

<4> Here the `capabilities` key denotes that a request is made to enable the static MAC address functionality of a CNI plug-in.

The above network attachment may then be referenced in a JSON formatted annotation, along with keys to specify which
static IP and MAC address will be assigned to a given pod.

Edit the desired pod with:

[source,terminal]
----
$ oc edit pod <name>
----

.macvlan CNI plug-in JSON configuration object using static IP and MAC address

[source,yaml]
----
apiVersion: v1
kind: Pod
metadata:
  name: example-pod
  annotations:
    k8s.v1.cni.cncf.io/networks: '[
      {
        "name": "<name>", <1>
        "ips": [ "192.0.2.205/24" ], <2>
        "mac": "CA:FE:C0:FF:EE:00" <3>
      }
    ]'
----

<1> Use the `<name>` as provided when creating the `rawCNIConfig` above.

<2> Provide the desired IP address.

<3> Provide the desired MAC address.

[NOTE]
====
Static IP addresses and MAC addresses do not have to be used at the same time, you may use them
individually, or together.
====

To verify the IP address and MAC properties of a pod with additional networks, use the `oc` command to execute the ip command within a pod.

[source,terminal]
----
$ oc exec -it <pod_name> -- ip a
----