From b271f8b86947b779cc12c4e280a151cc6353d19b Mon Sep 17 00:00:00 2001 From: Aidan Reilly <74046732+aireilly@users.noreply.github.com> Date: Tue, 16 Apr 2024 16:46:20 +0100 Subject: [PATCH] Dual NIC HA PTP T-BC docs Aneesh's review comments Updates for Aneesh May 2 Jeana's review comments dual-NIC is a compound adj Update for Hen --- ...are-version-for-intel-800-series-nics.adoc | 87 +++++++++ ...ervices-as-grandmaster-clock-dual-nic.adoc | 4 +- ...tp-dual-wpc-hardware-config-reference.adoc | 6 +- modules/nw-ptp-installing-operator-cli.adoc | 2 +- ...w-ptp-installing-operator-web-console.adoc | 2 +- .../nw-ptp-operator-metrics-reference.adoc | 6 + ...p-services-as-boundary-clock-dual-nic.adoc | 10 +- ...nuxptp-services-as-ha-bc-for-dual-nic.adoc | 171 ++++++++++++++++++ modules/ptp-dual-nics.adoc | 20 +- modules/telco-ran-ptp-operator.adoc | 2 +- networking/ptp/configuring-ptp.adoc | 4 + snippets/ztp_PtpConfigForHA.yaml | 22 +++ 12 files changed, 319 insertions(+), 17 deletions(-) create mode 100644 modules/cnf-getting-the-dpll-firmware-version-for-intel-800-series-nics.adoc create mode 100644 modules/ptp-configuring-linuxptp-services-as-ha-bc-for-dual-nic.adoc create mode 100644 snippets/ztp_PtpConfigForHA.yaml diff --git a/modules/cnf-getting-the-dpll-firmware-version-for-intel-800-series-nics.adoc b/modules/cnf-getting-the-dpll-firmware-version-for-intel-800-series-nics.adoc new file mode 100644 index 0000000000..d005bb9d04 --- /dev/null +++ b/modules/cnf-getting-the-dpll-firmware-version-for-intel-800-series-nics.adoc @@ -0,0 +1,87 @@ +// Module included in the following assemblies: +// +// * networking/ptp/configuring-ptp.adoc + +:_mod-docs-content-type: PROCEDURE +[id="cnf-getting-the-dpll-firmware-version-for-intel-800-series-nics_{context}"] += Getting the DPLL firmware version for the CGU in an Intel 800 series NIC + +You can get the digital phase-locked loop (DPLL) firmware version for the Clock Generation Unit (CGU) in an Intel 800 series NIC by opening a debug shell to the cluster node and querying the NIC hardware. + +.Prerequisites + +* You have installed the OpenShift CLI (`oc`). + +* You have logged in as a user with `cluster-admin` privileges. + +* You have installed an Intel 800 series NIC in the cluster host. + +* You have installed the PTP Operator on a bare-metal cluster with hosts that support PTP. + +.Procedure + +. Start a debug pod by running the following command: ++ +[source,terminal] +---- +$ oc debug node/ +---- ++ +where: ++ +-- +:: Is the node where you have installed the Intel 800 series NIC. +-- + +. Check the CGU firmware version in the NIC by using the `devlink` tool and the bus and device name where the NIC is installed. +For example, run the following command: ++ +[source,terminal] +---- +sh-4.4# devlink dev info / | grep cgu +---- ++ +where: ++ +-- + :: Is the bus where the NIC is installed. For example, `pci`. + :: Is the NIC device name. For example, `0000:51:00.0`. +-- ++ +.Example output +[source,terminal] +---- +cgu.id 36 <1> +fw.cgu 8032.16973825.6021 <2> +---- +<1> CGU hardware revision number +<2> The DPLL firmware version running in the CGU, where the DPLL firmware version is `6201`, and the DPLL model is `8032`. +The string `16973825` is a shorthand representation of the binary version of the DPLL firmware version (`1.3.0.1`). + ++ +[NOTE] +==== +The firmware version has a leading nibble and 3 octets for each part of the version number. +The number `16973825` in binary is `0001 0000 0011 0000 0000 0000 0001`. +Use the binary value to decode the firmware version. +For example: + +.DPLL firmware version +[cols="1,2", width="90%", options="header"] +|==== +|Binary part +|Decimal value + +|`0001` +|1 + +|`0000 0011` +|3 + +|`0000 0000` +|0 + +|`0000 0001` +|1 +|==== +==== diff --git a/modules/nw-ptp-configuring-linuxptp-services-as-grandmaster-clock-dual-nic.adoc b/modules/nw-ptp-configuring-linuxptp-services-as-grandmaster-clock-dual-nic.adoc index 3dcd41ca14..373d5a3702 100644 --- a/modules/nw-ptp-configuring-linuxptp-services-as-grandmaster-clock-dual-nic.adoc +++ b/modules/nw-ptp-configuring-linuxptp-services-as-grandmaster-clock-dual-nic.adoc @@ -8,12 +8,12 @@ You can configure the `linuxptp` services (`ptp4l`, `phc2sys`, `ts2phc`) as grandmaster clock (T-GM) for dual E810 Westport Channel NICs by creating a `PtpConfig` custom resource (CR) that configures the host NICs. -For distributed RAN (D-RAN) use cases, you can configure PTP for dual NICs as follows: +For distributed RAN (D-RAN) use cases, you can configure PTP for dual-NICs as follows: * NIC one is synced to the global navigation satellite system (GNSS) time source. * NIC two is synced to the 1PPS timing output provided by NIC one. This configuration is provided by the PTP hardware plugin in the `PtpConfig` CR. -The dual NIC PTP T-GM configuration uses a single instance of `ptp4l` and one `ts2phc` process reporting two `ts2phc` instances, one for each NIC. +The dual-NIC PTP T-GM configuration uses a single instance of `ptp4l` and one `ts2phc` process reporting two `ts2phc` instances, one for each NIC. The host system clock is synchronized from the NIC that is connected to the GNSS time source. [NOTE] diff --git a/modules/nw-ptp-dual-wpc-hardware-config-reference.adoc b/modules/nw-ptp-dual-wpc-hardware-config-reference.adoc index 291aeb540c..a682640304 100644 --- a/modules/nw-ptp-dual-wpc-hardware-config-reference.adoc +++ b/modules/nw-ptp-dual-wpc-hardware-config-reference.adoc @@ -8,13 +8,13 @@ Use this information to understand how to use the link:https://github.com/openshift/linuxptp-daemon/blob/release-4.14/addons/intel/e810.go[Intel E810-XXVDA4T hardware plugin] to configure a pair of E810 network interfaces as PTP grandmaster clock (T-GM). -Before you configure the dual NIC cluster host, you must connect the two NICs with an SMA1 cable using the 1PPS faceplace connections. +Before you configure the dual-NIC cluster host, you must connect the two NICs with an SMA1 cable using the 1PPS faceplace connections. -When you configure a dual NIC T-GM, you need to compensate for the 1PPS signal delay that occurs when you connect the NICs using the SMA1 connection ports. +When you configure a dual-NIC T-GM, you need to compensate for the 1PPS signal delay that occurs when you connect the NICs using the SMA1 connection ports. Various factors such as cable length, ambient temperature, and component and manufacturing tolerances can affect the signal delay. To compensate for the delay, you must calculate the specific value that you use to offset the signal delay. -.E810 dual NIC T-GM PtpConfig CR reference +.E810 dual-NIC T-GM PtpConfig CR reference [cols="1,2" width="90%", options="header"] |==== |PtpConfig field diff --git a/modules/nw-ptp-installing-operator-cli.adoc b/modules/nw-ptp-installing-operator-cli.adoc index 04b89578bd..ccfd3a2892 100644 --- a/modules/nw-ptp-installing-operator-cli.adoc +++ b/modules/nw-ptp-installing-operator-cli.adoc @@ -1,6 +1,6 @@ // Module included in the following assemblies: // -// * networking/ptp/about-ptp.adoc +// * networking/ptp/configuring-ptp.adoc :_mod-docs-content-type: PROCEDURE [id="install-ptp-operator-cli_{context}"] diff --git a/modules/nw-ptp-installing-operator-web-console.adoc b/modules/nw-ptp-installing-operator-web-console.adoc index 8e32081f67..93726203d4 100644 --- a/modules/nw-ptp-installing-operator-web-console.adoc +++ b/modules/nw-ptp-installing-operator-web-console.adoc @@ -1,6 +1,6 @@ // Module included in the following assemblies: // -// * networking/ptp/about-ptp.adoc +// * networking/ptp/configuring-ptp.adoc :_mod-docs-content-type: PROCEDURE [id="install-ptp-operator-web-console_{context}"] diff --git a/modules/nw-ptp-operator-metrics-reference.adoc b/modules/nw-ptp-operator-metrics-reference.adoc index 8559305e91..48e2592f42 100644 --- a/modules/nw-ptp-operator-metrics-reference.adoc +++ b/modules/nw-ptp-operator-metrics-reference.adoc @@ -99,4 +99,10 @@ Possible values are 0 (`UNAVAILABLE`) and 1 (`AVAILABLE`). GNSS provides satellite-based positioning, navigation, and timing services globally. Possible values are 0 (`NOFIX`), 1 (`DEAD RECKONING ONLY`), 2 (`2D-FIX`), 3 (`3D-FIX`), 4 (`GPS+DEAD RECKONING FIX`), 5, (`TIME ONLY FIX`). |`{from="gnss",iface="ens2fx",node="compute-1.example.com",process="gnss"} 3` + +|`openshift_ptp_ha_profile_status` +|Returns the current status of the highly available system clock when there are multiple time sources on different NICs. +Possible values are 0 (`INACTIVE`), and 1 (`ACTIVE`). +|`{node="node1",process="phc2sys",profile="profile1"} 1` +`{node="node1",process="phc2sys",profile="profile2"} 0` |==== diff --git a/modules/ptp-configuring-linuxptp-services-as-boundary-clock-dual-nic.adoc b/modules/ptp-configuring-linuxptp-services-as-boundary-clock-dual-nic.adoc index 0f3261acb1..ad1e3b4015 100644 --- a/modules/ptp-configuring-linuxptp-services-as-boundary-clock-dual-nic.adoc +++ b/modules/ptp-configuring-linuxptp-services-as-boundary-clock-dual-nic.adoc @@ -4,12 +4,12 @@ :_mod-docs-content-type: PROCEDURE [id="ptp-configuring-linuxptp-services-as-bc-for-dual-nic_{context}"] -= Configuring linuxptp services as boundary clocks for dual NIC hardware += Configuring linuxptp services as boundary clocks for dual-NIC hardware -:FeatureName: Precision Time Protocol (PTP) hardware with dual NIC configured as boundary clocks +:FeatureName: Precision Time Protocol (PTP) hardware with dual-NIC configured as boundary clocks include::snippets/technology-preview.adoc[leveloffset=+1] -You can configure the `linuxptp` services (`ptp4l`, `phc2sys`) as boundary clocks for dual NIC hardware by creating a `PtpConfig` custom resource (CR) object for each NIC. +You can configure the `linuxptp` services (`ptp4l`, `phc2sys`) as boundary clocks for dual-NIC hardware by creating a `PtpConfig` custom resource (CR) object for each NIC. Dual NIC hardware allows you to connect each NIC to the same upstream leader clock with separate `ptp4l` instances for each NIC feeding the downstream clocks. @@ -76,7 +76,7 @@ spec: You must completely remove the `phc2sysOpts` field from the second `PtpConfig` CR to disable the `phc2sys` service on the second NIC. ==== -. Create the dual NIC `PtpConfig` CRs by running the following commands: +. Create the dual-NIC `PtpConfig` CRs by running the following commands: .. Create the CR that configures PTP for the first NIC: + @@ -94,7 +94,7 @@ $ oc create -f boundary-clock-ptp-config-nic2.yaml .Verification -* Check that the PTP Operator has applied the `PtpConfig` CRs for both NICs. Examine the logs for the `linuxptp` daemon corresponding to the node that has the dual NIC hardware installed. For example, run the following command: +* Check that the PTP Operator has applied the `PtpConfig` CRs for both NICs. Examine the logs for the `linuxptp` daemon corresponding to the node that has the dual-NIC hardware installed. For example, run the following command: + [source,terminal] ---- diff --git a/modules/ptp-configuring-linuxptp-services-as-ha-bc-for-dual-nic.adoc b/modules/ptp-configuring-linuxptp-services-as-ha-bc-for-dual-nic.adoc new file mode 100644 index 0000000000..b2eedf5254 --- /dev/null +++ b/modules/ptp-configuring-linuxptp-services-as-ha-bc-for-dual-nic.adoc @@ -0,0 +1,171 @@ +// Module included in the following assemblies: +// +// * networking/ptp/configuring-ptp.adoc + +:_mod-docs-content-type: PROCEDURE +[id="ptp-configuring-linuxptp-services-as-ha-bc-for-dual-nic_{context}"] += Configuring linuxptp as a highly available system clock for dual-NIC Intel E810 PTP boundary clocks + +You can configure the `linuxptp` services `ptp4l` and `phc2sys` as a highly available (HA) system clock for dual PTP boundary clocks (T-BC). + +The highly available system clock uses multiple time sources from dual-NIC Intel E810 Salem channel hardware configured as two boundary clocks. +Two boundary clocks instances participate in the HA setup, each with its own configuration profile. +You connect each NIC to the same upstream leader clock with separate `ptp4l` instances for each NIC feeding the downstream clocks. + +Create two `PtpConfig` custom resource (CR) objects that configure the NICs as T-BC and a third `PtpConfig` CR that configures high availability between the two NICs. + +[IMPORTANT] +==== +When you create the `PtpConfig` CRs, ensure that the `phc2sysOpts` field is an empty string to prevent setting up the `phc2sys` processes on these two profiles. +==== + +The third `PtpConfig` CR configures a highly available system clock service. +The CR sets the `ptp4lOpts` field to an empty string to prevent the `ptp4l` process from running. +The CR adds profiles for the `ptp4l` configurations under the `spec.profile.ptpSettings.haProfiles` key and passes the kernel socket path of those profiles to the `phc2sys` service. +When a `ptp4l` failure occurs, the `phc2sys` service switches to the backup `ptp4l` configuration. +When the primary profile becomes active again, the `phc2sys` service reverts to the original state. + +.Prerequisites + +* Install the {oc-first}. + +* Log in as a user with `cluster-admin` privileges. + +* Install the PTP Operator. + +* Configure a cluster node with Intel E810 Salem channel dual-NIC. + +.Procedure + +. Create two separate `PtpConfig` CRs, one for each NIC, using the CRs in "Configuring linuxptp services as boundary clocks for dual-NIC hardware" as a reference for each CR. + +.. Create the `ha-ptp-config-nic1.yaml` file, specifying an empty string for the `phc2sysOpts` field. +For example: ++ +[source,yaml] +---- +apiVersion: ptp.openshift.io/v1 +kind: PtpConfig +metadata: + name: ha-ptp-config-nic1 + namespace: openshift-ptp +spec: + profile: + - name: "ha-ptp-config-profile1" + ptp4lOpts: "-2 --summary_interval -4" + ptp4lConf: | <1> + [ens5f1] + masterOnly 1 + [ens5f0] + masterOnly 0 + #... + phc2sysOpts: "" <2> +---- +<1> Specify the required interfaces to start `ptp4l` as a boundary clock. For example, `ens5f0` synchronizes from a grandmaster clock and `ens5f1` synchronizes connected devices. +<2> Set `phc2sysOpts` with an empty string. +These values are populated from the `spec.profile.ptpSettings.haProfiles` field of the `PtpConfig` CR that configures high availability. + +.. Apply the `PtpConfig` CR for NIC 1 by running the following command: ++ +[source,terminal] +---- +$ oc create -f ha-ptp-config-nic1.yaml +---- + +.. Create the `ha-ptp-config-nic2.yaml` file, specifying an empty string for the `phc2sysOpts` field. +For example: ++ +[source,yaml] +---- +apiVersion: ptp.openshift.io/v1 +kind: PtpConfig +metadata: + name: ha-ptp-config-nic2 + namespace: openshift-ptp +spec: + profile: + - name: "ha-ptp-config-profile2" + ptp4lOpts: "-2 --summary_interval -4" + ptp4lConf: | + [ens7f1] + masterOnly 1 + [ens7f0] + masterOnly 0 + #... + phc2sysOpts: "" +---- + +.. Apply the `PtpConfig` CR for NIC 2 by running the following command: ++ +[source,terminal] +---- +$ oc create -f ha-ptp-config-nic2.yaml +---- + +. Create the `PtpConfig` CR that configures the HA system clock. +For example: + +.. Create the `ptp-config-for-ha.yaml` file: ++ +[source,yaml] +---- +include::snippets/ztp_PtpConfigForHA.yaml[] +---- ++ +[IMPORTANT] +==== +Do not apply the high availability `PtpConfig` CR before the `PtpConfig` CRs that configure the individual NICs. +==== + +.. Apply the HA `PtpConfig` CR by running the following command: ++ +[source,terminal] +---- +$ oc create -f ptp-config-for-ha.yaml +---- + +.Verification + +* Verify that the PTP Operator has applied the `PtpConfig` CRs correctly. +Perform the following steps: + +.. Get the list of pods in the `openshift-ptp` namespace by running the following command: ++ +[source,terminal] +---- +$ oc get pods -n openshift-ptp -o wide +---- ++ +.Example output +[source,terminal] +---- +NAME READY STATUS RESTARTS AGE IP NODE +linuxptp-daemon-4xkrb 1/1 Running 0 43m 10.1.196.24 compute-0.example.com +ptp-operator-657bbq64c8-2f8sj 1/1 Running 0 43m 10.129.0.61 control-plane-1.example.com +---- ++ +[NOTE] +==== +There should be only one `linuxptp-daemon` pod. +==== + +.. Check that the profile is correct by running the following command. +Examine the logs of the `linuxptp` daemon that corresponds to the node you specified in the `PtpConfig` profile. ++ +[source,terminal] +---- +$ oc logs linuxptp-daemon-4xkrb -n openshift-ptp -c linuxptp-daemon-container +---- ++ +.Example output +[source,terminal] +---- +I1115 09:41:17.117596 4143292 daemon.go:107] in applyNodePTPProfile +I1115 09:41:17.117604 4143292 daemon.go:109] updating NodePTPProfile to: +I1115 09:41:17.117607 4143292 daemon.go:110] ------------------------------------ +I1115 09:41:17.117612 4143292 daemon.go:102] Profile Name: ha-ptp-config-profile1 +I1115 09:41:17.117616 4143292 daemon.go:102] Interface: +I1115 09:41:17.117620 4143292 daemon.go:102] Ptp4lOpts: -2 +I1115 09:41:17.117623 4143292 daemon.go:102] Phc2sysOpts: -a -r -n 24 +I1115 09:41:17.117626 4143292 daemon.go:116] ------------------------------------ +---- diff --git a/modules/ptp-dual-nics.adoc b/modules/ptp-dual-nics.adoc index aeeb19f0c0..89547f1b65 100644 --- a/modules/ptp-dual-nics.adoc +++ b/modules/ptp-dual-nics.adoc @@ -4,12 +4,12 @@ :_mod-docs-content-type: CONCEPT [id="ptp-dual-nics_{context}"] -= Using dual Intel E810 NIC hardware with PTP += Using dual-NIC Intel E810 hardware with PTP -{product-title} supports single and dual NIC Intel E810 hardware for precision PTP timing in grandmaster clocks (T-GM) and boundary clocks (T-BC). +{product-title} supports single and dual-NIC Intel E810 hardware for precision PTP timing in grandmaster clocks (T-GM) and boundary clocks (T-BC). Dual NIC grandmaster clock:: -You can use a cluster host that has dual NIC hardware as PTP grandmaster clock. +You can use a cluster host that has dual-NIC hardware as PTP grandmaster clock. One NIC receives timing information from the global navigation satellite system (GNSS). The second NIC receives the timing information from the first using the SMA1 Tx/Rx connections on the E810 NIC faceplate. The system clock on the cluster host is synchronized from the NIC that is connected to the GNSS satellite. @@ -22,10 +22,22 @@ image::561_OpenShift_Using_PTP_network_0124.png[Dual NIC PTP grandmaster clock c + [NOTE] ==== -In a dual NIC T-GM configuration, a single `ts2phc` process reports as two `ts2phc` instances in the system. +In a dual-NIC T-GM configuration, a single `ts2phc` process reports as two `ts2phc` instances in the system. ==== Dual NIC boundary clock:: For 5G telco networks that deliver mid-band spectrum coverage, each virtual distributed unit (vDU) requires connections to 6 radio units (RUs). To make these connections, each vDU host requires 2 NICs configured as boundary clocks. + Dual NIC hardware allows you to connect each NIC to the same upstream leader clock with separate `ptp4l` instances for each NIC feeding the downstream clocks. + +Highly available system clock with dual-NIC boundary clocks:: +You can configure Intel E810-XXVDA4 Salem channel dual-NIC hardware as dual PTP boundary clocks that provide timing for a highly available system clock. +This is useful when you have multiple time sources on different NICs. +High availability ensures that the node does not lose timing synchronisation if one of the two timing sources is lost or disconnected. ++ +Each NIC is connected to the same upstream leader clock. +Highly available boundary clocks use multiple PTP domains to synchronize with the target system clock. +When a T-BC is highly available, the host system clock can maintain the correct offset even if one or more `ptp4l` instances syncing the NIC PHC clock fails. +If any single SFP port or cable failure occurs, the boundary clock stays in sync with the leader clock. ++ +Boundary clock leader source selection is done using the A-BMCA algorithm. For more information, see link:https://www.itu.int/rec/T-REC-G.8275.1/en[ITU-T recommendation G.8275.1]. diff --git a/modules/telco-ran-ptp-operator.adoc b/modules/telco-ran-ptp-operator.adoc index 2d76cdf4cf..5f70fded54 100644 --- a/modules/telco-ran-ptp-operator.adoc +++ b/modules/telco-ran-ptp-operator.adoc @@ -30,7 +30,7 @@ Optional: subscribe applications to PTP events that happen on the node that the You subscribe the application to events via HTTP. Limits and requirements:: -* High availability is not supported with dual NIC configurations. +* High availability is not supported with dual-NIC configurations. * GPS offsets are not reported. Use a default offset of less than or equal to 5. diff --git a/networking/ptp/configuring-ptp.adoc b/networking/ptp/configuring-ptp.adoc index 1aced4565b..0d0c903e6b 100644 --- a/networking/ptp/configuring-ptp.adoc +++ b/networking/ptp/configuring-ptp.adoc @@ -46,6 +46,8 @@ include::modules/nw-ptp-configuring-linuxptp-services-as-boundary-clock.adoc[lev include::modules/ptp-configuring-linuxptp-services-as-boundary-clock-dual-nic.adoc[leveloffset=+2] +include::modules/ptp-configuring-linuxptp-services-as-ha-bc-for-dual-nic.adoc[leveloffset=+2] + include::modules/nw-ptp-configuring-linuxptp-services-as-ordinary-clock.adoc[leveloffset=+1] [role="_additional-resources"] @@ -68,4 +70,6 @@ include::modules/cnf-configuring-log-filtering-for-linuxptp.adoc[leveloffset=+1] include::modules/cnf-troubleshooting-common-ptp-operator-issues.adoc[leveloffset=+1] +include::modules/cnf-getting-the-dpll-firmware-version-for-intel-800-series-nics.adoc[leveloffset=+1] + include::modules/cnf-about-collecting-ptp-data.adoc[leveloffset=+1] diff --git a/snippets/ztp_PtpConfigForHA.yaml b/snippets/ztp_PtpConfigForHA.yaml new file mode 100644 index 0000000000..821ec85868 --- /dev/null +++ b/snippets/ztp_PtpConfigForHA.yaml @@ -0,0 +1,22 @@ +apiVersion: ptp.openshift.io/v1 +kind: PtpConfig +metadata: + name: boundary-ha + namespace: openshift-ptp + annotations: + ran.openshift.io/ztp-deploy-wave: "10" +spec: + profile: + - name: "boundary-ha" + ptp4lOpts: " " + phc2sysOpts: "-a -r -n 24" + ptpSchedulingPolicy: SCHED_FIFO + ptpSchedulingPriority: 10 + ptpSettings: + logReduce: "true" + haProfiles: "ha-ptp-config-nic1,ha-ptp-config-nic2" + recommend: + - profile: "boundary-ha" + priority: 4 + match: + - nodeLabel: "node-role.kubernetes.io/$mcp"