diff --git a/README.md b/README.md index 1a7d045135..e2a50bb919 100644 --- a/README.md +++ b/README.md @@ -2,22 +2,17 @@ ## Supported Platforms -* [AWS](docs/user/aws/README.md) -* [AWS (UPI)](docs/user/aws/install_upi.md) -* [Azure](docs/user/azure/README.md) -* [Bare Metal (UPI)](docs/user/metal/install_upi.md) -* [Bare Metal (IPI)](docs/user/metal/install_ipi.md) -* [GCP](docs/user/gcp/README.md) -* [GCP (UPI)](docs/user/gcp/install_upi.md) -* [Libvirt with KVM](docs/dev/libvirt/README.md) (development only) -* [OpenStack](docs/user/openstack/README.md) -* [OpenStack (UPI)](docs/user/openstack/install_upi.md) -* [Power](docs/user/power/install_upi.md) -* [oVirt](docs/user/ovirt/install_ipi.md) -* [oVirt (UPI)](docs/user/ovirt/install_upi.md) -* [vSphere](docs/user/vsphere/README.md) -* [vSphere (UPI)](docs/user/vsphere/install_upi.md) -* [z/VM](docs/user/zvm/install_upi.md) +* [AWS](docs/user/aws/) ([Official Docs](https://docs.openshift.com/container-platform/latest/installing/installing_aws/preparing-to-install-on-aws.html)) +* [Azure](docs/user/azure/) ([Official Docs](https://docs.openshift.com/container-platform/latest/installing/installing_azure/preparing-to-install-on-azure.html)) +* [Bare Metal](docs/user/metal/) ([Official Docs](https://docs.openshift.com/container-platform/latest/installing/installing_bare_metal/preparing-to-install-on-bare-metal.html)) +* [GCP](docs/user/gcp/) ([Official Docs](https://docs.openshift.com/container-platform/latest/installing/installing_gcp/preparing-to-install-on-gcp.html)) +* IBM Cloud ([Official Docs](https://docs.openshift.com/container-platform/latest/installing/installing_ibm_cloud/preparing-to-install-on-ibm-cloud.html)) +* Nutanix ([Official Docs](https://docs.openshift.com/container-platform/latest/installing/installing_nutanix/preparing-to-install-on-nutanix.html)) +* [OpenStack](docs/user/openstack/) ([Official Docs](https://docs.openshift.com/container-platform/latest/installing/installing_openstack/preparing-to-install-on-openstack.html)) +* [Power](docs/user/power/) ([Official Docs](https://docs.openshift.com/container-platform/latest/installing/installing_ibm_power/preparing-to-install-on-ibm-power.html)) +* Power VS ([Official Docs](https://docs.openshift.com/container-platform/latest/installing/installing_ibm_powervs/preparing-to-install-on-ibm-power-vs.html)) +* [vSphere](docs/user/vsphere/) ([Official Docs](https://docs.openshift.com/container-platform/latest/installing/installing_vsphere/preparing-to-install-on-vsphere.html)) +* [z/VM](docs/user/zvm/) ([Official Docs](https://docs.openshift.com/container-platform/latest/installing/installing_ibm_z/preparing-to-install-on-ibm-z.html)) ## Quick Start diff --git a/docs/dev/libvirt/OWNERS b/docs/dev/libvirt/OWNERS deleted file mode 100644 index 5d37d841ef..0000000000 --- a/docs/dev/libvirt/OWNERS +++ /dev/null @@ -1,7 +0,0 @@ -# See the OWNERS docs: https://git.k8s.io/community/contributors/guide/owners.md -# This file just uses aliases defined in OWNERS_ALIASES. - -approvers: - - libvirt-approvers -reviewers: - - libvirt-reviewers diff --git a/docs/dev/libvirt/README.md b/docs/dev/libvirt/README.md deleted file mode 100644 index 22546be097..0000000000 --- a/docs/dev/libvirt/README.md +++ /dev/null @@ -1,464 +0,0 @@ -# Libvirt Support Update - -OpenShift Installer has stopped support for the libvirt platform starting 4.16. You can still use it with versions -4.15 and older. - -# Libvirt HOWTO - -Launching clusters via libvirt is especially useful for operator development. - -**NOTE:** Some aspects of the installation can be customized through the -`install-config.yaml` file. See -[how to create an install-config.yaml file](../../user/overview.md#multiple-invocations) and [the libvirt platform customization](customization.md) documents. - -## One-time setup - -It's expected that you will create and destroy clusters often in the course of development. These steps only need to be run once. - -Before you begin, install the [build dependencies](../dependencies.md). - -### Enable KVM - -Make sure you have KVM enabled by checking for the device: - -```console -$ ls -l /dev/kvm -crw-rw-rw-+ 1 root kvm 10, 232 Oct 31 09:22 /dev/kvm -``` - -If it is missing, try some of the ideas [here][kvm-install]. - -### Install and Enable Libvirt - -On CentOS 7, first enable the -[kvm-common](http://mirror.centos.org/centos/7/virt/x86_64/kvm-common/) -repository to ensure you get a new enough version of qemu-kvm. - -On Fedora, CentOS/RHEL: - -```sh -sudo dnf install libvirt-devel libvirt-daemon-kvm libvirt-client -``` - -Then start libvirtd: - -```sh -sudo systemctl enable --now libvirtd -``` - -### Pick names - -In this example, we'll set the base domain to `tt.testing` and the cluster name to `test1`. - -### Clone the project - -```sh -git clone https://github.com/openshift/installer.git -cd installer -``` - -### Enable IP Forwarding - -Libvirt creates a bridged connection to the host machine, but in order for the -network bridge to work IP forwarding needs to be enabled. The following command -will tell you if forwarding is enabled: - -```sh -sysctl net.ipv4.ip_forward -``` - -If the command output is: - -```sh -net.ipv4.ip_forward = 0 -``` - -then forwarding is disabled and proceed with the rest of this section. If IP -forwarding is enabled then skip the rest of this section. - -To enable IP forwarding for the current boot: - -```sh -sysctl net.ipv4.ip_forward=1 -``` - -or to persist the setting across reboots (recommended): - -```sh -echo "net.ipv4.ip_forward = 1" | sudo tee /etc/sysctl.d/99-ipforward.conf -sudo sysctl -p /etc/sysctl.d/99-ipforward.conf -``` - -### Configure libvirt to accept TCP connections - -The Kubernetes [cluster-api](https://github.com/kubernetes-sigs/cluster-api) -components drive deployment of worker machines. The libvirt cluster-api -provider will run inside the local cluster, and will need to connect back to -the libvirt instance on the host machine to deploy workers. - -In order for this to work, you'll need to enable unauthenticated TCP -connections for libvirt. - -**NOTE:** The following configuration disables all encryption and authentication -options in libvirtd and causes it to listen on all network interfaces and IP -addresses. **A connection to this privileged libvirtd gives the client -privileges equivalent to those of a root shell.** This configuration has a -security impact on a par with running a telnet server with no root password set. -It is critical to follow the steps below to **configure the firewall to prevent -access to libvirt from other hosts on the LAN/WAN**. - -#### For systemd activated libvirt - -This applies only if the libvirt daemon is configured to use socket activation. -This is currently the case on Fedora 31 (and later) and Arch Linux. - -First, you need to start the libvirtd TCP socket, which is managed by systemd: - -```sh -sudo systemctl start libvirtd-tcp.socket -``` - -To make this change persistent across reboots you can optionally enable it: - -```sh -sudo systemctl enable libvirtd-tcp.socket -``` - -Then to enable TCP access to libvirtd, modify `/etc/libvirt/libvirtd.conf` and -set the following: - -``` -auth_tcp = "none" -``` - -Then restart libvirt: - -```sh -sudo systemctl restart libvirtd -``` - -#### For permanently running libvirt daemon - -This applies only if the libvirt daemon is started only through -`libvirtd.service` and without making use of systemd socket activation (through -`libvirtd.socket` and similar systemd units). - - -For RHEL/CentOS, make sure that the following is set in -`/etc/sysconfig/libvirtd`: - -``` -LIBVIRTD_ARGS="--listen" -``` - -For Debian based distros, make sure that the following is set in -`/etc/default/libvirtd`: - -``` -libvirtd_opts="--listen" -``` - -Then to enable TCP access to libvirtd, modify `/etc/libvirt/libvirtd.conf` and -set the following: - -``` -listen_tls = 0 -listen_tcp = 1 -auth_tcp = "none" -tcp_port = "16509" -``` - -Then restart libvirt: - -```sh -sudo systemctl restart libvirtd -``` - -#### Configure qemu.conf - -On Debian/Ubuntu it might be needed to configure security driver for qemu. -Installer uses terraform libvirt, and it has a known issue, that might cause -unexpected `Could not open '/var/lib/libvirt/images/': Permission denied` -errors. Double check that `security_driver = "none"` line is present in -`/etc/libvirt/qemu.conf` and not commented out. - -#### Firewall - -Finally, if you have a firewall, you may have to allow connections to the -libvirt daemon from the IP range used by your cluster nodes. - -The following examples use the default cluster IP range of `192.168.126.0/24` (which is currently not configurable) and a libvirt `default` subnet of `192.168.122.0/24`, which might be different in your configuration. -If you're uncertain about the libvirt *default* subnet you should be able to see its address using the command `ip -4 a show dev virbr0` or by inspecting `virsh --connect qemu:///system net-dumpxml default`. -Ensure the cluster IP range does not overlap your `virbr0` IP address. - -#### iptables - -```sh -sudo iptables -I INPUT -p tcp -s 192.168.126.0/24 -d 192.168.122.1 --dport 16509 -j ACCEPT -m comment --comment "Allow insecure libvirt clients" -``` - -#### Firewalld - -If using `firewalld`, the specifics will depend on how your distribution has set -up the various zones. The following instructions should work as is for Fedora, -CentOS, RHEL and Arch Linux. - -First, as we don't want to expose the libvirt port externally, we will need to -actively block it: - -```sh -sudo firewall-cmd --add-rich-rule "rule service name="libvirt" reject" -``` - -For systems with libvirt version 5.1.0 and later, libvirt will set new bridged -network interfaces in the `libvirt` zone. We thus need to allow `libvirt` -traffic from the VMs to reach the host: - -```sh -sudo firewall-cmd --zone=libvirt --add-service=libvirt -``` - -For system with an older libvirt, we will move the new bridge interface to a -dedicated network zone and enable incoming libvirt, DNS & DHCP traffic: - -```sh -sudo firewall-cmd --zone=dmz --change-interface=tt0 -sudo firewall-cmd --zone=dmz --add-service=libvirt -sudo firewall-cmd --zone=dmz --add-service=dns -sudo firewall-cmd --zone=dmz --add-service=dhcp -``` - -NOTE: When the firewall rules are no longer needed, `sudo firewall-cmd --reload` -will remove the changes made as they were not permanently added. For persistence, -add `--permanent` to the `firewall-cmd` commands and run them a second time. - -### Set up DNS - -This step allows installer and users to resolve cluster-internal hostnames from your host. - -#### Using systemd-resolved - -If your system is using `systemd-resolved` (the default since Fedora 33), then you can setup DNS without using `dnsmasq`. - -Just point `resolvectl` to use `192.168.126.1` for your `baseDomain`. Replace `baseDomain` in the example accordingly. - -```sh -sudo resolvectl dns tt0 192.168.126.1 -sudo resolvectl domain tt0 ~ -``` - -Your cluster might run on a different network interface. To find which one it is, either check your `install-config.yaml` -or run `nmcli` and find the interface that's running the `192.168.126.0/24` net. - -Please note that this is not persistent across libvirt network creation/deletion of the tt0 interface. Once the tt0 interface is deleted and recreated, it will need to be updated again with the proper resolver. - -#### Using NetworkManager DNS overlay - -1. Tell NetworkManager to use `dnsmasq`: - - ```sh - echo -e "[main]\ndns=dnsmasq" | sudo tee /etc/NetworkManager/conf.d/openshift.conf - ``` - -2. Tell dnsmasq to use your cluster. The syntax is `server=//`. - - For this example: - - ```sh - echo server=/tt.testing/192.168.126.1 | sudo tee /etc/NetworkManager/dnsmasq.d/openshift.conf - ``` - -3. Reload NetworkManager to pick up the `dns` configuration change: `sudo systemctl reload NetworkManager` - -## Build the installer - -Set `TAGS=libvirt` to add support for libvirt; this is not enabled by default because libvirt is [development only](../../../README.md#supported-platforms). - -```sh -TAGS=libvirt hack/build.sh -``` - -## Run the installer - -With [libvirt configured](#install-and-enable-libvirt), you can proceed with [the usual quick-start](../../../README.md#quick-start). - -## Cleanup - -To remove resources associated with your cluster, run: - -```sh -openshift-install destroy cluster -``` - -You can also use [`virsh-cleanup.sh`](../../../scripts/maintenance/virsh-cleanup.sh), but note that it will currently destroy *all* libvirt resources. - -### Firewall - -With the cluster removed, you no longer need to allow libvirt nodes to reach your `libvirtd`. Restart -`firewalld` to remove your temporary changes as follows: - -```sh -sudo firewall-cmd --reload -``` - -## Exploring your cluster - -Some things you can do: - -### SSH access - -The bootstrap node, e.g. `test1-bootstrap.tt.testing`, runs the bootstrap process. You can watch it: - -```sh -ssh "core@${CLUSTER_NAME}-bootstrap.${BASE_DOMAIN}" -sudo journalctl -f -u bootkube -u openshift -``` - -You'll have to wait for etcd to reach quorum before this makes any progress. - -Using the domain names above will only work if you [set up the DNS overlay](#set-up-networkmanager-dns-overlay) or have otherwise configured your system to resolve cluster domain names. -Alternatively, if you didn't set up DNS on the host, you can use: - -```sh -virsh -c "${LIBVIRT_URI}" domifaddr "${CLUSTER_NAME}-master-0" # to get the master IP -ssh core@$MASTER_IP -``` - -Here `LIBVIRT_URI` is the libvirt connection URI which you [passed to the installer](../../../README.md#quick-start). - -### Inspect the cluster with kubectl - -You'll need a `kubectl` binary on your path and [the kubeconfig from your `cluster` call](../../../README.md#connect-to-the-cluster). - -```sh -export KUBECONFIG="${DIR}/auth/kubeconfig" -kubectl get --all-namespaces pods -``` - -Alternatively, you can run `kubectl` from the bootstrap or master nodes. -Use `scp` or similar to transfer your local `${DIR}/auth/kubeconfig`, then [SSH in](#ssh-access) and run: - -```sh -export KUBECONFIG=/where/you/put/your/kubeconfig -kubectl get --all-namespaces pods -``` - -## FAQ - -### Libvirt vs. AWS - -1. There isn't a load balancer on libvirt. - -## Troubleshooting - -If following the above steps hasn't quite worked, please review this section for well known issues. - -### Console doesn't come up - -In case of libvirt there is no wildcard DNS resolution and console depends on the route which is created by auth operator ([Issue #1007](https://github.com/openshift/installer/issues/1007)). -To make it work we need to first create the manifests and edit the `domain` for ingress config, before directly creating the cluster. - -- Add another domain entry in the openshift.conf which used by dnsmasq. -Here `tt.testing` is the domain which I choose when running the installer. -Here the IP in the address belong to one of the worker node. - -```console -$ cat /etc/NetworkManager/dnsmasq.d/openshift.conf -server=/tt.testing/192.168.126.1 -address=/.apps.tt.testing/192.168.126.51 -``` - -- An alternate method to specify the dnsmasq option, if the system is using libvirt version 5.6.0+, is to specify the option in the install config under the platform's network section in the following way. - -``` - platform: - libvirt: - network: - dnsmasqOptions: - - name: "address" - value: "/.apps.tt.testing/192.168.126.51" - if: tt0 -``` - -- Make sure you restart the NetworkManager after change in `openshift.conf`: - -```console -$ sudo systemctl restart NetworkManager -``` - -- Create the manifests: - -```console -$ openshift-install --dir $INSTALL_DIR create manifests -``` - -- Domain entry in cluster-ingress-02-config.yml file should not contain cluster name: - -```console -# Assuming `test1` as cluster name -$ sed -i 's/test1.//' $INSTALL_DIR/manifests/cluster-ingress-02-config.yml -``` - -- Start the installer to create the cluster: - -```console -$ openshift-install --dir $INSTALL_DIR create cluster -``` - -### Install throws an `Unable to resolve address 'localhost'` error - -If you're seeing an error similar to - -``` -Error: Error refreshing state: 1 error(s) occurred: - -* provider.libvirt: virError(Code=38, Domain=7, Message='Unable to resolve address 'localhost' service '-1': Servname not supported for ai_socktype') - - -FATA[0019] failed to run Terraform: exit status 1 -``` - -it is likely that your install configuration contains three backslashes after the protocol (e.g. `qemu+tcp:///...`), when it should only be two. - -### Random domain creation errors due to libvirt race condition - -Depending on your libvirt version you might encounter [a race condition][bugzilla_libvirt_race] leading to an error similar to: - -``` -* libvirt_domain.master.0: Error creating libvirt domain: virError(Code=43, Domain=19, Message='Network not found: no network with matching name 'test1'') -``` -This is also being [tracked on the libvirt-terraform-provider][tfprovider_libvirt_race] but is likely not fixable on the client side, which is why you should upgrade libvirt to >=4.5 or a patched version, depending on your environment. - -### MacOS support currently broken - -* Support for libvirt on Mac OS [is currently broken and being worked on][brokenmacosissue201]. - -### Error with firewall initialization on Arch Linux - -If you're on Arch Linux and get an error similar to - -``` -libvirt: “Failed to initialize a valid firewall backend” -``` - -or - -``` -error: Failed to start network default -error: internal error: Failed to initialize a valid firewall backend -``` - -please check out [this thread on superuser][arch_firewall_superuser]. - -### GitHub Issue Tracker - -You might find other reports of your problem in the [Issues tab for this repository][issues_libvirt] where we ask you to provide any additional information. -If your issue is not reported, please do. - -[arch_firewall_superuser]: https://superuser.com/questions/1063240/libvirt-failed-to-initialize-a-valid-firewall-backend -[brokenmacosissue201]: https://github.com/openshift/installer/issues/201 -[bugzilla_libvirt_race]: https://bugzilla.redhat.com/show_bug.cgi?id=1576464 -[issues_libvirt]: https://github.com/openshift/installer/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+libvirt -[libvirt_selinux_issues]: https://github.com/dmacvicar/terraform-provider-libvirt/issues/142#issuecomment-409040151 -[tfprovider_libvirt_race]: https://github.com/dmacvicar/terraform-provider-libvirt/issues/402#issuecomment-419500064 -[kvm-install]: http://www.linux-kvm.org/page/FAQ#Preparing_to_use_KVM diff --git a/docs/dev/libvirt/customization.md b/docs/dev/libvirt/customization.md deleted file mode 100644 index 0738cd4082..0000000000 --- a/docs/dev/libvirt/customization.md +++ /dev/null @@ -1,22 +0,0 @@ -# Libvirt Platform Customization - -The following options are available when using libvirt: - -- `platform.libvirt.network.if` - the network bridge attached to the libvirt network (`tt0` by default) - -## Examples - -An example `install-config.yaml` is shown below. This configuration has been modified to show the customization that is possible via the install config. - -```yaml -apiVersion: v1 -baseDomain: example.com -... -platform: - libvirt: - URI: qemu+tcp://192.168.122.1/system - network: - if: mybridge0 -pullSecret: '{"auths": ...}' -sshKey: ssh-ed25519 AAAA... -``` diff --git a/docs/dev/libvirt/load_balancer.md b/docs/dev/libvirt/load_balancer.md deleted file mode 100644 index b7036c77ad..0000000000 --- a/docs/dev/libvirt/load_balancer.md +++ /dev/null @@ -1,125 +0,0 @@ -# Load Balancer Setup - -The libvirt deployment does not deploy a load balancer for development purposes. - -This doc goes over an example configuration of HAProxy for doing local development. - -### Installing HAProxy -These instructions are for Fedora 34+. - -Install the RPM for `HAProxy`. -```sh -sudo dnf install haproxy -``` - -Configure `haproxy.cfg`. A default configuration follows, replace with the appropriate IP addresses for your environment: - -```sh -#--------------------------------------------------------------------- -# Global settings -#--------------------------------------------------------------------- -global - log 127.0.0.1 local2 - - chroot /var/lib/haproxy - pidfile /var/run/haproxy.pid - maxconn 4000 - user haproxy - group haproxy - daemon - - # turn on stats unix socket - stats socket /var/lib/haproxy/stats - - # utilize system-wide crypto-policies - # ssl-default-bind-ciphers PROFILE=SYSTEM - # ssl-default-server-ciphers PROFILE=SYSTEM - -#--------------------------------------------------------------------- -# common defaults that all the 'listen' and 'backend' sections will -# use if not designated in their block -#--------------------------------------------------------------------- -defaults - mode tcp - log global - option httplog - option dontlognull - option http-server-close - option forwardfor except 127.0.0.0/8 - option redispatch - retries 3 - timeout http-request 10s - timeout queue 1m - timeout connect 10s - timeout client 1m - timeout server 1m - timeout http-keep-alive 10s - timeout check 10s - maxconn 3000 - -#--------------------------------------------------------------------- -# main frontend which proxys to the backends -#--------------------------------------------------------------------- - -frontend api - bind :6443 - default_backend controlplaneapi - -frontend internalapi - bind :22623 - default_backend controlplaneapiinternal - -frontend secure - bind :443 - default_backend secure - -frontend insecure - bind :80 - default_backend insecure - -#--------------------------------------------------------------------- -# static backend -#--------------------------------------------------------------------- - -backend controlplaneapi - balance source - server bootstrap :6443 check - server master0 :6443 check - server master1 :6443 check - server master2 :6443 check - -backend controlplaneapiinternal - balance source - server bootstrap :22623 check - server master0 :22623 check - server master1 :22623 check - server master2 :22623 check - -backend secure - balance source - server compute0 :443 check - server compute1 :443 check - server compute2 :443 check - -backend insecure - balance source - server worker0 :80 check - server worker1 :80 check - server worker2 :80 check -``` - -Start and (optionally, enable) the systemd daemon. - -```sh -# If you want it enabled -sudo systemctl enable --now haproxy.service -# If you want to start it manually every time -sudo systemctl start haproxy.service -``` - -Ensure it's running by checking the systemd journal: - -```sh -# Hit Ctrl+C when done following the logs. -sudo journalctl -f -u haproxy.service -``` \ No newline at end of file diff --git a/docs/user/ovirt/OWNERS b/docs/user/ovirt/OWNERS deleted file mode 100644 index 0c8d12c37a..0000000000 --- a/docs/user/ovirt/OWNERS +++ /dev/null @@ -1,7 +0,0 @@ -# See the OWNERS docs: https://git.k8s.io/community/contributors/guide/owners.md -# This file just uses aliases defined in OWNERS_ALIASES. - -approvers: - - ovirt-approvers -reviewers: - - ovirt-reviewers diff --git a/docs/user/ovirt/customization.md b/docs/user/ovirt/customization.md deleted file mode 100644 index b1e66f2ce2..0000000000 --- a/docs/user/ovirt/customization.md +++ /dev/null @@ -1,104 +0,0 @@ -# oVirt Platform Customization - -Beyond the [platform-agnostic `install-config.yaml` properties](../customization.md#platform-customization), the installer supports additional, ovirt-specific properties. - -## Cluster-scoped properties - -* `ovirt_cluster_id` (required string): The Cluster where the VMs will be created. -* `ovirt_storage_domain_id` (required string): The Storage Domain ID where the VM disks will be created. -* `ovirt_network_name` (required string): The network name where the VM nics will be created. -* `vnicProfileID` (required string): The ID of the [vNic profile][vnic-profile] used for the VM network interfaces. - This can be inferred if the cluster network has a single profile. -* `api_vip` (required string): An IP address on the machineNetwork that will be assigned to the API VIP. -* `ingress_vip` (required string): An IP address on the machineNetwork that will be assigned to the Ingress VIP. - -## Machine pools - -* `cpu` (optional object): Defines the CPU of the VM. - * `cores` (required integer): The number of cores. Total vCPUS is cores * sockets. - * `sockets` (required integer): The number of sockets. Total vCPUS is cores * sockets. -* `memoryMB` (optional integer): Memory of the VM in MiB. -* `instanceTypeID` (optional string): The VM [instance-type][instance-type]. -* `osDisk` (optional string): Defines the first and bootable disk of the VM. - * `sizeGB` (required number): Size of the disk in GiB. -* `vmType` (optional string): The VM workload type. One of [high-performance][high-perf], server or desktop. - - -## Installing to Existing VPC & Subnetworks - -The installer can use an existing VPC and subnets when provisioning an OpenShift cluster. A VPC will be inferred from the provided subnets. For a standard installation, a private and public subnet should be specified. ([see example below](#pre-existing-vpc--subnets)). Both of the subnets must be within the IP range specified in `networking.machineNetwork`. - -## Examples - -Some example `install-config.yaml` are shown below. -For examples of platform-agnostic configuration fragments, see [here](../customization.md#examples). - -### Minimal - -An example minimal install config is: - -```yaml -apiVersion: v1 -baseDomain: example.com -metadata: - name: test-cluster -platform: - ovirt: - api_vip: 10.46.8.230 - ingress_vip: 10.46.8.232 - ovirt_cluster_id: 68833f9f-e89c-4891-b768-e2ba0815b76b - ovirt_storage_domain_id: ed7b0f4e-0e96-492a-8fff-279213ee1468 - ovirt_network_name: ovirtmgmt - vnicProfileID: 3fa86930-0be5-4052-b667-b79f0a729692 -pullSecret: '{"auths": ...}' -sshKey: ssh-ed25519 AAAA... -``` - -### Custom machine pools - -An example install config with custom machine pools: - -```yaml -apiVersion: v1 -baseDomain: example.com -controlPlane: - name: master - platform: - ovirt: - cpu: - cores: 4 - sockets: 2 - memoryMB: 65536 - osDisk: - sizeGB: 100 - vmType: high_performance - replicas: 3 -compute: -- name: worker - platform: - ovirt: - cpu: - cores: 4 - sockets: 4 - memoryMB: 65536 - osDisk: - sizeGB: 200 - vmType: high_performance - replicas: 5 -metadata: - name: test-cluster -platform: - ovirt: - api_vip: 10.46.8.230 - ingress_vip: 10.46.8.232 - ovirt_cluster_id: 68833f9f-e89c-4891-b768-e2ba0815b76b - ovirt_storage_domain_id: ed7b0f4e-0e96-492a-8fff-279213ee1468 - ovirt_network_name: ovirtmgmt - vnicProfileID: 3fa86930-0be5-4052-b667-b79f0a729692 -pullSecret: '{"auths": ...}' -sshKey: ssh-ed25519 AAAA... -``` - -[instance-type]: https://www.ovirt.org/develop/release-management/features/virt/instance-types.html -[vnic-profile]: https://www.ovirt.org/develop/release-management/features/sla/vnic-profiles.html -[high-perf]: https://www.ovirt.org/develop/release-management/features/virt/high-performance-vm.html diff --git a/docs/user/ovirt/images/oVirt-logo.png b/docs/user/ovirt/images/oVirt-logo.png deleted file mode 100644 index 3fd3211f93..0000000000 Binary files a/docs/user/ovirt/images/oVirt-logo.png and /dev/null differ diff --git a/docs/user/ovirt/install_ipi.md b/docs/user/ovirt/install_ipi.md deleted file mode 100644 index 6e6fe07898..0000000000 --- a/docs/user/ovirt/install_ipi.md +++ /dev/null @@ -1,252 +0,0 @@ -![oVirt Logo](./images/oVirt-logo.png#center) - -**Table of Contents** - -- [Install using oVirt platform provider](#install-using-ovirt-platform-provider) - * [Overview](#overview) - * [Prerequisite](#prerequisite) - * [Minimum resources](#minimum-resources) - * [Install](#install) - + [Minimum permission for installation](#minimum-permission-for-installation) - + [ovirt-config.yaml](#ovirt-configyaml) - + [ovirt-credentials](#ovirt-credentials) - + [Bootstrap VM](#bootstrap-vm) - + [Install using the wizard](#install-using-the-wizard) - + [Install in stages when customization is needed](#install-in-stages-when-customization-is-needed) - -# Install using oVirt platform provider - -## Overview - -This provider enables the OpenShift Installer to provision VM resources in an -oVirt data center, that will be used as worker and masters of the clusters. It -will also create the bootstrap machine, and the configuration needed to get -the initial cluster running by supplying DNS a service and load balancing, all -using static pods. -This work is related to the Bare-Metal provider because oVirt does not supply -DNS and LB services but is a platform provider. See also [Bare Metal IPI Networking Infrastructure] - - -## Prerequisite - -1. oVirt/RHV version 4.3.9.4 or later. -2. Allocate 2 IP on the VM network: - - IP for the internal kubernetes api, that all components will interact with - - IP for the Ingress, the load balancer in front of the cluster apps - To work with this provider one must supply 2 IPs excluded from the MAC range - in the virtualization env, where the cluster will run. Those IPs will be active - by keepalived, on, initially the bootstrap machine, and then the masters, after - a fail-over, when the bootstrap is killed. - Locate those IP's in the target network. If you want the network details, go to - oVirt's webadmin and look for the designated cluster details and its networks. - One way to check if an IP is in use is to check if it has ARP associated with it - perform this check while on one of the hosts that would run the VMs: - ```console - $ arp 10.35.1.19 - 10.35.1.1 (10.35.1.1) -- no entry - ``` -3. Name resolution of `api_vip` from your installing machine -The installer must resolve the `api_vip` during the installation, as it will -interact with the API to follow the cluster version progress. - - -## Minimum resources - -The default master/worker: -- 4 CPUs -- 16 RAM -- 120 GB disk - -For 3 masters/3 workers, the target Cluster **must have at least**: -- 96RAM -- 24vCPUs -- 720GiB storage -- Storage that is fast enough for etcd, [using-fio-to-tell-whether-your-storage-is-fast-enough-for-etcd](https://www.ibm.com/cloud/blog/using-fio-to-tell-whether-your-storage-is-fast-enough-for-etcd) - -> Worker count can be reduced to 2 in `install-config.yaml` in case needed. - -The cluster will create by default 1 bootstrap, 3 master, and 3 workers machines. -By the time the first worker is up the bootstrap VM should be destroyed, and this -is included in the minimum resources calculation. - - -## Install - -### Minimum permission for installation - -It's **not recommended** to users use admin@internal during the installation. Instead, create an exclusive user to install and manage OCP on oVirt. - -The minimum permissions are: -- DiskOperator -- DiskCreator -- UserTemplateBasedVm -- TemplateOwner -- TemplateCreator -- ClusterAdmin (on the specific cluster targeted for OCP deployment) - -There is an [ansible playbook available](https://github.com/oVirt/ocp-on-ovirt/tree/master/installer-tools/ocpadmin) which helps to setup an internal user and group with the minimum privileges to run the openshift-install on oVirt. - -### ovirt-config.yaml - -The ovirt-config.yaml is created under ${HOME}/.ovirt directory by the installer. -It contains all information how the installer connects to oVirt and can be re-used -if required to re-trigger a new installation. - -Below the description of all config options in ovirt-config.yaml. - -| Name | Value | Type | Example | -| ---------------|:------------------------------:|:--------:|:------------------------------------------------------------------------------------------------------:| -| ovirt_url | URL for Engine API | string | https://engine.fqdn.home/ovirt-engine/api | -| ovirt_fqdn | Engine FQDN | string | engine.fqdn.home | -| ovirt_username | User to connect with Engine | string | admin@internal | -| ovirt_password | Password for the user provided | string | superpass | -| ovirt_insecure | TLS verification disabled | boolean | false | -| ovirt_ca_bundle| CA Bundle | string | -----BEGIN CERTIFICATE----- MIIDvTCCAqWgAwIBAgICEAA.... ----- END CERTIFICATE ----- | -| ovirt_cafile | path to a file containing the | string | /path/to/ca.pm | -| | engine cert | | | -| ovirt_pem_url | PEM URL | string | https://engine.fqdn.home/ovirt-engine/services/pki-resource?resource=ca-certificate&format=X509-PEM-CA | - -### ovirt-credentials -During installation ${HOME}/.ovirt/ovirt-config.yaml is converted to a **secret** named as **ovirt-credentials** -and every openshift component with permission can use it. - -$ oc get secrets --all-namespaces | grep ovirt-credentials -``` -kube-system ovirt-credentials -openshift-machine-api ovirt-credentials -``` - -$ oc get secret ovirt-credentials -n kube-system -o yaml -$ oc get secret ovirt-credentials -n openshift-machine-api -o yaml -``` -apiVersion: v1 -data: - ovirt_ca_bundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR2VENDFTE1Ba0dBMVV... - ovirt_cafile: "" - ovirt_insecure: Zm2U= - ovirt_password: cmGF0 - ovirt_url: aHR0cHM3Z5lL2FwaQ== - ovirt_username: YWRtaJuYWw= -kind: Secret -metadata: - creationTimestamp: "2020-07-30T15:03:06Z" - managedFields: - - apiVersion: v1 - fieldsType: FieldsV1 - fieldsV1: - f:data: - .: {} - f:ovirt_ca_bundle: {} - f:ovirt_cafile: {} - f:ovirt_insecure: {} - f:ovirt_password: {} - f:ovirt_url: {} - f:ovirt_username: {} - f:type: {} - manager: cluster-bootstrap - operation: Update - time: "2020-07-30T15:03:06Z" - name: ovirt-credentials - namespace: kube-system - resourceVersion: "94" - selfLink: /api/v1/namespaces/kube-system/secrets/ovirt-credentials - uid: 642dbc91-12eb-4111-baa7-d79cbc9b79e4 -type: Opaque -``` - -### Bootstrap VM - -The bootstrap will perform ignition fully and will advertise the IP in the -pre-login msg. Go to Engine webadmin UI, and open the console of the bootstrap -VM to get it. - - -### Install using the wizard - -At this stage the installer can create a cluster by gathering all the information -using a wizard: -```console -$ openshift-install create cluster --dir=install_dir -? SSH Public Key /home/user/.ssh/id_dsa.pub -? Platform ovirt -? Engine FQDN[:PORT] [? for help] ovirt-engine-fqdn -? Enter ovirt-engine username admin@internal -? Enter password *** -? oVirt cluster xxxx -? oVirt storage xxxx -? oVirt network xxxx -? Internal API virtual IP 10.0.0.1 -? Ingress virtual IP 10.0.0.3 -? Base Domain example.org -? Cluster Name test -? Pull Secret [? for help] -INFO Consuming Install Config from target directory -INFO Creating infrastructure resources... -INFO Waiting up to 20m0s for the Kubernetes API at https://api.test.example.org:6443... -INFO API v1.17.1 up -INFO Waiting up to 40m0s for bootstrapping to complete... -INFO Destroying the bootstrap resources... -INFO Waiting up to 30m0s for the cluster at https://api.test.example.org:6443 to initialize... -INFO Waiting up to 10m0s for the openshift-console route to be created... -INFO Install complete! -INFO To access the cluster as the system:admin user when using 'oc', run - export KUBECONFIG=/home/user/install_dir/auth/kubeconfig -INFO Access the OpenShift web-console here: https://console-openshift-console.apps.test.example.org -INFO Login to the console with user: kubeadmin, password: xxxxxxxxx -``` - - -### Install in stages when customization is needed - -Start the installation by creating an `install-config` interactively, using a work-dir: - -```console -$ openshift-install create install-config --dir=install_dir -``` - -The resulting `install_dir/install-config.yaml` can be further customized if needed. -For general customization please see [docs/user/customization.md](../customization.md#platform-customization) -For ovirt-specific see [customization.md](./customization.md) -Continue the installation using the install-config in the new folder `install_dir` - -```console -$ openshift-install create cluster --dir=install_dir -``` - -When the all prompts are done the installer will create ${HOME}/.ovirt/ovirt-config.yaml -containing all required information about the connection with Engine. -The installation process will create a temporary VM which will trigger bootstrap VM -for later create three masters nodes. The masters nodes will create all services and -checks required. Finally, the cluster will create the three workers node. - -In the end the installer finishes and the cluster should be up. - -To access the cluster as the system:admin user: - -```console -$ export KUBECONFIG=$PWD/install_dir/auth/kubeconfig -$ oc get nodes -``` - -[Bare Metal IPI Networking Infrastructure]: https://github.com/openshift/installer/blob/master/docs/design/baremetal/networking-infrastructure.md - -#### Installing OpenShift on RHV/oVirt in *insecure* mode - - - -Starting OpenShift 4.7 we are sunsetting the “insecure” option from the OpenShift Installer. Starting with this version, the installer only supports installation methods from the user interface that lead to using verified certificates. - -This change also means that setting up the CA certificate for RHV is no longer required before running the installer. The installer will ask you for confirmation about the certificate and store the CA certificate for use during the installation. - -Should you, nevertheless, require an installation without certificate verification you can create a file named ovirt-config.yaml in the .ovirt directory in your home directory (~/.ovirt/ovirt-config.yaml) before running the installer with the following content: - -```yaml -ovirt_url: https://ovirt.example.com/ovirt-engine/api -ovirt_fqdn: ovirt.example.com -ovirt_pem_url: "" -ovirt_username: admin@internal -ovirt_password: super-secret-password -ovirt_insecure: true -``` - -Please note that this option is **not recommended** as it will allow a potential attacker to perform a Man-in-the-Middle attack and capture sensitive credentials on the network. diff --git a/docs/user/ovirt/install_upi.md b/docs/user/ovirt/install_upi.md deleted file mode 100644 index 8e1a76f320..0000000000 --- a/docs/user/ovirt/install_upi.md +++ /dev/null @@ -1,820 +0,0 @@ -# Install: oVirt/RHV User-Provided Infrastructure - -This User-Provisioned Infrastructure (UPI) process is based on several and -customizable steps to allow the user to integrate into an existing infrastructure. - -Creating and configuring oVirt/RHV resources is the responsibility of the user -deploying OpenShift. - -The OpenShift installer will still be used in several steps of the process to generate -mandatory ignition files and to monitor the installation process itself. - -## Table of Contents - -* [Prerequisites](#prerequisites) - * [Ansible and oVirt roles](#ansible-and-ovirt-roles) - * [Network Requirements](#network-requirements) - * [Load Balancers](#load-balancers) - * [DNS](#dns) - * [RHCOS image](#rhcos-image) -* [Getting Ansible playbooks](#getting-ansible-playbooks) -* [Assets directory](#assets-directory) -* [Inventory explained](#inventory-explained) - * [General section](#general-section) - * [RHCOS section](#rhcos-section) - * [Profiles and VMs](#profiles-and-vms) - * [Profiles section](#profiles-section) - * [VMs section](#vms-section) -* [Install Config](#install-config) - * [Set compute replicas to zero](#set-compute-replicas-to-zero) - * [Set machine network](#set-machine-network) - * [Set platform to none](#set-platform-to-none) -* [Manifests](#manifests) - * [Set control-plane nodes unschedulable](#set-control-plane-nodes-unschedulable) -* [Ignition configs](#ignition-configs) -* [Create templates and VMs](#create-templates-and-vms) -* [Bootstrap](#bootstrap) -* [Master nodes](#master-nodes) -* [Wait for Control Plane](#wait-for-control-plane) -* [OpenShift API](#openshift-api) -* [Retire Bootstrap](#retire-bootstrap) -* [Worker nodes](#worker-nodes) - * [Approve CSRs](#approve-csrs) -* [Wait for Installation Complete](#wait-for-installation-complete) -* [Destroy OpenShift cluster](#destroy-openshift-cluster) - -# Prerequisites -The [inventory.yml](../../../upi/ovirt/inventory.yml) file all the variables used by this installation can be customized as per -user needs. -The requirements for the UPI in terms of minimum resources are broadly the same as -the [IPI](./install_ipi.md#minimum-resources). - -- oVirt/RHV account stored in the [ovirt-config.yaml](https://github.com/openshift/installer/blob/master/docs/user/ovirt/install_ipi.md#ovirt-configyaml) - - this file is generated by the `openshift-install` binary installer following the CLI wizard. -- Name of the oVirt/RHV cluster to use - - contained in the [inventory.yml](../../../upi/ovirt/inventory.yml) and input in the `openshift-install`. -- A base domain name of the OpenShift cluster - - input in the `openshift-install`. -- A name of the OpenShift cluster - - input in the `openshift-install`. -- OpenShift Pull Secret - - input in the `openshift-install`. -- A DNS zone - - to configure the resolution names for the OpenShift cluster base domain. -- LoadBalancers - - for bootstrap and control-plane machines. - - for machines running the ingress router (usually compute nodes). - -## Ansible and oVirt roles -To use the UPI process described here the following are required: - -- Python3 - -**Note**: -Currently most of Linux distros provides Python 3 by default. - -- Ansible - -**Note for CentOS users**: -Depending on which version the system is running will be required [epel-release](https://fedoraproject.org/wiki/EPEL) repo enabled. - -``` - $ sudo dnf install ansible -``` -- python3-ovirt-engine-sdk4 - -``` - $ sudo dnf install python3-ovirt-engine-sdk4 -``` -- ovirt.image-template Ansible role (distributed as ovirt-ansible-image-template package on oVirt Manager) -- ovirt.vm-infra Ansible role (distributed as ovirt-ansible-vm-infra package on oVirt Manager) - -``` - $ sudo ansible-galaxy install ovirt-ansible-vm-infra ovirt-ansible-image-template -``` - - -To be sure to follow the UPI installation process, Ansible scripts and the binary openshift-install -should be executed from the oVirt/RHV Manager or from a machine with access to the REST API of the -oVirt/RHV Manager and with all the oVirt roles available (installed by default on the Manager -machine). - -## Network Requirements -The UPI installation process assumes that the user satisfies some network requirements providing them through the -existing infrastructure. -During the boot the RHCOS based machines require an IP address in `initramfs` in order to establish a network connection to get their -ignition config files. -One of the recommended ways is to use a DHCP server to manage the machines in the long-term, maybe configuring the DHCP server -itself to provide persistent IP addresses and host names to the cluster machines. - -Network connectivity between machines should be configured to allow cluster components to communicate: - -- Kubernetes NodePort - Machines require connectivity to every other machine for OpenShift platform components through the port range `30000`-`32767` . - -- OpenShift reserved - Connectivity to reserved port ranges `10250`-`10259` and `9000`-`9999` should be granted on every machine. - -- Machines to control-plane - Connectivity to ports on ranges `2379`-`2380` (for etcd, peer and metrics) is required for control-plane machines and on - port `6443` for Kubernetes API. - - -### Load Balancers -Before installing the OpenShift Container Platform, two load balancers (layer-4) must be provided by the user infrastructure, -one for the API and one for the Ingress Controller (to allow ingress to applications). - -- Load balancer for port `6443` and `22623` on control-plane and bootstrap machines (the bootstrap can be removed after control-plane - initialization completes). - The `6443` must be both internal and external reachable and is needed by the Kubernetes API server. - Port `22623` must be accessible to nodes within the cluster. - -- Load balancer for port `443` and `80` for machines running the ingress router (usually worker nodes in the default configuration). - Both ports must be accessible from within and outside the cluster. - -**NOTE**: the rules above can also be set on the same load balancer server. - - -### DNS -The UPI installation process requires the user to setup the existing infrastructure provided DNS to allow the correct resolution of -the main components and services - -- Kubernetes API - DNS records `api..` (internal and external resolution) and `api-int..` - (internal resolution) must be added to point to the Load balancer targeting the control plane machines. - -- OpenShift routes - A DNS record `*.apps..` must be provided to point to the Load balancer configured to manage the - traffic for the ingress router (ports `443` and `80` of the compute machines). - -**NOTE**: the DNS records above may also point to the same IP in case you are using only one load balancer configured with the rules described -in the [previous section](#load-balancers). - -## RHCOS image -This UPI installation process requires a proper RHCOS (Red Hat Enterprise Linux CoreOS) image URL to be set in the [inventory.yml](../../../upi/ovirt/inventory.yml) file. - -The RHCOS images can be found [here](https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/) and you have to choose -the URL related to the `OpenStack` qcow2 image type, like in the example below - -``` -https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/pre-release/4.6.0-0.nightly-2020-07-16-122837/rhcos-4.6.0-0.nightly-2020-07-16-122837-x86_64-openstack.x86_64.qcow2.gz -``` - -The version of the image should be chosen according to the OpenShift version you're about to install (in general less than or equal to the OCP -version). -Once you have the URL set in the [inventory.yml](../../../upi/ovirt/inventory.yml) a dedicated Ansible playbook will be in charge to download the `qcow2.gz` file, uncompress it -in a specified folder and use it to create oVirt/RHV templates. - -## Getting Ansible playbooks -All the Ansible playbooks used in this UPI installation process are available [here](https://github.com/openshift/installer/tree/master/upi/ovirt) -and can be downloaded with the following utility script - -```sh -RELEASE="release-4.6"; \ -curl -L -X GET https://api.github.com/repos/openshift/installer/contents/upi/ovirt\?ref\=${RELEASE} | -grep 'download_url.*\.yml' | -awk '{ print $2 }' | sed -r 's/("|",)//g' | -xargs -n 1 curl -O -``` - -Different versions of the oVirt UPI playbooks can be downloaded changing the RELEASE environment variable to the desired branch -(please be aware that this UPI work started with the `release-4.6`). - -## Assets directory -Before proceeding with the installation is **required** to set an environment variable with the path (absolute or relative according to your preferences) -of the directory in which the `openshift-install` command will put all the artifacts and that we'll also refer to in the [inventory.yml](../../../upi/ovirt/inventory.yml)`. - -```sh -$ export ASSETS_DIR=./wrk -``` - -## Inventory Explained -This section shows an example of [inventory.yml](../../../upi/ovirt/inventory.yml), used to specify the variables needed for the UPI installation process, with a brief explanation of the sections included. - -```YAML ---- -all: - vars: - - # --- - # General section - # --- - ovirt_cluster: "Default" - ocp: - assets_dir: "{{ lookup('env', 'ASSETS_DIR') }}" - ovirt_config_path: "{{ lookup('env', 'HOME') }}/.ovirt/ovirt-config.yaml" - - # --- - # RHCOS section - # --- - rhcos: - image_url: "https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/pre-release/latest-4.6/rhcos-4.6.0-0.nightly-2020-07-16-122837-x86_64-openstack.x86_64.qcow2.gz" - local_cmp_image_path: "/tmp/rhcos.qcow2.gz" - local_image_path: "/tmp/rhcos.qcow2" - - # --- - # Profiles section - # --- - control_plane: - cluster: "{{ ovirt_cluster }}" - memory: 16GiB - sockets: 4 - cores: 1 - template: rhcos_tpl - operating_system: "rhcos_x64" - type: high_performance - graphical_console: - headless_mode: false - protocol: - - spice - - vnc - disks: - - size: 120GiB - name: os - interface: virtio_scsi - storage_domain: depot_nvme - nics: - - name: nic1 - network: lab - profile: lab - - compute: - cluster: "{{ ovirt_cluster }}" - memory: 16GiB - sockets: 4 - cores: 1 - template: worker_rhcos_tpl - operating_system: "rhcos_x64" - type: high_performance - graphical_console: - headless_mode: false - protocol: - - spice - - vnc - disks: - - size: 120GiB - name: os - interface: virtio_scsi - storage_domain: depot_nvme - nics: - - name: nic1 - network: lab - profile: lab - - # --- - # VMs section - # --- - vms: - - name: "{{ metadata.infraID }}-bootstrap" - ocp_type: bootstrap - profile: "{{ control_plane }}" - type: server - - name: "{{ metadata.infraID }}-master0" - ocp_type: master - profile: "{{ control_plane }}" - - name: "{{ metadata.infraID }}-master1" - ocp_type: master - profile: "{{ control_plane }}" - - name: "{{ metadata.infraID }}-master2" - ocp_type: master - profile: "{{ control_plane }}" - - name: "{{ metadata.infraID }}-worker0" - ocp_type: worker - profile: "{{ compute }}" - - name: "{{ metadata.infraID }}-worker1" - ocp_type: worker - profile: "{{ compute }}" - - name: "{{ metadata.infraID }}-worker2" - ocp_type: worker - profile: "{{ compute }}" -``` - -### General section -Variables in this section are mandatory and allow the user to specify - -* `ovirt_cluster`: the name of the ovirt cluster in which you'll install the OCP cluster. -* `ocp.assets_dir`: is the path of the folder in which the `openshift-install` command will put all the files built in different stages. -* `ocp.ovirt_config_path`: path of the `ovirt-config.yaml`, generated by the `openshift-install` in the [first stage](#install-config), containing the ovirt credentials -(necessary to interact with the oVirt/RHV Manager REST API). - -### RHCOS section -The `rhcos` variable contains the RHCOS public URL (`image_url`) for downloading the image in the local specified path (`local_cmp_image_path`) and -uncompressing it (in a file described by `local_image_path`) before being able to use it. - -```YAML - rhcos: - image_url: "https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/pre-release/4.6.0-0.nightly-2020-07-16-122837/rhcos-4.6.0-0.nightly-2020-07-16-122837-x86_64-openstack.x86_64.qcow2.gz" - local_cmp_image_path: "/tmp/rhcos.qcow2.gz" - local_image_path: "/tmp/rhcos.qcow2" -``` - -Please refer to the [specific paragraph](#rhcos-image) to learn more about which version of RHCOS image to choose and where to find it. - -### Profiles and VMs -The latest and important part of the [inventory.yml](../../../upi/ovirt/inventory.yml) is related to `profiles` and `vms` definition using the capabilities -offered by the ovirt.vm-infra role, whose documentation can be found [here](#https://github.com/oVirt/ovirt-ansible-vm-infra). -In the following paragraphs we'll explain the meaning of the basic parameters from the OCP point of view. - -#### Profiles section -This section is mainly composed of two variable, `control_plane` and `compute`, that define the two different profiles used respectively -for masters (and bootstrap) and workers. - -```YAML - control_plane: - cluster: "{{ ovirt_cluster }}" - memory: 16GiB - sockets: 4 - cores: 1 - template: rhcos_tpl - operating_system: "rhcos_x64" - type: high_performance - disks: - - size: 120GiB - name: os - interface: virtio_scsi - storage_domain: depot_nvme - nics: - - name: nic1 - network: lab - profile: lab - - compute: - cluster: "{{ ovirt_cluster }}" - memory: 16GiB - sockets: 4 - cores: 1 - template: worker_rhcos_tpl - operating_system: "rhcos_x64" - type: high_performance - disks: - - size: 120GiB - name: os - interface: virtio_scsi - storage_domain: depot_nvme - nics: - - name: nic1 - network: lab - profile: lab -``` - -The user can customize the parameters of both profiles according to the needs and the minimum requirements. - -* `cluster`: it's already set according to the value of the variable in the [General Section](#general-section). -* `memory, sockets, cores`: mandatory parameters necessary to define the common specs of the VMs generated from this profile. -* `template`: name of the template that the virtual machine will be based on (refer to [this](#create-templates-and-vms) for further information about templates). -* `operating_system`: sets the vm OS type. With oVirt/RHV 4.4 it's mandatory to use the value `rhcos_x64` to allow the `ignition script` -to be correctly passed to the VM. -* `type`: it's the type that the VM will have once created. -* `disks`: in this section the specs of the disk must be set according to the basic requirements of OCP in terms of capacity and storage -performances. It's possible to choose different storage-domains for control_plane and compute nodes. -* `nics`: defines the specs like the name of the nic and the network that the Vms will use. The virtual network interface profile can also be specified. -The MAC address will be taken from the oVirt/RHV MAC pool. - -#### VMs section -In this last section of the [inventory.yml](../../../upi/ovirt/inventory.yml) there's the definition of the `vms` variable, containing all the node instance that the user plans to create -to deploy the OCP cluster (remember that there are minimum requirements in terms of number of master and worker nodes). - -In the last section there's the list of all the vms that will be created and their role expressed by the `ocp_type`. - -```YAML - vms: - - name: "{{ metadata.infraID }}-bootstrap" - ocp_type: bootstrap - profile: "{{ control_plane }}" - type: server - - name: "{{ metadata.infraID }}-master0" - ocp_type: master - profile: "{{ control_plane }}" - - name: "{{ metadata.infraID }}-master1" - ocp_type: master - profile: "{{ control_plane }}" - - name: "{{ metadata.infraID }}-master2" - ocp_type: master - profile: "{{ control_plane }}" - - name: "{{ metadata.infraID }}-worker0" - ocp_type: worker - profile: "{{ compute }}" - - name: "{{ metadata.infraID }}-worker1" - ocp_type: worker - profile: "{{ compute }}" -``` - -As you can see above the `vms` variable is basically defined by a list of elements each one with at least three mandatory attributes - -* `name`: name of the virtual machine to create. -* `ocp_type`: the role of the virtual machine in the OCP cluster (possible values are `bootstrap`, `master`, `worker`). -* `profile`: name of the profile (`control_plane` or `compute`) from which to inherit common specs - -Additional attributes can be specified to override the ones defined in the inheriting profile - -* `type`: is re-defined as `server` in the `bootstrap` vm - -It's also possible to use all the attributes documented in the [oVirt.vm-infra role](#https://github.com/oVirt/ovirt-ansible-vm-infra) -(e.g.: fixed MAC address for each vm that could help to assign permanent IP through a DHCP). - -**Note**: -Looking at the `vms` attribute `name` setting, you can see that we are using the variable `metadata.infraID` whose value is -obtained parsing the `metadata.json` file generated using the command `openshift-install create ignition-configs` (read more about it -[here](#ignition-configs)). -There's a specific set of Ansible tasks ([common-auth.yml](https://github.com/openshift/installer/blob/master/upi/ovirt/common-auth.yml)) included in all -the UPI playbooks that contains the code to read the `infraID` from the specific file located in the `ocp.assets_dir` - -```YAML ---- -- name: include metadata.json vars - include_vars: - file: "{{ ocp.assets_dir }}/metadata.json" - name: metadata - - ... - - -``` - -## Install config -Run the `openshift-install` to create the initial `install-config` using as assets directory the same that is specified in -the [inventory.yml](../../../upi/ovirt/inventory.yml) (`working_path`). - -```sh -$ openshift-install create install-config --dir $ASSETS_DIR -? SSH Public Key /home/user/.ssh/id_dsa.pub -? Platform -? Engine FQDN[:PORT] [? for help] -? Enter ovirt-engine username -? Enter password <******> -? oVirt cluster -? oVirt storage -? oVirt network -? Internal API virtual IP <172.16.0.252> -? Ingress virtual IP <172.16.0.251> -? Base Domain -? Cluster Name -? Pull Secret [? for help] <********> -``` - -*Internal API* and *Ingress* are the IPs added following the above DNS instructions - -- `api.ocp4.example.org`: 172.16.0.252 -- `*.apps.ocp4.example.org`: 172.16.0.251 - -*Cluster Name* (`ocp4`) and *Base Domain* (`example.org`) joint together will form the FQDN of the OCP cluster -used to expose the API interface (`https://api.ocp4.example.org:6443/`) -and the newly created applications (e.g. `https://console-openshift-console.apps.ocp4.example.org`). - -You can obtain a new Pull secret from [here](https://console.redhat.com/openshift/install/pull-secret). - -The result of this first step is the creation of a `install-config.yaml` in the specified assets directory: - -```sh -$ tree -. -└── wrk - └── install-config.yaml -``` - -File `$HOME/.ovirt/ovirt-config.yaml` was also created for you by the `openshift-install` containing all the connection -parameters needed to reach the oVirt/RHV engine and use its REST API. - -**NOTE:** -Some of the parameters added during the `openshift-install` workflow, in particular the `Internal API virtual IP` and -`Ingress virtual IP`, will not be used because already configured in your infrastructure DNS (see [DNS](#dns) section). -Other parameters like `oVirt cluster`, `oVirt storage`, `oVirt network`, will be used as specified in the [inventory.yml](../../../upi/ovirt/inventory.yml) -and removed from the `install-config.yaml` with the previously mentioned `virtual IPs`, using a script reported in a -[section below](#set-platform-to-none). - -### Set compute replicas to zero -Machine API will not be used by the UPI to create nodes, we'll create compute nodes explicitly with Ansible scripts. -Therefore we'll set the number of compute nodes to zero replicas using the following python script: - -```sh -$ python3 -c 'import os, yaml -path = "%s/install-config.yaml" % os.environ["ASSETS_DIR"] -conf = yaml.safe_load(open(path)) -conf["compute"][0]["replicas"] = 0 -open(path, "w").write(yaml.dump(conf, default_flow_style=False))' -``` - -**NOTE**: All the Python snippets in this document work with both Python 3 and Python 2. - -### Set machine network -OpenShift installer sets a default IP range for nodes and we need to change it according to our infrastructure. -We'll set the range to `172.16.0.0/16` (we can use the following python script for this): - -```sh -$ python3 -c 'import os, yaml -path = "%s/install-config.yaml" % os.environ["ASSETS_DIR"] -conf = yaml.safe_load(open(path)) -conf["networking"]["machineNetwork"][0]["cidr"] = "172.16.0.0/16" -open(path, "w").write(yaml.dump(conf, default_flow_style=False))' -``` - -### Set platform to none -The UPI for oVirt/RHV is similar to the bare-metal installation process and for now we don't need the specific ovirt -platform section in the `install-config.yaml`, all the settings needed are specified in the [inventory.yml](../../../upi/ovirt/inventory.yml). -We'll remove the section - -```sh -$ python3 -c 'import os, yaml -path = "%s/install-config.yaml" % os.environ["ASSETS_DIR"] -conf = yaml.safe_load(open(path)) -platform = conf["platform"] -del platform["ovirt"] -platform["none"] = {} -open(path, "w").write(yaml.dump(conf, default_flow_style=False))' -``` - -## Manifests -Editing manifests is an action required for the UPI and to generate them we can use again the binary installer - -```sh -$ openshift-install create manifests --dir $ASSETS_DIR -``` -```sh -$ tree -. -└── wrk - ├── manifests - │   ├── 04-openshift-machine-config-operator.yaml - │   ├── cluster-config.yaml - │   ├── cluster-dns-02-config.yml - │   ├── cluster-infrastructure-02-config.yml - │   ├── cluster-ingress-02-config.yml - │   ├── cluster-network-01-crd.yml - │   ├── cluster-network-02-config.yml - │   ├── cluster-proxy-01-config.yaml - │   ├── cluster-scheduler-02-config.yml - │   ├── cvo-overrides.yaml - │   ├── kube-cloud-config.yaml - │   ├── kube-system-configmap-root-ca.yaml - │   ├── machine-config-server-tls-secret.yaml - │   └── openshift-config-secret-pull-secret.yaml - └── openshift - ├── 99_kubeadmin-password-secret.yaml - ├── 99_openshift-cluster-api_master-user-data-secret.yaml - ├── 99_openshift-cluster-api_worker-user-data-secret.yaml - ├── 99_openshift-machineconfig_99-master-ssh.yaml - ├── 99_openshift-machineconfig_99-worker-ssh.yaml - └── openshift-install-manifests.yaml -``` - -The command above will write manifests consuming the `install-config.yaml` and will show a warning message. -If you plan on reusing the `install-config.yaml` file, back it up before you generate manifests. - -```sh -INFO Consuming Install Config from target directory -WARNING Making control-plane schedulable by setting MastersSchedulable to true for Scheduler cluster settings -``` - -### Set control-plane nodes unschedulable -Setting compute replicas to zero makes control-plane nodes schedulable which is something we don't want for now -(router pods can run also on control-plane nodes but there are some Kubernetes limitation that will prevent those pods -to be reachable by the ingress load balancer). -Setting the control-plan as unschedulable means modifying the `manifests/cluster-scheduler-02-config.yml` setting -`masterSchedulable` to `False`. - -```sh -$ python3 -c 'import os, yaml -path = "%s/manifests/cluster-scheduler-02-config.yml" % os.environ["ASSETS_DIR"] -data = yaml.safe_load(open(path)) -data["spec"]["mastersSchedulable"] = False -open(path, "w").write(yaml.dump(data, default_flow_style=False))' -``` - -## Ignition configs -The next step is the one needed to build [Ignition](https://coreos.com/ignition/docs/latest/) files from the manifests just modified. -Ignition files have to be fetched by RHCOS machine initramfs to perform configurations that will bring to a live final node. -We will use again the binary installer. - -```sh -$ openshift-install create ignition-configs --dir $ASSETS_DIR -``` - -```sh -$ tree -. -└── wrk - ├── auth - │   ├── kubeadmin-password - │   └── kubeconfig - ├── bootstrap.ign - ├── master.ign - ├── metadata.json - └── worker.ign -``` -Other than the ignition files the installer generated - -- `auth` folder containing the admin credentials necessary to - connect to the cluster via the `oc` or `kubectl` CLI utilities. -- `metadata.json` with information like the OCP cluster name, OCP cluster ID and the `infraID` - (generated for the current running installation). - -The `infraID` will be used by the UPI Ansible playbooks as prefix for the VMs created during the installation -process avoiding name clashes in case of multiple installations in the same oVirt/RHV cluster. - -**Note:** certificates contained into ignition config files expire after 24 hours. You must complete the cluster installation -and keep the cluster running for 24 hours in a non-degraded state to ensure that the first certificate rotation has finished. - - -## Create templates and VMs -After having checked that all our variables in the [inventory.yml](../../../upi/ovirt/inventory.yml) fit the needs, we can run the first of our Ansible provisioning -playbooks. - -```sh -$ ansible-playbook -i inventory.yml create-templates-and-vms.yml -``` - -This playbook will use the connection parameters for the oVirt/RHV engine stored in the `$HOME/.ovirt/ovirt-config.yaml` -reading also the `metadata.json` from the assets directory. - -According to the variables - -```YAML - rhcos: - image_url: "https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/pre-release/4.6.0-0.nightly-2020-07-16-122837/rhcos-4.6.0-0.nightly-2020-07-16-122837-x86_64-openstack.x86_64.qcow2.gz" - local_cmp_image_path: "/tmp/rhcos.qcow2.gz" - local_image_path: "/tmp/rhcos.qcow2" -``` - -the RHCOS image will be downloaded (in case not already existing locally), stored locally and extracted to be uploaded on the oVirt/RHV node -and used for template creation. -The user can check the RHCOS image for the OpenShift version he want to use from [here](https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/). - -Templates will be created according to the names specified in the [inventory.yml](../../../upi/ovirt/inventory.yml) for the `control_plane` and `compute` profile (in case of -different names two different templates will be created). -In case the user want to have different templates for each OCP installation in the same cluster, it is possible to customize the template name -in the [inventory.yml](../../../upi/ovirt/inventory.yml) prepending the `infraID` (like it is done for VMs' names). - -```YAML - control_plane: - cluster: "{{ ovirt_cluster }}" - memory: 16GiB - sockets: 4 - cores: 1 - template: "{{ metadata.infraID }}-rhcos_tpl" - operating_system: "rhcos_x64" - ... -``` - -At the end of the execution the VMs specified will be created and left in stopped mode. This allows the user to fetch any information from the -VMs that can help configuring other infrastructure elements (e.g.: getting MAC addresses to feed a DHCP server to assign permanent IPs). - -## Bootstrap - -```sh -$ ansible-playbook -i inventory.yml bootstrap.yml -``` -The playbook starts the bootstrap VM passing it the `bootstrap.ign` ignition file contained in the assets directory. That will allow the bootstrap -node to configure itself and be ready to serve ignition files for the master nodes. -The user can check the console inside oVirt/RHV UI or can connect to the VM via SSH. -Running the following command from inside the bootstrap VM enables them to closely monitor the bootstrap process. - -```sh -$ ssh core@ -[core@ocp4-lk6b4-bootstrap ~]$ journalctl -b -f -u release-image.service -u bootkube.service -``` - -## Master nodes - -```sh -$ ansible-playbook -i inventory.yml masters.yml -``` - -The `masters.yml` will start our control-plane made of three masters (but it can be customized) passing the `master.ign` ignition file to each of the VMs. -`master.ign` ignition file contains a directive that instructs masters to fetch the ignition from the URL - -```sh -https://api-int.ocp4.example.org:22623/config/master -``` - -targeted by the Load balancer that manages the traffic on port `22623` (accessible only inside the cluster) driving it to masters and bootstrap. - -## Wait for Control Plane -The user can monitor the control-plane bootstrap process with the following command: - -```sh -$ openshift-install wait-for bootstrap-complete --dir $ASSETS_DIR -``` - -After some time the output of the command will be the following - -```sh -INFO API v1.18.3+b74c5ed up -INFO Waiting up to 40m0s for bootstrapping to complete... -``` - -After all the pods on master nodes and the etcd will be up and running the installer will show the following output: - -```sh -INFO It is now safe to remove the bootstrap resources -``` - -## OpenShift API -The OpenShift API can be accessed via the `oc` or `kubectl` using the admin credentials contained in the assets directory -in the file `auth/kubeconfig`: - -```sh -$ export KUBECONFIG=$ASSETS_DIR/auth/kubeconfig -$ oc get nodes -$ oc get pods -A -``` -## Retire Bootstrap -After the `wait-for` command says that the bootstrap process is complete, it is possible to remove the bootstrap VM - -```sh -$ ansible-playbook -i inventory.yml retire-bootstrap.yml -``` - -and the user can remove it also from the Load balancer directives. - -## Worker nodes -```sh -$ ansible-playbook -i inventory.yml workers.yml -``` - -This is similar to what we did for masters but in this case workers won't automatically join the cluster, we'll need to -approve their respective pending CSRs (Certificate Signing Requests). - -### Approve CSRs -CSRs for nodes joining the cluster will need to be approved by the administrator. The following command helps to list -pending requests - -```sh -$ oc get csr -A -``` - -Eventually one pending CSR per node will be shown - -```sh -NAME AGE SIGNERNAME REQUESTOR CONDITION -csr-2lnxd 63m kubernetes.io/kubelet-serving system:node:ocp4-lk6b4-master0.ocp4.example.org Approved,Issued -csr-hff4q 64m kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued -csr-hsn96 60m kubernetes.io/kubelet-serving system:node:ocp4-lk6b4-master2.ocp4.example.org Approved,Issued -csr-m724n 6m2s kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Pending -csr-p4dz2 60m kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued -csr-t9vfj 60m kubernetes.io/kubelet-serving system:node:ocp4-lk6b4-master1.ocp4.example.org Approved,Issued -csr-tggtr 61m kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued -csr-wcbrf 7m6s kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Pending -``` - -To filter and watch pending CSRs the following command can be used - -```sh -$ watch "oc get csr -A | grep pending -i" -``` -that refresh the output every two seconds -```sh -Every 2.0s: oc get csr -A | grep pending -i - -csr-m724n 10m kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Pending -csr-wcbrf 11m kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Pending -``` - -Every `Pending` request should be inspected - -```sh -$ oc describe csr csr-m724n -Name: csr-m724n -Labels: -Annotations: -CreationTimestamp: Sun, 19 Jul 2020 15:59:37 +0200 -Requesting User: system:serviceaccount:openshift-machine-config-operator:node-bootstrapper -Signer: kubernetes.io/kube-apiserver-client-kubelet -Status: Pending -Subject: - Common Name: system:node:ocp4-lk6b4-worker1.ocp4.example.org - Serial Number: - Organization: system:nodes -Events: -``` -and finally approved - -```sh -$ oc adm certificate approve csr-m724n -``` - -After approving the first requests, another CSR per each worker will be issued and will need to be approved to have worker nodes not only -joining the cluster, but also becoming `Ready` and having pods scheduled on them. - -## Wait for Installation Complete -The following command can now be run to follow the installation process till it's complete: - -```sh -$ openshift-install wait-for install-complete --dir $ASSETS_DIR --log-level debug -``` - -Eventually it will give as output - -- URL to reach the OpenShift Console (web UI). -- Username and password for the admin login. - - -## Destroy OpenShift cluster - -```ssh -$ ansible-playbook -i inventory.yml \ - retire-bootstrap.yml \ - retire-masters.yml \ - retire-workers.yml -``` - -Removing DNS added records, Load balancers and any other infrastructure configuration is left to the user.