openshift-docs/modules/storage-persistent-storage-pv.adoc

// Module included in the following assemblies:
//
// * storage/understanding-persistent-storage.adoc
//* microshift_storage/understanding-persistent-storage-microshift.adoc

:_mod-docs-content-type: REFERENCE
[id="persistent-volumes_{context}"]
= Persistent volumes

Each PV contains a `spec` and `status`, which is the specification and status of the volume, for example:

.`PersistentVolume` object definition example
[source,yaml]
----
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv0001 <1>
spec:
  capacity:
    storage: 5Gi <2>
  accessModes:
    - ReadWriteOnce <3>
  persistentVolumeReclaimPolicy: Retain <4>
  ...
status:
  ...
----
<1> Name of the persistent volume.
<2> The amount of storage available to the volume.
<3> The access mode, defining the read-write and mount permissions.
<4> The reclaim policy, indicating how the resource should be handled once it is released.

You can view the name of a PVC that is bound to a PV by running the following command:

[source,terminal]
----
$ oc get pv <pv_name> -o jsonpath='{.spec.claimRef.name}'
----

ifndef::microshift[]
[id="types-of-persistent-volumes_{context}"]
== Types of PVs

ifndef::openshift-rosa,openshift-rosa-hcp[]
{product-title} supports the following persistent volume plugins:
endif::openshift-rosa,openshift-rosa-hcp[]
ifdef::openshift-rosa,openshift-rosa-hcp[]
{product-title} supports the following persistent volume storage options:
endif::openshift-rosa,openshift-rosa-hcp[]

// - GlusterFS
// - Ceph RBD
// - OpenStack Cinder
- AWS Elastic Block Store (EBS), which is installed by default.
ifdef::openshift-enterprise,openshift-webscale,openshift-origin,openshift-rosa,openshift-rosa-hcp[]
- AWS Elastic File Store (EFS)
endif::openshift-enterprise,openshift-webscale,openshift-origin,openshift-rosa,openshift-rosa-hcp[]
ifdef::openshift-enterprise,openshift-webscale,openshift-origin,openshift-aro[]
- Azure Disk
- Azure File
endif::openshift-enterprise,openshift-webscale,openshift-origin,openshift-aro[]
ifdef::openshift-enterprise,openshift-webscale,openshift-origin[]
- Cinder
- Fibre Channel
endif::openshift-enterprise,openshift-webscale,openshift-origin[]
ifndef::openshift-rosa,openshift-rosa-hcp[]
- GCP Persistent Disk
- GCP Filestore
endif::openshift-rosa,openshift-rosa-hcp[]
ifdef::openshift-enterprise,openshift-webscale,openshift-origin[]
- {ibm-power-server-title} Block
- {ibm-cloud-name} VPC Block
endif::openshift-enterprise,openshift-webscale,openshift-origin[]
ifdef::openshift-enterprise,openshift-webscale,openshift-origin,openshift-aro[]
- HostPath
- iSCSI
- Local volume
- {lvms}
- NFS
- OpenStack Manila
- {rh-storage-first}
endif::openshift-enterprise,openshift-webscale,openshift-origin,openshift-aro[]
ifdef::openshift-enterprise,openshift-webscale,openshift-origin[]
- CIFS/SMB
- VMware vSphere
// - Local
endif::openshift-enterprise,openshift-webscale,openshift-origin[]
endif::microshift[]

ifdef::openshift-rosa,openshift-rosa-hcp[]
{product-title} functions with Kubernetes Container Storage Interface (CSI) compatible volume provisioners from other storage vendors. For more information about CSI drivers in {product-title}, see _Configuring CSI volumes_ in the _Additional resources_ section.
endif::openshift-rosa,openshift-rosa-hcp[]

[id="pv-capacity_{context}"]
== Capacity

Generally, a persistent volume (PV) has a specific storage capacity. This is set by using the `capacity` attribute of the PV.

Currently, storage capacity is the only resource that can be set or requested. Future attributes may include IOPS, throughput, and so on.

ifndef::microshift[]
[id="pv-access-modes_{context}"]
== Access modes

A persistent volume can be mounted on a host in any way supported by the resource provider. Providers have different capabilities and each PV's access modes are set to the specific modes supported by that particular volume. For example, NFS can support multiple read-write clients, but a specific NFS PV might be exported on the server as read-only. Each PV gets its own set of access modes describing that specific PV's capabilities.

Claims are matched to volumes with similar access modes. The only two matching criteria are access modes and size. A claim's access modes represent a request. Therefore, you might be granted more, but never less. For example, if a claim requests RWO, but the only volume available is an NFS PV (RWO+ROX+RWX), the claim would then match NFS because it supports RWO.

Direct matches are always attempted first. The volume's modes must match or contain more modes than you requested. The size must be greater than or equal to what is expected. If two types of volumes, such as NFS and iSCSI, have the same set of access modes, either of them can match a claim with those modes. There is no ordering between types of volumes and no way to choose one type over another.

All volumes with the same modes are grouped, and then sorted by size, smallest to largest. The binder gets the group with matching modes and iterates over each, in size order, until one size matches.

[IMPORTANT]
====
Volume access modes describe volume capabilities. They are not enforced constraints. The storage provider is responsible for runtime errors resulting from invalid use of the resource. Errors in the provider show up at runtime as mount errors.

ifndef::openshift-dedicated,openshift-rosa,openshift-rosa-hcp[]
For example, NFS offers `ReadWriteOnce` access mode. If you want to use the volume's ROX capability, mark the claims as `ReadOnlyMany`.

iSCSI and Fibre Channel volumes do not currently have any fencing mechanisms. You must ensure the volumes are only used by one node at a time. In certain situations, such as draining a node, the volumes can be used simultaneously by two nodes. Before draining the node, delete the pods that use the volumes.
endif::openshift-dedicated,openshift-rosa,openshift-rosa-hcp[]
====

The following table lists the access modes:

.Access modes
[cols="1,1,3",options="header"]
|===
|Access Mode |CLI abbreviation |Description
|ReadWriteOnce
|`RWO`
|The volume can be mounted as read-write by a single node.
|ReadWriteOncePod
|`RWOP`
|The volume can be mounted as read-write by a single pod on a single node.
ifdef::openshift-enterprise,openshift-webscale,openshift-origin[]
|ReadOnlyMany
|`ROX`
|The volume can be mounted as read-only by many nodes.
|ReadWriteMany
|`RWX`
|The volume can be mounted as read-write by many nodes.
endif::[]
|===

.Supported access modes for persistent volumes
[cols=",^v,^v,^v,^v", width="100%",options="header"]
|===
|Volume plugin  |ReadWriteOnce ^[1]^ | ReadWriteOncePod |ReadOnlyMany|ReadWriteMany
|AWS EBS ^[2]^ | ✅ | ✅ |  |
|AWS EFS | ✅ | ✅ | ✅ | ✅
ifdef::openshift-enterprise,openshift-webscale,openshift-origin[]
|Azure File | ✅ |✅ | ✅ | ✅
|Azure Disk | ✅ | ✅ |   |
//|Ceph RBD  | ✅ | ✅ |✅ |
//|CephFS  | ✅ | ✅ | ✅ |  ✅
|CIFS/SMB | ✅ | ✅ | ✅ | ✅
|Cinder  | ✅ | ✅ | |
|Fibre Channel  | ✅ | ✅ |✅ |  ✅ ^[3]^
endif::[]
ifndef::openshift-rosa,openshift-rosa-hcp[]
|GCP Persistent Disk  | ✅ ^[4]^ |✅ | ✅ | ✅ ^[4]^
|GCP Filestore | ✅ | ✅ |✅ | ✅
endif::openshift-rosa,openshift-rosa-hcp[]
ifdef::openshift-enterprise,openshift-webscale,openshift-origin[]
//|GlusterFS  | ✅ |✅ | ✅ | ✅
|HostPath  | ✅ |✅ |   |
|{ibm-power-server-title}  Disk | ✅ |✅  | ✅ |  ✅
|{ibm-cloud-name} VPC Disk | ✅ |✅ |  |
|iSCSI  | ✅ | ✅ |✅ |  ✅ ^[3]^
|Local volume | ✅ |✅ |  |
endif::[]
|LVM Storage | ✅ | ✅ |   |
ifdef::openshift-enterprise,openshift-webscale,openshift-origin[]
|NFS  | ✅ | ✅ |✅ | ✅
|OpenStack Manila  |  |✅ |  | ✅
|{rh-storage-first}  | ✅ |✅ |  | ✅
|VMware vSphere | ✅ |✅ |  |  ✅ ^[5]^
endif::[]
|===
[.small]
--
1. ReadWriteOnce (RWO) volumes cannot be mounted on multiple nodes. If a node fails, the system does not allow the attached RWO volume to be mounted on a new node because it is already assigned to the failed node. If you encounter a multi-attach error message as a result, force delete the pod on a shutdown or crashed node to avoid data loss in critical workloads, such as when dynamic persistent volumes are attached.

2. Use a recreate deployment strategy for pods that rely on AWS EBS.

3. Only raw block volumes support the `ReadWriteMany` (RWX) access mode for Fibre Channel and iSCSI. For more information, see "Block volume support".

4. For GCP hyperdisk-balanced disks:
+
* The supported access modes are:
** `ReadWriteOnce`
** `ReadWriteMany`

* Cloning and snapshotting is disabled for disks with `ReadWriteMany` access mode enabled.

* You can attach a single hyperdisk-balanced disk volume in `ReadWriteMany` to a maximum of 8 instances.

* You can only resize a disk in `ReadWriteMany` if you detach the disk from all instances.

* https://cloud.google.com/compute/docs/disks/attach-disks[Additional limitations].

ifndef::openshift-dedicated,openshift-rosa,openshift-rosa-hcp[]
5. If the underlying vSphere environment supports the vSAN file service, the vSphere Container Storage Interface (CSI) Driver Operator installed by {product-title} supports provisioning of ReadWriteMany (RWX) volumes. If you do not have vSAN file service configured, and you request RWX, the volume fails to get created and an error is logged. For more information, see "Using Container Storage Interface" -> "VMware vSphere CSI Driver Operator".
endif::openshift-dedicated,openshift-rosa,openshift-rosa-hcp[]
// GCE Persistent Disks, or Openstack Cinder PVs.
--
endif::microshift[]

ifdef::microshift[]
[id="supported-access-modes_{context}"]
== Supported access modes
LVMS is the only CSI plugin {product-title} supports. The hostPath and LVs built in to {OCP} also support RWO.
endif::microshift[]

ifdef::openshift-online[]
[id="pv-restrictions_{context}"]
== Restrictions

The following restrictions apply when using PVs with {product-title}:
endif::[]

ifdef::openshift-online[]
 * PVs are provisioned with EBS volumes (AWS).
 * Only RWO access mode is applicable, as EBS volumes and GCE Persistent Disks cannot be mounted to multiple nodes.
 * Docker volumes are disabled.
   ** VOLUME directive without a mapped external volume fails to be
instantiated
.
 * *emptyDir* is restricted to 512 Mi per project (group) per node.
   ** A single pod for a project on a particular node can use up to 512 Mi
of *emptyDir* storage.
   ** Multiple pods for a project on a particular node share the 512 Mi of
*emptyDir* storage.
 *  *emptyDir* has the same lifecycle as the pod:
   ** *emptyDir* volumes survive container crashes/restarts.
   ** *emptyDir* volumes are deleted when the pod is deleted.
endif::[]

[id="pv-phase_{context}"]
== Phase

Volumes can be found in one of the following phases:

.Volume phases
[cols="1,2",options="header"]
|===

|Phase
|Description

|Available
|A free resource not yet bound to a claim.

|Bound
|The volume is bound to a claim.

|Released
|The claim was deleted, but the resource is not yet reclaimed by the
cluster.

|Failed
|The volume has failed its automatic reclamation.

|===

=== Last phase transition time
The `LastPhaseTransitionTime` field has a timestamp that updates every time a persistent volume (PV) transitions to a different phase (`pv.Status.Phase`). To find the time of the last phase transition for a PV, run the following command:

[source,terminal]
----
$ oc get pv <pv_name> -o json | jq '.status.lastPhaseTransitionTime' <1>
----
<1> Specify the name of the PV that you want to see the last phase transition.

[id="pv-mount-options_{context}"]
=== Mount options

You can specify mount options while mounting a PV by using the attribute `mountOptions`.

For example:

ifndef::microshift[]
.Mount options example
[source,yaml]
----
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv0001
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  mountOptions: <1>
    - nfsvers=4.1
  nfs:
    path: /tmp
    server: 172.17.0.2
  persistentVolumeReclaimPolicy: Retain
  claimRef:
    name: claim1
    namespace: default
----
<1> Specified mount options are used while mounting the PV to the disk.

The following PV types support mount options:

// - GlusterFS
// - Ceph RBD
- AWS Elastic Block Store (EBS)
- AWS Elastic File Storage (EFS)
ifndef::openshift-dedicated,openshift-rosa,openshift-rosa-hcp[]
- Azure Disk
- Azure File
- Cinder
endif::openshift-dedicated,openshift-rosa,openshift-rosa-hcp[]
ifndef::openshift-rosa,openshift-rosa-hcp[]
- GCE Persistent Disk
endif::openshift-rosa,openshift-rosa-hcp[]
ifndef::openshift-dedicated,openshift-rosa,openshift-rosa-hcp[]
- iSCSI
- Local volume
- NFS
- {rh-storage-first} (Ceph RBD only)
- CIFS/SMB
- VMware vSphere

[NOTE]
====
Fibre Channel and HostPath PVs do not support mount options.
====
endif::openshift-dedicated,openshift-rosa,openshift-rosa-hcp[]
endif::microshift[]

ifdef::microshift[]
.Mount options example
[source,yaml]
----
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
  name: topolvm-provisioner
mountOptions:
  - uid=1500
  - gid=1500
parameters:
  csi.storage.k8s.io/fstype: xfs
provisioner: topolvm.io
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
----

[NOTE]
====
The `mountOptions` parameter values are not validated. Incorrect values cause the mount to fail and an event to be logged to the PVC.
====
endif::microshift[]