CephFS driver¶
The CephFS driver enables manila to export shared filesystems backed by Ceph’s File System (CephFS) using either the Ceph network protocol or NFS protocol. Guests require a native Ceph client or an NFS client in order to mount the filesystem.
When guests access CephFS using the native Ceph protocol, access is controlled via Ceph’s cephx authentication system. If a user requests share access for an ID, Ceph creates a corresponding Ceph auth ID and a secret key if they do not already exist, and authorizes the ID to access the share. The client can then mount the share using the ID and the secret key. To learn more about configuring Ceph clients to access the shares created using this driver, please see the Ceph documentation
And when guests access CephFS through NFS, an NFS-Ganesha server mediates access to CephFS. The driver enables access control by managing the NFS-Ganesha server’s exports.
Supported Operations¶
The following operations are supported with CephFS backend:
Create/delete share
Allow/deny access to share
Only
cephx
access type is supported for CephFS native protocol.Only
ip
access type is supported for NFS protocol.read-only
andread-write
access levels are supported.
Extend/shrink share
Create/delete snapshot
Create/delete share groups
Create/delete share group snapshots
Prerequisites¶
Important
A manila share backed by CephFS is only as good as the underlying filesystem. Take care when configuring your Ceph cluster, and consult the latest guidance on the use of CephFS in the Ceph documentation.
Ceph testing matrix¶
As Ceph and Manila continue to grow, it is essential to test and support combinations of releases supported by both projects. However, there is little community bandwidth to cover all of them. Below is the current state of testing for Ceph releases with this project. Adjacent components such as devstack-plugin-ceph and tripleo are added to the table below. Contributors to those projects can determine what versions of ceph are tested and supported with manila by those components; however, their state is presented here for ease of access.
Important
From the Victoria cycle, the Manila CephFS driver is not tested or supported with Ceph clusters older than Nautilus. Future releases of Manila may be incompatible with Nautilus too! We suggest always running the latest version of Manila with the latest release of Ceph.
OpenStack release |
manila |
devstack-plugin-ceph |
tripleo |
---|---|---|---|
Queens |
Luminous |
Luminous |
Luminous |
Rocky |
Luminous |
Luminous |
Luminous |
Stein |
Nautilus |
Luminous, Nautilus |
Nautilus |
Train |
Nautilus |
Luminous, Nautilus |
Nautilus |
Ussuri |
Nautilus |
Luminous, Nautilus |
Nautilus |
Victoria |
Nautilus |
Nautilus |
Nautilus |
Common Prerequisites¶
A Ceph cluster with a filesystem configured (See Create ceph filesystem on how to create a filesystem.)
ceph-common
package installed in the servers running the manila-share service.Network connectivity between your Ceph cluster’s public network and the servers running the manila-share service.
Authorizing the driver to communicate with Ceph¶
Run the following commands to create a Ceph identity for a driver instance to use:
read -d '' MON_CAPS << EOF
allow r,
allow command "auth del",
allow command "auth caps",
allow command "auth get",
allow command "auth get-or-create"
EOF
ceph auth get-or-create client.manila -o manila.keyring \
mds 'allow *' \
osd 'allow rw' \
mon "$MON_CAPS"
manila.keyring
, along with your ceph.conf
file, will then need to be
placed on the server running the manila-share service.
Important
To communicate with the Ceph backend, a CephFS driver instance (represented as a backend driver section in manila.conf) requires its own Ceph auth ID that is not used by other CephFS driver instances running in the same controller node.
In the server running the manila-share service, you can place the
ceph.conf
and manila.keyring
files in the /etc/ceph directory. Set the
same owner for the manila-share process and the manila.keyring
file. Add the following section to the ceph.conf
file.
[client.manila]
client mount uid = 0
client mount gid = 0
log file = /opt/stack/logs/ceph-client.manila.log
admin socket = /opt/stack/status/stack/ceph-$name.$pid.asok
keyring = /etc/ceph/manila.keyring
It is advisable to modify the Ceph client’s admin socket file and log file locations so that they are co-located with manila services’s pid files and log files respectively.
Enabling snapshot support in Ceph backend¶
CephFS Snapshots were experimental prior to the Nautilus release of Ceph. There may be some limitations on snapshots based on the Ceph version you use.
From Ceph Nautilus, all new filesystems created on Ceph have snapshots enabled by default. If you’ve upgraded your ceph cluster and want to enable snapshots on a pre-existing filesystem, you can do so:
ceph fs set {fs_name} allow_new_snaps true
Configuring CephFS backend in manila.conf¶
Space considerations¶
The CephFS driver reports total and free capacity available across the Ceph
cluster to manila to allow provisioning. All CephFS shares are thinly
provisioned, i.e., empty shares do not consume any significant space
on the cluster. The CephFS driver does not allow controlling oversubscription
via manila. So, as long as there is free space, provisioning will continue,
and eventually this may cause your Ceph cluster to be over provisioned and
you may run out of space if shares are being filled to capacity. It is advised
that you use Ceph’s monitoring tools to monitor space usage and add more
storage when required in order to honor space requirements for provisioned
manila shares. You may use the driver configuration option
reserved_share_percentage
to prevent manila from filling up your Ceph
cluster, and allow existing shares to grow.
Known restrictions¶
A CephFS driver instance, represented as a backend driver section in manila.conf, requires a Ceph auth ID unique to the backend Ceph Filesystem. Using a non-unique Ceph auth ID will result in the driver unintentionally evicting other CephFS clients using the same Ceph auth ID to connect to the backend.
Snapshots are read-only. A user can read a snapshot’s contents from the
.snap/{manila-snapshot-id}_{unknown-id}
folder within the mounted share.
Security¶
Each share’s data is mapped to a distinct Ceph RADOS namespace. A guest is restricted to access only that particular RADOS namespace. https://docs.ceph.com/docs/nautilus/cephfs/file-layouts/
An additional level of resource isolation can be provided by mapping a share’s contents to a separate RADOS pool. This layout would be preferred only for cloud deployments with a limited number of shares needing strong resource separation. You can do this by setting a share type specification,
cephfs:data_isolated
for the share type used by the cephfs driver.manila type-key cephfstype set cephfs:data_isolated=True