RADOS is Ceph's object store. Clients like RADOS Gateway (RGW), RADOS Block Devices (RBD), and CephFS use it to store data and provide more complex APIs.

Those complex APIs use RADOS objects as building blocks for block device images (RBD), files and directories (CephFS), and Amazon S3/OpenStack Swift-style objects. The RADOS pools used by a client not only store objects that correspond to the client-side data, but also metadata objects, for example for, directories, bucket indexes and parameters. How clients map their data representations to RADOS objects is topic of another article. This article focuses on the properties of RADOS objects and the rich API RADOS provides around them.

Related Links:

• Ceph Documentation: APIs
• Ceph Documentation: RADOS API

Example Code:

• Ceph Repository: librados examples - Setup, Teardown, basic RADOS object operations
• Ceph Repository: rados tool - Command line tool with almost the complete RADOS API. Also shows how to use the libradosstriper API, that automatically stripes data over multiple objects according to a pre-defined layout.

Other interesting places:

• librados implementation
• Object Classes complex operations executed by the OSDs

Object names are a string of arbitrary bytes. They are the main input to CRUSH and therefore determine the OSDs that are responsible for storing them. OSDs use them to name local files or to key entries in their databases.

Their length is, by default, limited to 2048 bytes. In comparison with local filesystems (see table below) this is huge. It is configurable with the osd max object name len setting.

#+NAME:tab:fsnamelimits

The commit that introduced the setting has more details and rationale about this setting:

7e0aca1 2014-07-16 Sage Weil <sage@redhat.com>
osd: add config for osd_max_object_name_len = 2048 (was hard-coded at 4096)
Previously we had a hard coded limit of 4096.  Objects > 3k crash the OSD
when running on ext4, although they probably work on xfs.  But rgw only
generates objects a bit over 1024 bytes (maybe 1200 tops?), so let set a
more reasonable limit here.  2048 is a nice round number and should be
safe.

Add a test.

Fixes: #8174
Signed-off-by: Sage Weil <sage@redhat.com>

The OSDs check the object names as part of its transaction / operation processing. If an object is too long, the client receive an ENAMETOOLONG reply.

Example:

./rados --pool=test put $(xxd -l $((2048/2)) -p /dev/urandom | tr -d '\n') \
	<(dd if=/dev/random count=10 bs=1M) # works
./rados --pool=test put $(xxd -l $((4096/2)) -p /dev/urandom | tr -d '\n') \
	<(dd if=/dev/random count=10 bs=1M) # File name too long

There is a second value determining the maximum length, provided by the used object store (get_max_object_name_length())

void ReplicatedPG::do_op(OpRequestRef& op)
  [..]

  // object name too long?
  unsigned max_name_len = MIN(g_conf->osd_max_object_name_len,
                              osd->osd->store->get_max_object_name_length());
  if (m->get_oid().name.size() > max_name_len) {
    dout(4) << "do_op '" << m->get_oid().name << "' is longer than "
            << max_name_len << " bytes" << dendl;
    osd->reply_op_error(op, -ENAMETOOLONG);
    return;
  }
 [..]
}

The way OSDs use object names depends on the object store used. They have all in common, that they use the object name in combination with a namespace, pool id and other information, such as the snapshot name, to identify objects. This information combined is the object ID. BlueStore and KStore use RocksDB, whereas FileStore uses a filesystem to locate objects by their ID.

As mentioned before, filesystems have a filename limit of 255 bytes. Therefore, FileStore may has to go an extra mile to map object names to filesystem filenames.

class LFNIndex : public CollectionIndex {
  /// Hash digest output size.
  static const int FILENAME_LFN_DIGEST_SIZE = CEPH_CRYPTO_SHA1_DIGESTSIZE;
  /// Length of filename hash.
  static const int FILENAME_HASH_LEN = FILENAME_LFN_DIGEST_SIZE;
  /// Max filename size.
  static const int FILENAME_MAX_LEN = 4096;
  /// Length of hashed filename.
  static const int FILENAME_SHORT_LEN = 255;
  /// Length of hashed filename prefix.
  static const int FILENAME_PREFIX_LEN;
  /// Length of hashed filename cookie.
  static const int FILENAME_EXTRA = 4;
  /// Lfn cookie value.
  static const string FILENAME_COOKIE;
  /// Name of LFN attribute for storing full name.
  static const string LFN_ATTR;
  /// Prefix for subdir index attributes.
  static const string PHASH_ATTR_PREFIX;
  /// Prefix for index subdirectories.
  static const string SUBDIR_PREFIX;

const int LFNIndex::FILENAME_PREFIX_LEN =  FILENAME_SHORT_LEN - FILENAME_HASH_LEN -
								FILENAME_COOKIE.size() -
								FILENAME_EXTRA;

const string LFNIndex::FILENAME_COOKIE = "long";

RADOS objects are not merely data containers: The also provide synchronization, locking, and access to a key value database. The data portion has file like operations.

Ceph clients almost universally use a default object size of 4 MB. This is usually configured once on initialization. In RBD, for example, when creating a new image.

Most clients use the Ceph Striper API to split their data units (files, images, S3 objects) into RADOS objects. 4 MB is the default size.

Still, 4 MB is not the maximum size. It is configurable with the osd max object size setting and defaults to 100 GB. Writes beyond that limit result in an error. The commit that introduced this feature has more details:

f1b6bd7 2013-06-13 David Zafman <david.zafman@inktank.com>
osd: EINVAL from truncate causes osd to crash
Maximum object size is 100GB configurable with osd_max_object_size
Error EFBIG if attempt to WRITE/WRITEFULL/TRUNCATE beyond osd_max_object_size
Error EINVAL if length < 1 for WRITE/WRITEFULL/ZERO
Make ZERO beyond existing size a no-op

Fixes: #5252
Fixes: #5340

Signed-off-by: David Zafman <david.zafman@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>

diff --git a/src/common/config_opts.h b/src/common/config_opts.h
index 8a02dd5..094d124 100644
--- a/src/common/config_opts.h
+++ b/src/common/config_opts.h
@@ -485,6 +485,8 @@ OPTION(osd_recovery_op_priority, OPT_INT, 10)
+OPTION(osd_max_object_size, OPT_U64, 100*1024L*1024L*1024L) // OSD's maximum object size