Write up of a recent RGW error handling deep dive.

From the perspective of a storage backend(1) (1)… like the s3gw filesystem+SQLite backend .

How to convert storage errors to client errors in a meaningful way?

S3 uses HTTP return codes first and adds detail via the XML error response.

<?xml version="1.0" encoding="UTF-8"?>
<Error>
  <Code>CamelCasedError code from ErrorCodeList </Code>
  <Message>Message for human consumption</Message>
  <Resource>Bucket or object that caused the error</Resource>
  <RequestId>ID of request associated with the error</RequestId>
</Error>

• S3 docs
• List of S3 Error Codes

RGW uses negative return values to signal errors. C++ exceptions are not expected and generally bubble up to the top crashing the application.

Error codes are a mixture of custom (ERR_...) and errno.h (E...) errors. The errno.h errors usually make only sense if you look at them from a client perspective(2) (2)This is important! RGW uses EINVAL, EPERM, EACCES, ENOENT, EEXIST. Passing filesystem errors further up will result in weird behavior. .

For completeness, HTTP codes are nicely documented here:

• Wikipedia: List of HTTP status codes
• MDN: HTTP response status codes

Amazon has a list of error codes in their docs:

• Amazon S3 API: Error Responses

RGW defines more on top, as do other vendors:

• Isilon S3 error codes doc
• Minio api-errors.go

There is a global mapping typed rgw_http_errors of error number (err_no) to HTTP code and an error code string(3) (3)RGW supports multiple protocols, therefore there are mappings for S3, SWIFT, STS and IAM protocol frontends. .

RGW Operations(4) (4)Descendants of RGWOp. Counted 119 while writing this. Examples: GetObj, ListBuckets, etc. have an error state as part of their req_state. The request processing converts this to HTTP return codes and an S3 style error response. Responses may contain a message field, if req_state.err.message is set.

The SAL layer does not have access to the error state. SAL backends may only return error numbers, no additional message.

Source links:

• rgw/rgw_common.cc contains the error number mappings, and the conversion logic
• rgw/rgw_common.h contains the ERR_ constant definitions

Based on the lookup table in rgw/rgw_common.cc, but reordered a bit.

Permanent errors have to be handled appropriately to their meaning. Boto3, for example, throws a matching exception

Temporary errors are a bit more interesting. To quote the S3 docs: