Storage Service API
Terminology
There are many di!erent standards and protocols for the le upload process. Here is to
summarize and clarify the di!erent terms and mechanics for le upload.
HTTP PUT/POST
The simple HTTP POST/PUT operation
HTTP multipart
RFC
HTTP chunked transfer encoding
RFC
S3 style multipart
This is a non-standard protocol which allows user to upload one le in pieces and in
parallel. The server will reassembly it once the whole upload is complete as indicated by
the user. The size of each upload is known to the server through the content-length
header. Upload is resumable within the given time window (for example 24 hours). Drop ,
Google Drive, Youtube etc. all have this kind of upload mechanism.
Objects & Chunks
Objects are individual les stored in Storage Service which may have multi chunks stored
separately.
Comparison of di!erent upload options
POST/PUT HTTP multipart HTTP chunked S3 style multipart
Single Session Y Y Y N
Pause N N N Y
Resume N N Y Y
Multiple Files N Y N N
The storage service will support single POST/PUT with the ability to resume in case the transfer
is interrupted. Storage Service will accept one object at a time, therefore HTTP multipart
support is not relevant. The S3 style multipart can provide upload acceleration and resumable
upload at the cost of client to split the content before uploading. This will be supported by
storage service. The storage service will be the one to split the content when it sees the need
for it.
Since existing API uses HTTP multipart post as default method. One challenge caused by this is
that the content-length is not set for each le. The support for HTTP chunked transfer encoding
can be added without changes in the Storage Service.
Goal
The goal of the design is to provide resumable streaming upload with no le size limitation.
Out of Scope
Single object upload accelerator, for example parallel uploads to speed up the process.
Support HTTP chunked transfer encoding could be added between client and streaming
upload service without changes in the Storage Service. Although this will increase the
number of unknown size uploads and hence increase the memory footprint of Storage
Service for bu!ering as described below.
Upload Content Length Complications
Upload with known content-length
Storage Service will chunk the object with optimal size for the backend.
Upload with known content-length-max
Storage Service will use content-length-max to try nd the optimal chunking size. For
HTTP multipart uploads, this would be the size of all parts and no individual object will
exceed this size.
Upload with unknown size
In this scenario, Storage Service will split the object at a predened boundary (64M,
128M or 256M). The content will be bu!ered inside the Storage Service up to this
predened size and then ush the chunk to the backend storage system. This will put
memory pressure on Storage Service. We will need to limit the number of concurrent
unknown size uploads.
In all cases, each chunk will be the multiple of the encryption/decryption boundary (512 bytes).
Best Practices
Upload process should check for status after communication failure instead of simply
retry the upload
In the long term, Storage Service will enforce the lifecycle of pending upload requests. As
a best practice, client should try not to leave incomplete uploads on the server.
The client should try to use the same encryption key when resuming an interrupted
upload. To achieve that, the client may want to save the key rst before the content.
API
ClientID BinID GlobalID AuxiliaryID
Assigned by storage
team
Assigned by storage
team
Assigned by
server
Specied by
client
In all examples, ClientID=webapp, Bin=zVNpoQNsOSxZKqOZgckhpQ,
GlobalID=51d28108ea7144a8b42e2670cd791c76. These sample IDs are just to make the
discussion here easier and their format could change at any time.
Quick Reference
POST /webapp/zVNpoQNsOSxZKqOZgckhpQ
PUT /webapp/zVNpoQNsOSxZKqOZgckhpQ/[Link]
GET /webapp/zVNpoQNsOSxZKqOZgckhpQ/[Link]/status
GET /webapp/zVNpoQNsOSxZKqOZgckhpQ/[Link]
GET /webapp/zVNpoQNsOSxZKqOZgckhpQ/51d28108ea7144a8b42e2670cd791c76/list
DELETE /webapp/zVNpoQNsOSxZKqOZgckhpQ/[Link]
HEAD /webapp/zVNpoQNsOSxZKqOZgckhpQ/[Link]
PUT /webapp/zVNpoQNsOSxZKqOZgckhpQ/[Link]?part=1
POST /webapp/zVNpoQNsOSxZKqOZgckhpQ/[Link]/commit
Upload Process
Initiate the upload
To initiate the upload, send a POST request to the Storage Service with ClientID and Bin.
POST /webapp/zVNpoQNsOSxZKqOZgckhpQ
DESCRIPTION
Storage Service will generate a globally unique ID (GlobalID) under the Bin specied.
HEADERS
None required.
RESPONSE
51d28108ea7144a8b42e2670cd791c76
ERRORS
Start upload
Next, implement a PUT request that sends the object to Storage Service.
PUT /webapp/zVNpoQNsOSxZKqOZgckhpQ/[Link]
DESCRIPTION
If the PUT request is not interrupted and the object is successfully uploaded, Storage Service
responds with a 200 OK status code. If the upload is interrupted, you can resume the upload.
HEADERS
Content-Range: 0
If the size of the object is known, set Content-Length header.
Content-Length: 10618736
If the size of the object is unknown, but the size limitation is known, set Content-Length-Max
header.
Content-Length-Max: 10918736
If both the size of the object and the size limitation of the object are unknown, set both Content-
Length and Content-Length-Max to zero.
Content-Length: 0
Content-Length-Max: 0
RESPONSE
10618736
ERRORS
400 Invalid Request - when the content offset is not expected
405 Method Not Allowed - when the object already exists
Query upload status
If the upload operation is interrupted or gets an HTTP 503 or 500 response, you should query
for the number of bytes it has received by implementing a GET request.
GET /webapp/zVNpoQNsOSxZKqOZgckhpQ/[Link]/status
DESCRIPTION
Storage Service will return the current upload status with range information so that the client
can resume the upload if needed.
HEADERS
None required.
RESPONSE
TODO: Paste exact response
[{
"ObjectID" : "[Link]$1"
"Start" : 0
"End" : 10612736
}]
The example indicates that the rst 10612736 bytes of the object has been received.
ERRORS
404 Cannot find the object.
Resume upload
Finally, you can resume the upload operation by implementing another PUT request.
PUT /webapp/zVNpoQNsOSxZKqOZgckhpQ/[Link]
DESCRIPTION
The Content-Range header has to be set and matches the o!set that the Storage Service is
expecting. If the PUT request is not interrupted and the object is successfully uploaded, Storage
Service responds with a 200 OK status code. If the upload is interrupted, you can resume the
upload.
HEADERS
Content-Range: 10612737
Content-Length-Max: 20618736
RESPONSE
10618736
ERRORS
400 Invalid Request - when the content offset is not expected
405 Method Not Allowed - when the object already exists
S3 Style Multipart Requests
Upload object
Upload is similar to normal upload, just need to specify the part number with the part parameter
in the http request. And each part is resumable as well.
PUT /webapp/zVNpoQNsOSxZKqOZgckhpQ/[Link]?part=1
Commit object
After all parts are uploaded, client can commit the upload. After this, the object can be
downloaded as one object.
POST /webapp/zVNpoQNsOSxZKqOZgckhpQ/[Link]/commit
DESCRIPTION
After all parts are uploaded, client can commit the upload. After this, the object can be
downloaded as one object.
HEADERS
None
RESPONSE
10618736
ERRORS
400 Invalid Request - when the upload is not committable/completed
404 Not Found
405 Method Not Allowed - when the object already exists
400 Invalid Request - when the content offset is not expected
405 Method Not Allowed - when the object already exists
Other Requests
Retrieve object
GET /webapp/zVNpoQNsOSxZKqOZgckhpQ/[Link]
DESCRIPTION
The Storage Service will retrieve the object specied.
HEADERS
None required.
RESPONSE
Object download
ERRORS
404 Cannot find the object.
List object
GET /webapp/zVNpoQNsOSxZKqOZgckhpQ/51d28108ea7144a8b42e2670cd791c76/list
DESCRIPTION
The Storage Service will retrieve information of all objects associated with the specied
GlobalID.
HEADERS
None required.
Query parameters supported.
start-objectID: O!set for pagination
max-keys: Limit
RESPONSE
TODO: Paste exact response
{
}
ERRORS
None
Delete object
DELETE /webapp/zVNpoQNsOSxZKqOZgckhpQ/[Link]
DESCRIPTION
The Storage Service will delete the object specied.
HEADERS
None required.
RESPONSE
TODO: Paste exact response
{
}
ERRORS
404 Cannot find the object.
Head object
HEAD /webapp/zVNpoQNsOSxZKqOZgckhpQ/[Link]
DESCRIPTION
Get detail information about one object. This can be used for consistency checking purpose
which means that the require will reach the backend storage system for verication.
HEADERS
None required.
RESPONSE
TODO: Paste exact response
{
}
ERRORS
None