Comparing Windows Azure Blob Storage and Amazon Simple Storage Service (S3)

the Second and third parts compare storage services Windows Azure and Amazon. Is the last part — comparing of Windows Azure Blob Storage and Google Cloud Storage.

In the first part of this article, we started comparing Windows Azure Blob Storage and Amazon Simple Storage Service (S3), looked at the basic concepts and compared pricing and functionality on containers, blobs, and baskets. to Read the first part.
In this section we compare the functionality on blobs and objects, and close comparison of the storage services of both systems, reducing the resulting data into a single table.

the First part of a cycle — Comparing of Windows Azure Table Storage and Amazon DynamoDB
the Second part of a cycle — Comparing of Windows Azure Blob Storage and Amazon Simple Storage Service (S3) — part I

Abbreviations: Windows Azure Blob Storage WABS and Amazon Simple Storage Service AS3.

the Concept

Before detailed discussion of the functions I consider important to clarify some of the concepts behind the blobs and objects.

the Blobs and objects: Blobs WABS and AS3 objects are files in your cloud file system, located in the blobs containers and baskets.

Comments:

the

has no limit to the number of stored blobs and objects in AS3 is the number simply not known, in WABS is the number limited by the size of the storage account (100 TB).
Maximum size of an object in AS3 – 5 TB, in WABS is 1 TB.
IN WABS there are two types of blobs – block, convenient for streaming (e.g., images, video, documents) and having a maximum size of 200 GB, and page, convenient for random access/write and a maximum size of 1 TB. Generally, the use of a page BLOB is a VHD mounted as a disk in an Azure role. In AS3 this separation.
Both systems provide rich functionality for managing blobs and objects. You can copy, upload, download and perform other operations.
Both systems provide the ability to protect content from unauthorized access, and the mechanism of access control lists in more detail the custom in AS3 where each file in the basket you can create your own ACL. In WABS all happens at the container level blobs.

Two of the most important functions is uploading and downloading, so I propose to discuss them first, then compare the other functions.

Uploading blobs and objects

Let's talk about downloading blobs and objects in containers and baskets. There are two mechanisms of loading – it is possible to load BLOB or object completely within one request or to divide them into pieces (units or pages in WABS and parts in AS3).

Download in a single request

If the data to be downloaded are small in size and you have a good connection speed, you can load these data in a single request. In WABS is used for this Put Blob. In AS3 — Put Object.

Download pieces

You can divide the big data that is inefficient to load in one request completely. Both systems allow you to break the data into pieces (units or pages in WABS and parts in AS3) and to load gradually. In WABS for block blobs, you must use PutBlock and Put Block List, for page — Put Page. In AS3 for this purpose, the Initiate Multipart Upload, Upload Part and Complete Multipart Upload or Abort Multipart Upload.
There are many reasons why you may decide to load the data pieces:

the

Needs to load very large data. Pay attention that in WABS one block BLOB limited to the size of 200 GB, page – 1 TB, in AS3 you can have one object to 5 TB. Such volumes are impractical to load in a single request.
Low speed connection.
Both systems are cloud services designed to handle requests of hundreds and thousands users simultaneously, and both systems will restrict your queries if they run beyond the statutory limit – in WABS it is 10 minutes to download 1 MB of data.
Split large data into pieces allows you to execute parallel loading (or, more quickly load data).
In case of an open load piece you can repeat the download, if will break the loading of large data loading in a single query have to download all over again, which is very inefficient.
Limitations of system: WABS does not allow in a single request to download data if their size exceeds 64 MB. In AS3 there is no such restriction, but the company recommends that you divide the data into pieces, if their size exceeds 100 MB.

Now as for how to load the data pieces in each of the systems. For example, you want to upload pieces of a file size of 100 MB. Let's see what will need to do when you use each of the platforms.

WABS

For example, each piece has a size of 1 MB (in spite of the fact that there is no need to have pieces of the same size) and you need to make loading of 100 pieces. Take a block BLOB, the blocks (pieces) of which has a unique identifier (BlockId). To download it use the function Put Block. BlockId – a string, Base64-encoded, the maximum size of which is limited to 64 bytes. All BlockId (100 in our case) must be of the same length. It does not matter what order you load the blocks – you can load them in parallel. After loading the block WABS puts it somewhere in the repository, and stores 7 days. After downloading all the blocks called Put Block List confirming (commit) those blocks. Until you call this function, the BLOB can not handle and if you have not confirmed blocks during the 7 days, they will be deleted by the system. After calling the function, based on the list order BlockId, WABS will recreate the BLOB and mark it as available. No matter what the values would have BlockId (they can all be GUID), but it is important in which order you will be sent a list of BlockId when using Put Block List.

the Limitation:

the

BLOB can be split a maximum of 50000 blocks.
BLOB can have a maximum of 100,000 uncommitted blocks at any given time.
the Set of uncommitted blocks cannot have a size larger 400 GB.
All BlockId of units one BLOB should be of the same length, i.e., unacceptable situation where their values will be equal block8, block9, block11.
BlockId Maximum length is 64 bytes.

AS3

For example, the size of each piece is 5 MB. Each piece is assigned a unique identifier (Part Number) identifying a piece and determining its position in the original object. This number can be any number from 1 to 10000, but they must be sorted, i.e., if you take our situation, the piece 0-5Mb should be number 1, a piece of 5-10 MB – 2, and so on. But when you start to download these parts, you must first call the function Initiate Multipart Upload which will tell AS3 that started the process of transfer of the object to pieces. The function returns an Upload Id. Next to each piece it is necessary to call the function Upload Part and pass the Upload Id and the number of the piece, and no matter in what order the pieces will be loaded (you can load in parallel). After downloading all the pieces you need to call the function Complete Multipart Upload, thus confirming pieces. To cancel the process, you can call Abort Multipart Upload.
Limitations:

the

Judging by the fact that the number of the piece must be from 1 to 10000, I assume that the object can be divided into up to 10000 pieces. the

as the room determines the position of a piece in the object, its value is important.
Each piece must be a minimum of 5 MB except for the last piece in the sequence, for which there is no size limitation.

Downloading of blobs and objects

To download blobs and objects there are two mechanisms – either the downloading of the BLOB or object completely within a single query or pieces.

Each system has only one function for downloading — Get Blob WABS and Get Object in AS3.

Download in a single request

If the data have a small size and at you good speed of connection, you could download object completely, using Get Blob WABS and Get Object in AS3.

pieces Download

If the object is large and you're not sure whether to download it all at once, you can swing the pieces, using the same function with the addition of a Range header and determining the range of bytes it takes to download.

The downloading process:

to Determine the size of the object. For example, he weighs 100 MB.
to Determine the size of the pieces. For example, you prefer to download chunks of 1 MB.
Call Get Blob or Get Object and pass them the appropriate values in the Range header. If you pump consistently, your first request will have the value of this header to “0 – 1048575” (0 – 1MB), the second request— “1048576 – 2097151” (1 – 2 MB) and so on.
After the download put the piece anywhere.
After downloading all the pieces to create an empty file of 100 MB and fill this file downloaded pieces.

Options

Functions provided by the two systems are summarized in the table below.
the the the the the the the the the the the the the the the the the the the the the the the the

	WABS	AS3
Put Blob/PUT Object	Yes	Yes
POST Object	No	Yes
Get Blob/GET Object	Yes	Yes
GET Object torrent	No	Yes
GET Object ACL	No	Yes
PUT Object ACL	No	Yes
Get Blob Properties/HEAD Object	Yes	Yes
Set Blob Properties	Yes
Get Blob Metadata/Head Object	Yes	Yes
Set Blob Metadata	Yes
Delete Blob/DELETE Object	Yes	Yes
Delete Multiple Objects	No	Yes
Copy Blob/Put Object – Copy	Yes	Yes
Snapshot Blob	Yes
Lease Blob	Yes
Initiate Multipart Upload	No	Yes
Block Put/Upload Part	Yes	Yes
Put Block List/Complete Multipart Upload	Yes	Yes
Get Block List/Parts List	Yes	Yes
Abort Multipart Upload	No	Yes
Upload Part – Copy	No	Yes
Put Page	Yes
the Get Page Ranges	Yes

We will discuss in more detail these functions.
the the

	WABS	AS3
Put Blob/PUT Object	Yes	Yes

The function adds the BLOB to the BLOB container and the object in the basket.

Comments:

the

In both systems, the function will overwrite an existing object with the specified name. In AS3 enabled on the basket, the versioning will create a copy of the object. In WABS you need to handle versioning their own, using Snapshot Blob.
Both systems allow you to send the MD5 content hash to check the data consistency.
In AS3 it is possible when you create the object to set ACL on it that you can't do in WABS.
AS3 allows you to store objects with reduced redundancy. When you call this function, you define an optional parameter whether to store the object with reduced redundancy. If the parameter is omitted the object is stored with standard redundancy.
Both systems allow you to specify metadata for blobs and objects in the form of a collection of pairs key-value. In both systems, maximum size of metadata – 8 KB.
When you create a page BLOB using this function, you only initiate a page BLOB, but don't put data into it. To insert data, you must use the function Put Page.
When you create a block BLOB or object in AS3, data is sent in the request.
Maximum size of block BLOB is created using this function is set to 64 MB. If it is, the BLOB has to be split into blocks and uploaded using Put Block and Put Block List.
WABS allows you to define the preconditions that must be satisfied for the successful completion of this function (IfModifiedSince, IfUnmodifiedSince, the IfMatch, IfNoneMatch).
AS3 supports encryption of objects on the server side, accordingly you can determine whether to encrypt the object when prompted. WABS does not support encryption on the server side, so to encrypt objects.

the the

	WABS	AS3
POST Object		Yes

The function adds the object to the specified shopping cart using an HTML form. POST is an alternative to PUT, and allows the download of the object browser. The parameters passed to PUT via HTTP headers are passed in the POST as the body of the encrypted message multipart/form-data.
the the

	WABS	AS3
Get Blob/GET Object	Yes	Yes

Feature allows you to download a BLOB from a container or basket.

Comments:

the

it is Possible to swing pieces, specifying the number of bytes in the Range header.
IN WABS function also returns metadata for the downloaded BLOB.
WABS allows you to define the preconditions that must be satisfied for the successful completion of this function (IfModifiedSince, IfUnmodifiedSince, the IfMatch, IfNoneMatch).
In AS3 you can override the values of specific response headers using query parameters (except for anonymous requests). The request must be signed with the Authorization header or use perepodpisanie URL.
you Can use this function to retrieve version of blobs and objects to obtain a version of the BLOB you need to specify the date/time the snapshot of the BLOB to retrieve the version of an object in AS3, you can specify a Version Id. If these parameters are omitted, returns the current version of the object.

the the

	WABS	AS3
GET Object torrent		Yes

AS3 allows you to use the features using the BitTorrent Protocol, which can also reduce the amount of transmitted traffic. Read more about the use of BitTorrent and AS3 — here.

The function can be called only on objects with sizes less than 5 GB.
the WABS
AS3
the GET Object ACL

Yes

In AS3, you can specify the ACL at the level of individual object (in WABS ACL can only specify a container-level blobs). This function is used to retrieve information about the ACL of the object

An interesting point is that if you use different versions of an object, each version will have separate ACL. To retrieve the ACL of a specific object version, you must pass a function as a Version Id the version.
the the

	WABS	AS3
PUT Object ACL		Yes

Function allows you to specify an ACL for the object. As mentioned above, each object has its own ACL. To specify an ACL for a specific version of a function object, you must pass the Version Id of this version.
the the

	WABS	AS3
Get Blob Properties/HEAD Object	Yes	Yes

Function used to retrieve BLOB properties and metadata of the object, but does not return the contents of BLOB.

Comments:

the

Get Blob Properties in WABS returns the set of user defined metadata, standard HTTP properties, and system properties for the BLOB, while the HEAD Object in AS3 only returns the metadata of the object.
WABS allows you to define the preconditions that must be satisfied for the successful completion of this function (IfModifiedSince, IfUnmodifiedSince, the IfMatch, IfNoneMatch).

This function can be used to retrieve the properties of a particular version of a BLOB or object. To obtain this information you must specify the date/time the snapshot BLOB in WABS and Version Id of the necessary version in AS3. If these parameters are omitted, returns information about the current version.

the the

	WABS	AS3
Set Blob Properties	Yes

Function specifies the system properties of the BLOB in WABS. Not available in AS3. When you create an object, you can specify only system properties.

Properties are available to define, include: Cache Control, Content Type, Content MD5, Content Encoding, Content Language. Please note that you do not have the right to override any of these properties for the snapshot of the BLOB.

WABS allows you to define the preconditions that must be satisfied for the successful completion of this function (IfModifiedSince, IfUnmodified-Since, IfMatch, the IfNoneMatch).
the the

	WABS	AS3
Get Blob Metadata/Head Object	Yes	Yes

The function returns user-defined metadata. This function can be used to retrieve the properties of a particular version of a BLOB or object. To obtain this information you must specify the date/time the snapshot BLOB in WABS and Version Id of the necessary version in AS3. If these parameters are omitted, returns information about the current version. the the

	WABS	AS3
Set Blob Metadata	Yes

Function is used to specify the dictionary metadata in a collection of records a key value for blobs in WABS.
Comments:

the

Function overwrites the existing metadata, so you can't just upgrade one pair key-value.
Maximum metadata size is 8 KB.
metadata Name must be authorized by C# identifier.

the the

	WABS	AS3
Delete Blob/DELETE Object	Yes	Yes

The function deletes the BLOB or object from the repository.

Comments:

the

In AS3, the function removes the null version (if any) of the object and inserts the token for deletion, which then becomes the version of the object. If not null, no objects are deleted.
IN WABS, you can use this function to delete only snapshots without deleting the source. If the source is deleted, all its snapshots are also deleted.
As I understand, in AS3 when you delete an object all versions of it are still intact, which I think is quite convenient protection from accidental delete operations.

This function can be used to remove specific versions of a BLOB or object you need to specify the date/time the snapshot BLOB in WABS and Version Id of the necessary version in AS3.

· WABS allows you to define the preconditions that must be satisfied for the successful completion of this function (IfModifiedSince, IfUnmodified-Since, IfMatch, the IfNoneMatch).

the

In AS3 you can use the Multi Factor Authentification (MFA) Delete, which provides an additional level of security. Read more about MFA SDAs.

the the

	WABS	AS3
Delete Multiple Objects		Yes

This function allows you to delete in one HTTP request multiple objects from basket – to do this you must know the keys of the deleted objects. You can delete specific versions of one or more objects. In one query you can delete up to 1000 objects.
the the

	WABS	AS3
Copy Blob/Put Object – Copy	Yes	Yes

The function copies the BLOB somewhere else from its original location.

Comments:

· Both systems allow you to specify preconditions which must be satisfied for the successful completion of this function (IfModifiedSince, IfUnmodified-Since, IfMatch, the IfNoneMatch). These terms can be defined as the source and destination of the copy in WABS and on a source in AS3.

· WABS enables you to copy objects from container to container within a storage account. In AS3 there is no such restriction. If baskets between whom the exchange takes place, belong to the same account, the object is copied. However, if you created the object, using the API to load the chunks, you will not be able to copy an object from one region to another.

· Both systems allow you to copy the existing metadata or to specify the metadata for the final copy.

· AS3 implements this functionality as a combination GET Object and PUT Object.

the

to Use this function with objects no more than 5 GB in size. If the size exceeds 5 GB, it is necessary to make loading of chunks.
In AS3 when you copy a specific ACL is removed and the object is the authorization as private.

the Helpful hints:

the

Both not support the renaming of the object. The renaming of an object can be implemented by first copying the object then deleting it.

the the

	WABS	AS3
Snapshot Blob	Yes

The function creates a read-only copy of the BLOB – implemented versioning WABS. As I mentioned in previous articles, AS3 independently implements versioning. In WABS the same versioning is implemented at the level of individual BLOB and this BLOB versioning goes into the responsibility of the developer.

Comments:

the

function creates a Read-only copy of the BLOB snapshot. Snapshots may be downloaded, copied or deleted but not modified.
unlike AS3, where the object version is assigned a Version Id to each snapshot assigned the value of the date/time stamp that uniquely identifies the snapshot.

· WABS allows you to define the preconditions that must be satisfied for the successful completion of this function (IfModifiedSince, IfUnmodified-Since, IfMatch, the IfNoneMatch).

the

Can be used to raise the snapshot to the version of the BLOB source code, using the Copy Blob.

the the

	WABS	AS3
Lease Blob	Yes

Feature allows you to put a one-minute lock on a BLOB, so it could not modify. Function is very convenient in a situation with many worker-s, trying to make the same operation, you want only one of them actually did change. Read more: http://blog.smarx.com/posts/managing-concurrency-in-windows-azure-with-leases.

the the

	WABS	AS3
Initiate Multipart Upload		Yes

Function is used to initiate the download of data pieces.
the the

	WABS	AS3
Block Put/Upload Part	Yes	Yes

Function is used for loading pieces (blocks in WABS and parts in AS3) of the data.
the the

	WABS	AS3
Put Block List/Complete Multipart Upload	Yes	Yes

Function is used to confirm the blobs or objects in the appropriate repository. You can call this function after downloading all the blocks or parts.
the the

	WABS	AS3
Get Block List/List Parts	Yes	Yes

The function returns a list of loaded blocks or parts.

Comments:

the

WABS has two list boxes for the BLOB – list successfully confirmed blocks and unconfirmed list of blocks that have been loaded but have not been confirmed.
In a single request function AS3 will return to 1000 numbered pieces. If there are still parts, AS3 will return a continuation token. You can specify how many rooms will be returned.

· WABS allows you to define the preconditions that must be satisfied for the successful completion of this function (IfModifiedSince, IfUnmodified-Since, IfMatch, the IfNoneMatch).

the the

	WABS	AS3
Abort Multipart Upload		Yes

Function is used to cancel the download process.

In this form, in which implemented this functionality in WABS, you don't need it. In WABS you can't interrupt the download process. If you started downloading blobs chunks and not caused Put Block List within 7 days, the downloaded blocks will be removed.
the the

	WABS	AS3
Upload Part – Copy		Yes

If the object you are trying to copy has a size larger of 5 GB, you will not be able to use Put Object – Copy. You will have to copy this object by loading chunks, this feature is designed to do just that. The difference between this function and the Upload Part is that instead of sending data in the request body, you specify a source object in the format source_bucket/source_object and the range of bytes to be copied. Thus the need to do this to all the parts of the copied object. Before calling this function, you need to initiate the download of the pieces and get the Upload Id, which you must pass to the function. To finish copying, you need to call “Complete Multipart Upload” or “Abort Multipart Upload”.
the the

	WABS	AS3
Put Page	Yes

When you create a page BLOB using Put Blob, you only initiate its creation, i.e. create an empty page BLOB without content. Using this function, you can insert/update the data in page BLOB, and to confirm the data is not necessary after calling this function, they will be automatically confirmed.

Comments:

the

To put data in a BLOB you must specify the value of the range header byte “Byte Range”. The initial shift range must be the absolute value of 512-1. Examples of allowed ranges bytes 0-511, 512-1023, and so on.
With this function you can either record the data in page BLOB, or to clear data from a specific range of bytes.

· WABS allows you to define the preconditions that must be satisfied for the successful completion of this function (IfModifiedSince, IfUnmodified-Since, IfMatch, the IfNoneMatch).

the the

	WABS	AS3
Get Page Ranges	Yes

An important difference between block and page BLOB is the pricing – you pay for a block BLOB completely, but in the case of a page BLOB only for occupied (non-zero) pages. For example, if you create a block BLOB on 2 GB and page BLOB on 2 GB and both of them unfilled (contains 0 bytes), you will pay for 2 GB block BLOB, page BLOB the fee will not be charged. Then you write the 1024 bytes in page BLOB using Put Page, and begin to pay for these 1024 bytes.

This function returns a sorted list of non-zero pages.

This function has another advantage – if you when downloading a page BLOB will call this function, you will be guaranteed to know where the BLOB is stored your data, then you will be able to download the entire BLOB, and only busy pages, with the use of a Range header in Get Blob function.

Summary

Both systems provide a similar set of functions. Each system has its own set of functions, which is absent in other system, but in General differences in functionality are not the same as in the case of BLOB containers and baskets.
the