DotNetCurry Logo

Azure Blob Storage Snapshots using Client Library and REST APIs

Posted by: Kunal Chandratre , on 1/15/2015, in Category Microsoft Azure
Views: 10808
Abstract: Azure Blob Storage Snapshots are the best option to provide read only access, avoid accidental writes and restore blobs to their previous state. This article explores how to use Snapshots using Client Library and REST APIs

A Blob storage can be thought of as a special table storage in the cloud. Azure Blob storage is a service for storing large amounts of data in binary format as data chunks that can be accessed via HTTP or HTTPS. One of the coolest features of the Azure blob storage is the ability to create snapshots of a blob file. The snapshot allows you to preserve the blob as of a specific point in time.

This article demonstrates various operations that can be used to create and delete snapshots as well as restore blob from snapshots using client library and REST APIs.

This article is published from the DotNetCurry .NET Magazine – A Free High Quality Digital Magazine for .NET professionals published once every two months. Subscribe to this eMagazine for Free and get access to hundreds of free .NET tutorials from experts


 

Introduction to Azure Blob Storage

Azure Storage is very widely used service in most Azure based applications. I personally have not seen a single project running on Azure and not using the Azure Storage. Azure Storage has four main offerings –

1. Blob

2. Table

3. Queue

4. Drive – Read Drives as Azure Files now as Drives will be deprecated in the year 2015.

Out of these offerings, this article will focus on the snapshot feature of Azure Blob storage and how it can be effectively used in azure based applications. This article assumes that you have basic background of Azure Blob storage concept and its primary use.

If you refer to the dictionary meaning of Snapshot, it states that – a photograph, taken with a handheld camera with no specific aim. To a developer’s delight, we can take snapshot of Azure blob storage. Of course not with the Camera, but it serves a similar purpose i.e. capture an instance at a specific point in time. Blob snapshots have a specific aim. This aim is of improving performance and maintaining backup or original copy of the blob present in azure storage.

Exploring the Azure Blob Snapshot

Snapshot offers you a read-only copy of the blob storage. As it is read-only, you can read, copy and delete, but cannot edit/modify. Blob snapshots are very fruitful in case of Azure Virtual Machines. For example, let’s say you are configuring non Microsoft software on Azure Virtual Machines and it involves minimum 50 to 60 steps to complete the configuration. You also need to make sure that after each step performed, VM is in a running condition and there are no errors. If at all an error occurs, you want to revert back to the previous working/ running condition of VM. Such scenarios are ideal for the use of Azure blob Storage snapshot. This is very similar to Team Foundation Server (TFS) capability where you maintain the versions of your code. Here you maintain different versions of blob differentiated by Date on which the snapshot is taken.

Create snapshot for a blob

The following steps will explain how you can create snapshot for a blob. For demo purposes, I am using Azure Storage Emulator. The Microsoft Azure storage emulator provides a local environment that emulates the Azure Blob, Queue, and Table services and allows you test your application without paying any money. Before creating a snapshot, it is important to upload a blob to Azure storage first. I used the standard code available here (http://azure.microsoft.com/en-us/documentation/articles/storage-dotnet-how-to-use-blobs/ ) to perform the upload to azure blob storage. I just made one change in this code. While uploading the content to blob storage, I also added property and metadata to the block blob with following code:

blockBlob.Properties.CacheControl = "max-age=60, must-revalidate";
blockBlob.SetProperties();

blockBlob.Metadata.Add("author", "kunal");

Now I opened the server explorer from Visual Studio 2013 and expanded the storage node for development environment, where I can see the blob uploaded with added property information as shown in the screenshot below

azure-blob-properties

Let us see how to programmatically create Blog snapshot using Client Library and REST API:

a.Using Azure Client Library –

The following method illustrates the create snapshot code using Azure storage client library -

private static void CreateSnapshotUsingClientLibrary()
{
    //specifying container name and blob name - change them as per your blob and container name
    string containerName = "mycontainer";
    string blobName = "AzureBootCamp.zip";

    // Retrieve storage account from connection string
    CloudStorageAccount storageAccount = CloudStorageAccount.Parse(storageConnectionString);

    // Create the blob client
    CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();

    // Get a reference to the container  and blob
    CloudBlobContainer container = blobClient.GetContainerReference(containerName);
    CloudBlockBlob blob = container.GetBlockBlobReference(blobName);
    
    //create snapshot
    CloudBlockBlob snapshot = blob.CreateSnapshot();

    //list down the Uri of snapshot created
    Console.WriteLine("SnapshotQualifiedUri: " + snapshot.SnapshotQualifiedUri);
    Console.WriteLine("Snap shot time:" + snapshot.SnapshotTime);
}

The Output window will display the URL of the snapshot taken. The format of the snapshot url is as shown here:

snapshot-url-format

b. Using REST API

For REST API, it is mandatory to provide the Authorization, Date and x-ms-version information while snapshot creation. For authorization, we will use Shared Access Signature(SAS) created on blob storage. The code to generate SAS based on the stored access policy is pretty straight forward and can be obtained from the link –

http://sanganakauthority.blogspot.com/2012/05/windows-azure-shared-access-signature.html

In this sample, we are using similar code to generate Ad-Hoc type of SAS. To create snapshot using REST API, you need to provide query string parameter as – &comp=snapshot

The complete method to create a snapshot is as follows –

private static void CreateSnapshotUsingREST()
{
    try
    {
        //specifying container name and blob name - change them as per your blob and container name
        string containerName = "mycontainer";
        string blobName = "AzureBootCamp.zip";

        string contentType = string.Empty;
        string snapshotTime = string.Empty;
        DateTime now = DateTime.UtcNow;

        //to perform any operation first lets generate the SAS url on the container, validity 1 minute
        SASManager sasMgr = new SASManager(storageConnectionString);
        string sasUrl = sasMgr.GetAdHocWriteOperationSAS(containerName, blobName);

        //perform operation to create snapshot
        HttpWebRequest requestCreateSnapshot = (HttpWebRequest)WebRequest.Create(sasUrl + "&comp=snapshot");
        requestCreateSnapshot.ContentLength = 0;                
        requestCreateSnapshot.Headers.Add("x-ms-version", "2014-02-14");
        requestCreateSnapshot.Headers.Add("x-ms-date", now.ToString("R", System.Globalization.CultureInfo.InvariantCulture));
        requestCreateSnapshot.Method = "PUT";

        using (HttpWebResponse respCreateSnapshot = (HttpWebResponse)requestCreateSnapshot.GetResponse())
        {
            if (respCreateSnapshot.StatusCode == HttpStatusCode.Created)//create operation returns CREATED response
            {
                if (respCreateSnapshot.Headers != null)
                {
                    snapshotTime = respCreateSnapshot.Headers.Get("x-ms-snapshot");
                }
            }
        }
        Console.WriteLine("snapshot time - " + snapshotTime);
    }
    catch (Exception ex)
    {
        throw ex;
    }
}

The snapshot time is the time value at which snapshot got created and unique identifier for your snapshot. So this parameter will be applied at the end of SAS url. Therefore full url to create snapshot of the blob will be as follows –

http://127.0.0.1:10000/devstoreaccount1/mycontainer/AzureBootCamp.zip?snapshot=2014-09-25T10:38:31.5670000Z

If you see the image snapshot-properties.png; you can observe that snapshot has all the properties/ metadata exactly same as original blob.

Listing Blob snapshots

The snapshot is distinguished by its time stamp. The time at which snapshot is taken is simply appended as query string to existing URL of the blob storage and that forms the complete url for snapshot of the blob. The same property can be used to list down snapshots. Let’s see how to do it using Client Library and REST API.

a.Using Client Library –

private static void ListSnapshotsForBlob()
{
    //specifying container name and blob name - change them as per your blob and container name
    string containerName = "mycontainer";
    string blobName = "AzureBootCamp.zip";

    // Retrieve storage account from connection string
    CloudStorageAccount storageAccount = CloudStorageAccount.Parse(storageConnectionString);

    // Create the blob client
    CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();

    // Get a reference to the container  and blob
    CloudBlobContainer container = blobClient.GetContainerReference(containerName);
    CloudBlockBlob blob = container.GetBlockBlobReference(blobName);
    
    //retrive list of snapshots
    IList snapshots = container.ListBlobs(null, true, BlobListingDetails.Snapshots).Where(x => ((CloudBlockBlob)x).IsSnapshot).ToList();

    //write total number of snapshots for blob
    Console.WriteLine("Total snapshots:" + snapshots.Count + Environment.NewLine);
    
    foreach (IListBlobItem snapshot in snapshots)
    {                
        Console.WriteLine("Snapshot Uri:" + ((CloudBlockBlob)snapshot).SnapshotQualifiedUri + Environment.NewLine + "Snapshot Timestamp:" + ((CloudBlockBlob)snapshot).SnapshotTime);
    }
}

The following screenshot shows the snapshot created on various timestamp for the blob.

snapshot-list

b.Using REST API

private static void ListSnapshotsUsingREST()
{
    try
    {
        //specifying container name and blob name - change them as per your blob and container name
        string containerName = "mycontainer";
        List snapshots = new List();

        //to perform any operation first let’s generate the SAS url on the container, validity 1 minute
        SASManager sasMgr = new SASManager(storageConnectionString);
        string sasUrl = sasMgr.GetAdHocListSnapshotsSAS(containerName);

        //perform operation to create snapshot
        HttpWebRequest request = (HttpWebRequest)WebRequest.Create(sasUrl + "&restype=container&comp=list&include=snapshots");
        request.ContentLength = 0;
        request.Headers.Add("x-ms-version", "2014-02-14");
        request.Method = "GET";

        using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
        {
            if (response.StatusCode == HttpStatusCode.OK)//list operation returns OK response
            {
                using (StreamReader reader = new StreamReader(response.GetResponseStream()))
                {
                    string result = reader.ReadToEnd();

                    //read snapshot from xml
                    XElement x = XElement.Parse(result);
                    foreach (XElement blob in x.Element("Blobs").Elements("Blob"))
                    {
                        if (blob.Element("Snapshot") != null)
                        {
                            snapshots.Add(blob.Element("Snapshot").Value);
                        }
                    }
                }
            }
        }
        //print snapshots name
        foreach (string snapshot in snapshots)
        {
            Console.WriteLine("Snapshot Name:" + snapshot);
        }
    }
    catch (Exception ex)
    {
        throw ex;
    }
}

Restore blob from snapshot

You can restore to the previous version of the blob storage using Snapshots. This is the main purpose of blob snapshot. The following code illustrates how you can restore blob from snapshot. For the restore purpose, I am retrieving the first snapshot and using its url to perform restore operation.Using Client library

a. Using Client library

private static void RestoreFromSnapshot()
{
    //specifying container name and destination blob name - change them as per your destination blob and container name
    string containerName = "mycontainer";            
    string destinationBlobName = "AzureBootCampRestored.zip";            

    // Retrieve storage account from connection string
    CloudStorageAccount storageAccount = CloudStorageAccount.Parse(storageConnectionString);

    // Create the blob client
    CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();

    // Get a reference to the container and blob
    CloudBlobContainer container = blobClient.GetContainerReference(containerName);            
    CloudBlockBlob destinationBlob = container.GetBlockBlobReference(destinationBlobName);

    //retrive the snapshot url from the first snapshot - here you can retrive or use snapshot url of your choice. For demo purpose I am considering the first snapshot here.
    string snapshotUri = ((CloudBlockBlob)(container.ListBlobs(null, true, BlobListingDetails.Snapshots).Where(x => ((CloudBlockBlob)x).IsSnapshot).FirstOrDefault())).SnapshotQualifiedUri.AbsoluteUri;
    //or you can specify the snapshot uri of your choice as below
    //string snapshotUri = "http://127.0.0.1:10000/devstoreaccount1/mycontainer/AzureBootCamp.zip?snapshot=2014-09-19T05:29:50.4570000Z";            

    //perform copy/restore operation from snapshot uri
    string taskId = destinationBlob.StartCopyFromBlob(new Uri(snapshotUri));

    Console.WriteLine("Restore blob url task Id:" + taskId);

    while (destinationBlob.CopyState.Status == CopyStatus.Pending)
    {
        Task.Delay(TimeSpan.FromSeconds(20d)).Wait();
        destinationBlob = (CloudBlockBlob)container.GetBlobReferenceFromServer(destinationBlobName);
    }
    Console.WriteLine("Copy operation complete");
}

After successful restore operation, a new blob gets created having same properties/ metadata as that of snapshot as shown in the image

restored-snapshot

b. Using REST API

In case of a REST based example, I am using the hardcoded snapshot uri parameter and it will change as per your snapshot.

private static void RestoreFromSnapshotUsingREST()
{
    try
    {
        //specifying container name and blob name - change them as per your blob and container name
        string containerName = "mycontainer";//my source and destination containers are same.
        string blobName = "AzureBootCamp.zip";                
        string destinationBlobName = "AzureBootCampRestored.zip";
        string copyStatus = string.Empty;                

        //to perform any operation first, let’s generate the SAS url on the source container-blob, validity 1 minute
        SASManager sasMgr = new SASManager(storageConnectionString);
        string sasUrl = sasMgr.GetAdHocWriteOperationSAS(containerName, blobName);

        //create sas on destination container-blob
        string destinationSASUrl = sasMgr.GetAdHocWriteOperationSAS(containerName, destinationBlobName);

        //create request for destination blob
        HttpWebRequest request = (HttpWebRequest)WebRequest.Create(destinationSASUrl);
        request.ContentLength = 0;                
        request.Headers.Add("x-ms-version", "2014-02-14");
        //add source information - this will be snapshot in our case. The uri query parameter of snapshot will be different for your snapshot
        request.Headers.Add("x-ms-copy-source", sasUrl + "&snapshot=2014-09-19T05:29:50.4570000Z");
        request.Method = "PUT";

        using (HttpWebResponse respCreateSnapshot = (HttpWebResponse)request.GetResponse())
        {
            if (respCreateSnapshot.StatusCode == HttpStatusCode.Accepted)//create operation returns ACCEPTED response
            {
                if (respCreateSnapshot.Headers != null)
                {
                    //retrive copy state information from header
                    copyStatus = respCreateSnapshot.Headers.Get("x-ms-copy-status");
                    while (copyStatus != "success")
                    {
                        copyStatus = respCreateSnapshot.Headers.Get("x-ms-copy-status");
                        Thread.Sleep(500);
                    }
                }
            }    
        }
        Console.WriteLine("restore operation status- " + copyStatus);
    }
    catch (Exception ex)
    {
        throw ex;
    }
}

So essentially restoring blob from snapshot is nothing more than a copy blob operation. In the above code sample, if I provide the destination blob name as original blob only (in above case it is "AzureBootCamp.zip") then original blob will be replaced entirely by the snapshot. This is how we restore the original blob. I don’t want my original blob to be replaced therefore I performed copy operation with different destination blob name.

Note- If you use the original blob to perform copy operation then all the associated snapshots of the original blob will not get copied.

Delete Azure Blob Snapshot

The way we used the copy operation of blob storage to perform restore operation; to delete blob snapshot, we need to use Delete Blob operation. Here too, I am retrieving the list of available snapshots to perform delete operation on it. First I am calling List blob snapshot operation to list total number of snapshots of the blob and after the delete operation, I again list the remaining snapshot. As expected, the deleted snapshot will not appear in the listing after the delete operation occurs. If your blob does not have any snapshot associated, then you may end up with 404 Not Found exception or Object reference not to set to instance of an object.

We can delete individual snapshots for a blob. However, to delete a blob having associated snapshots, it is mandatory to delete all the associated snapshots first. If you try to delete the blob without deleting its snapshots then you will end up with 409 Conflict exception. This is what is demonstrated in the code we will see shortly for the client library based snapshot delete operation. To specify delete operation on snapshot enum DeleteSnapshotsOption can be used. Using this enum, you can specify options whether you wish to delete all snapshots but retain a blob OR delete blob along with all snapshots. Following is the demonstration of deleting of snapshots but retaining blob.

a. Using Client Library

private static void DeleteSnapshotForBlob()
{
    try
    {
        //list the total number of snapshot present currently
        ListSnapshotsForBlob();

        //specifying container name and blob name - change them as per your blob and container name
        string containerName = "mycontainer";
        string blobName = "AzureBootCamp.zip";                

        // Retrieve storage account from connection string
        CloudStorageAccount storageAccount = CloudStorageAccount.Parse(storageConnectionString);

        // Create the blob client
        CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();

        // Get a reference to the container and blob snapshot
        CloudBlobContainer container = blobClient.GetContainerReference(containerName);
        DateTimeOffset? snapshotTime = ((CloudBlockBlob)(container.ListBlobs(null, true, BlobListingDetails.Snapshots).Where(x => ((CloudBlockBlob)x).IsSnapshot).FirstOrDefault())).SnapshotTime;

        CloudBlockBlob snapshot = container.GetBlockBlobReference(blobName, snapshotTime);

        #region Delete individual snapshot

        //delete individual snapshot
        snapshot.Delete();
        Console.WriteLine("Individual Snapshot delete operation complete");

        //list total number snapshot remaining after deleting one
        ListSnapshotsForBlob();

        #endregion

        #region Delete all snapshots but retain original blob

        CloudBlockBlob blob = container.GetBlockBlobReference(blobName);
        blob.Delete(DeleteSnapshotsOption.DeleteSnapshotsOnly);
        ListSnapshotsForBlob();

        #endregion
    }
    catch (Exception ex)
    {
        Console.WriteLine(ex.Message);
    }
}

b. Using REST API

private static void DeleteSnapshotForBlobUsingREST()
{
    try
    {
        //specifying container name and blob name - change them as per your blob and container name
        string containerName = "mycontainer";//my source and destination containers are same.
        string blobName = "AzureBootCamp.zip";                                

        //to perform any operation first lets generate the SAS url on the source container-blob, validity 1 minute
        SASManager sasMgr = new SASManager(storageConnectionString);
        string sasUrl = sasMgr.GetAdHocWriteOperationSAS(containerName, blobName);                

        //create request for destination blob
        HttpWebRequest request = (HttpWebRequest)WebRequest.Create(sasUrl);
        request.ContentLength = 0;
        request.Headers.Add("x-ms-version", "2014-02-14");
        //specify to delete all snapshots but retain the blob
        request.Headers.Add("x-ms-delete-snapshots", "only");
        request.Method = "DELETE";

        using (HttpWebResponse respCreateSnapshot = (HttpWebResponse)request.GetResponse())
        {
            if (respCreateSnapshot.StatusCode == HttpStatusCode.Accepted)//delete operation returns ACCEPTED response
            {                        
            }
        }
        Console.WriteLine("delete snapshots operation successfull");
        ListSnapshotsForBlob();
    }
    catch (Exception ex)
    {
        throw ex;
    }
}

Costing model of Blob Snapshots

Costing model of blob snapshots is very interesting. However before understanding costing model of snapshot, it is important to understand overall costing model of blob storage first. The blob storage costing is based on 3 main drivers –

1. Bandwidth or data transfer – This means data ingress and outgress. In other words you are charged for data going in data center and data coming out of data center. Bandwidth has another important aspect here. Data ingress is FREE, data transfer in/out inside the same data center is FREE, however data coming out of the data center is charged.

2. Transaction – This means you are charged based on number of operations you perform against blob storage. For example, if you read or write to blob storage, you are charged for one transaction.

3. Capacity – This means the amount of storage consumed by your data. For example, if you say my blob is 5GB means you will be charged for 5GB of capacity.

Now with the understanding of these main factors of blob storage cost, let’s understand the costing of Azure Blob snapshots.

1. Exactly identical

Bandwidth and transaction cost for snapshots is very similar to that of blob storage; however capacity cost is way different than the blob storage. As long as your snapshot “is exactly identical to original blob”, no additional cost of capacity is incurred for snapshots. However the meaning of “exact identical to original blob” is very important. For capacity, you are always charged based on number of unique blocks for block type of blobs and number of unique pages for page type of blobs. If you have any snapshot same as original blob and if you update a single block from base blob, you will be charged for corresponding unmatched blocks of snapshot as well. To understand this scenario understand the following snapshot:

azure-costing-scenario

Azure blob storage does not have a way by which we can determine if the blocks are identical in all respects. Therefore whenever an update is performed for the block, irrespective of the same id or data, the updated blocks are treated as unique and hence charged subsequently.

2. Expensive methods for Snapshots

All methods having Upload keyword are expensive for snapshots. For example UploadFile, UploadText, UploadStream, UploadByteArray are the methods which replaces all the blocks in a blob, thereby making your snapshot and base blob exactly NON-IDENTICAL to each other. Hence you will be charged for base blob plus all the snapshots.

Best Practice for Snapshots management

When you want to perform an update to the blob, it is recommended to delete all the associated snapshots of the blob using client library or REST option as demonstrated above and then create snapshots again. This ensures base blob and snapshots are identical and no extra charge in incurred to you.

As far as possible, try to use PutBlock and PutBlockList method to perform incremental updates to the blob storage instead of a complete replacement of blob using upload specific methods.

Conclusion

Snapshots are the best way to provide read only access, avoid accidental writes and restore blobs to previous state. If used wisely, they can prove very beneficial in terms of performance.

Download the entire source code from our GitHub Repository at bit.ly/dncm15-azuresnapshot

Was this article worth reading? Share it with fellow developers too. Thanks!
Share on Google+
Further Reading - Articles You May Like!
Author
Kunal Chandratre is a Microsoft Azure MVP. He is working as Azure SME in leading software company in (Pune) India. He also works as a Freelancer with various organizations for Azure support and provides quick start trainings on Azure to corporates and individuals on weekends. He regularly blogs about his Azure experience and is a very active member in various Microsoft Communities and also participates as a ‘Speaker’ in many events. You can follow him on Twitter at: @kunalchandratre or subscribe to his blog


Page copy protected against web site content infringement 	by Copyscape




Feedback - Leave us some adulation, criticism and everything in between!
Comment posted by Sanjay on Friday, January 16, 2015 4:45 AM
Good job Kunal. I downloaded all issues of your magazine and will read them this weekend.