Uploading Multiple Files in Chunks from ASP.NET MVC app to Azure Blob Storage

Posted by: Suprotim Agarwal , on 5/10/2013, in Category Windows Azure
Views: 17591
Abstract: Demo showing how to send files ‘chunked’ to an ASP.NET MVC Web Application which in turn stores the files in Azure Blob storage.

In this third part of our Azure Blob Storage series we will overcome a shortcoming that we saw in the previous ‘chunked’ file upload solution. Previously we were not able to utilize Html5 File Upload input dialog’s multiple file capability. This time, we’ll use it and see how we can send chunks of files in parallel to the server and upload multiple files at the same time.

We will not touch upon the Azure blob storage part or how chunking is working. We urge you to go through our previous article for those details.

Upload Multiple Files to Azure Storage and Manage them from an ASP.NET MVC 4 Application – Part I

Uploading Big files to Azure Storage from ASP.NET MVC – Part II

 

The JavaScript Client

Building a View Model

Since we have multiple files to upload, we must create a view model object for each file and store the details like, file, size, number of blocks it’s broken into, it’s position (index) in the list of files and so on. Towards this, we create the following JS object.

var chunkedFileUploader =
{
maxRetries: 3, // Number of retries before upload is aborted
blockLength: 1048576, // Size of chunk 1 MB
numberOfBlocks: 1, // The total number of chunks for the file
currentChunk: 1, // The current chunk being uploaded
retryAfterSeconds: 3, // Seconds after which to retry failed upload
fileToBeUploaded: null, // The File object from the HttpRequest
size: 0, // Size of the file
fileIndex: 0, // The index of the file in the list that is to be uploaded
name: "", // The file Name
init: function (file, index)
{
  this.fileToBeUploaded = file;
  this.size = file.size;
  this.name = file.name;
  this.fileIndex = index;
},

}

One object of chunkedFileUploader is initialized for every file selected with the File Input dialog. All these instances are stored in a global Knockout View Model object called uploaders. Uploaders has a single property called uploaderCollection and an uploadAll method that fires off the Upload process for each file. The uploaders object is as follows

var uploaders = {
uploaderCollection: ko.observableArray([]),
uploadAll: function ()
{
  for (var i = 0; i < this.uploaderCollection().length; i++)
  {
   var cful = this.uploaderCollection()[i];
   cful.uploadMetaData();
  }
}
}


multiple-file-upload-empty

The uploaders object is initialized when user selects the files and clicks on the Upload button. In the Upload button’s click handler we loop through all the files present selected and create a chunkedFileUploader instance and add them to the uploaderCollection.

$(document).ready(function ()
{
$(document).on("click", "#fileUpload", beginUpload);
ko.applyBindings(uploaders);
});

var beginUpload = function ()
{
var fileControl = document.getElementById("selectFile");
if (fileControl.files.length > 0)
{
  uploaders.uploaderCollection.removeAll();
  for (var i = 0; i < fileControl.files.length; i++)
  {
   cful = Object.create(chunkedFileUploader);
   cful.init(fileControl.files[i], i);
   uploaders.uploaderCollection.push(cful);
  }
  $(".progressBar").progressbar(0);
  uploaders.uploadAll();
}
}

The uploader collection is bound to a <ul> template. The template renders as many progress bars as the files selected. Once all the files have been loaded, we fire up the uploadAll function in the KO ViewModel. This loops through each object in the uploaderCollection and calls the uploadMetaData method.

uploadMetaData: function ()
{
this.numberOfBlocks = Math.ceil(this.size / this.blockLength);
this.currentChunk = 1;
$.ajax({
  type: "POST",
  async: true,
  url: "/Home/SetMetadata?blocksCount=" + this.numberOfBlocks
   + "&fileName=" + this.name
   + "&fileSize=" + this.size
   + "&fileIndex=" + this.fileIndex,
  }).done(function (state)
  {
   if (state.success == true)
   {
    var cufl = uploaders.uploaderCollection()[state.index]
    cufl.displayStatusMessage(cufl, "Starting Upload");
    cufl.sendFile(cufl);
   }
  });
},

The uploadMetaData method calculates the numberOfBlocks the file is going to be split into and sends the fileName, fileSize and the fileIndex to the server. The server stores this information in the session and returns success. Once we get success, we fire the displayStatusMessage(…) saying we are starting upload, and immediately call the sendFile method of the chunkedFileUploader object. This happens for each file independently. This is no shared state, each object is self-contained.

The sendFile function internally calls the sendNextChunk function which slices up the file based on the current position and sends it to the server.

When the server returns a success, the chunk position is updated, the start and end positions re-calculated, and sendNextChunk method is called recursively till all chunks are uploaded successfully.

sendFile: function (uploader)
{
var start = 0,
  end = Math.min(uploader.blockLength, uploader.fileToBeUploaded.size),
  retryCount = 0,
  sendNextChunk, fileChunk;
  this.displayStatusMessage(uploader,"");

var cful = uploader;
sendNextChunk = function ()
{
  fileChunk = new FormData();
  if (uploader.fileToBeUploaded.slice)
  {
   fileChunk.append('Slice', uploader.fileToBeUploaded.slice(start, end));
  }
  else if (uploader.fileToBeUploaded.webkitSlice)
  {
   fileChunk.append('Slice', uploader.fileToBeUploaded.webkitSlice(start, end));
  }
  else if (uploader.fileToBeUploaded.mozSlice)
  {
   fileChunk.append('Slice', uploader.fileToBeUploaded.mozSlice(start, end));
  }
  else
  {
   displayStatusMessage(cful, operationType.UNSUPPORTED_BROWSER);
   return;
  }
  jqxhr = $.ajax({
   async: true,
   url: ('/Home/UploadChunk?id=' + uploader.currentChunk + "&fileIndex=" +  
uploader.fileIndex),
   data: fileChunk,
   cache: false,
   contentType: false,
   processData: false,
   type: 'POST'
  }).fail(function (request, error)
  {
   if (error !== 'abort' && retryCount < maxRetries)
   {
    ++retryCount;
    setTimeout(sendNextChunk, retryAfterSeconds * 1000);
   }
   if (error === 'abort')
   {
    displayStatusMessage(cful, "Aborted");
   }
   else
   {
    if (retryCount === maxRetries)
    {
     displayStatusMessage(cful, "Upload timed out.");
     resetControls();
     uploader = null;
    }
    else
    {
     displayStatusMessage(cful, "Resuming Upload");
    }
   }
   return;
  }).done(function (state)
  {
   if (state.error || state.isLastBlock)
   {
    cful.displayStatusMessage(cful, state.message);
    return;
   }
   ++cful.currentChunk;
   start = (cful.currentChunk - 1) * cful.blockLength;
   end = Math.min(cful.currentChunk * cful.blockLength, cful.fileToBeUploaded.size);
   retryCount = 0;
   cful.updateProgress(cful);
   if (cful.currentChunk <= cful.numberOfBlocks)
   {
    sendNextChunk();
   }
  });
}
sendNextChunk();
},

As each chunk is uploaded the displayStatusMessage is called, this updates the respective progress bar and status labels.

multiple-file-upload-in-progress

Once the files are loaded, a success message is displayed as below

 

multiple-file-upload-complete

The Server Side

The server side code is almost the same except that we pass the File Index of every file, as we send the Chunk. This is because on the server side, each file’s metadata is keyed into the session using the File Index.

As we can see below, the CloudFile object has two new properties FileKey and FileIndex. The Key is created by concatenating the string “CurrentFile” and the fileIndex passed from the client. Once the FileKey is created, we store CloudFile object in the session using it.

[HttpPost]
public ActionResult SetMetadata(int blocksCount, string fileName, long fileSize, int fileIndex)
{
    var container = CloudStorageAccount.Parse(
        ConfigurationManager.AppSettings["StorageConnectionString"]).CreateCloudBlobClient()
        .GetContainerReference(ConfigurationManager.AppSettings["CloudStorageContainerReference"]);
    container.CreateIfNotExists();
    var fileToUpload = new CloudFile()
    {
        BlockCount = blocksCount,
        FileName = fileName,
        Size = fileSize,
        BlockBlob = container.GetBlockBlobReference(fileName),
        StartTime = DateTime.Now,
        IsUploadCompleted = false,
        UploadStatusMessage = string.Empty,
        FileKey = "CurrentFile" + fileIndex.ToString(),
        FileIndex = fileIndex
    };
    Session.Add(fileToUpload.FileKey, fileToUpload);
    return Json(new { success = true, index = fileIndex });
}

The CloudFile method stores in session, when the client calls UploadChunk method with the fileIndex. We restore the Storage Blob pointer from the Session and add the new chunk to the blob

[HttpPost]
[ValidateInput(false)]
public ActionResult UploadChunk(int id, int fileIndex)
{
HttpPostedFileBase request = Request.Files["Slice"];
byte[] chunk = new byte[request.ContentLength];
request.InputStream.Read(chunk, 0, Convert.ToInt32(request.ContentLength));
JsonResult returnData = null;
string fileSession = "CurrentFile" + fileIndex.ToString();
if (Session[fileSession] != null)
{
  CloudFile model = (CloudFile)Session[fileSession];
  returnData = UploadCurrentChunk(model, chunk, id);
  if (returnData != null)
  {
   return returnData;
  }
  if (id == model.BlockCount)
  {
   return CommitAllChunks(model);
  }
}
else
{
  returnData = Json(new
  {
   error = true,
   isLastBlock = false,
   message = string.Format(CultureInfo.CurrentCulture,
    "Failed to Upload file.", "Session Timed out")
  });
  return returnData;
}
return Json(new { error = false, isLastBlock = false, message = string.Empty, index = fileIndex });
}

Once all the chunks for a file has been saved, they are committed to Blob Storage as a Single blob via the CommitAllChunks method.

private ActionResult CommitAllChunks(CloudFile model)
{
model.IsUploadCompleted = true;
bool errorInOperation = false;
try
{
  var blockList = Enumerable.Range(1, 
   (int)model.BlockCount).ToList<int>().ConvertAll(rangeElement =>
   Convert.ToBase64String(Encoding.UTF8.GetBytes(
    string.Format(CultureInfo.InvariantCulture, "{0:D4}", rangeElement))));
  model.BlockBlob.PutBlockList(blockList);
  var duration = DateTime.Now - model.StartTime;
  float fileSizeInKb = model.Size / 1024;
  string fileSizeMessage = fileSizeInKb > 1024 ?
  string.Concat((fileSizeInKb / 1024).ToString(CultureInfo.CurrentCulture), " MB") :
  string.Concat(fileSizeInKb.ToString(CultureInfo.CurrentCulture), " KB");
  model.UploadStatusMessage = string.Format(CultureInfo.CurrentCulture,
   "File uploaded successfully. {0} took {1} seconds to upload",
   fileSizeMessage, duration.TotalSeconds);
}
catch (StorageException e)
{
  model.UploadStatusMessage = "Failed to Upload file. Exception - " + e.Message;
  errorInOperation = true;
}
finally
{
  Session.Remove(model.FileKey);
}
return Json(new
{
  error = errorInOperation,
  isLastBlock = model.IsUploadCompleted,
  message = model.UploadStatusMessage,
  index = model.FileIndex
});
}

The rest of the server side code is same as the Single File upload implementation we saw in our previous article.

Conclusion

We were able to implement multiple file upload in chunks to Azure Blob Storage. The client side implementation has a gotcha as in it uses recursion which can slow things down when uploading too many files at the same time. We can mitigate that by restricting the number of uploads possible at one time.

Download the entire source code of this article (Github)

Give a +1 to this article if you think it was well written. Thanks!
Recommended Articles
Suprotim Agarwal, ASP.NET Architecture MVP, MCSD, MCAD, MCDBA, MCSE, is the CEO of A2Z Knowledge Visuals Pvt. He primarily works as an Architect Consultant and provides consultancy on how to design and develop .NET centric database solutions.

Suprotim is the founder and primary contributor to DotNetCurry, DNC .NET Magazine, SQLServerCurry and DevCurry. He has also written an EBook 51 Recipes using jQuery with ASP.NET Controls. and is authoring another one at The Absolutely Awesome jQuery CookBook.

Follow him on twitter @suprotimagarwal


Page copy protected against web site content infringement by Copyscape


User Feedback
Comment posted by Nathan Nguyen on Monday, May 13, 2013 9:56 PM
Thanks Suprotim Agarwal for this Article, The Source code on Github is not have, please check it.
Comment posted by Suprotim Agarwal on Friday, May 17, 2013 7:15 AM
Sorry Nathan, the link has been fixed!
Comment posted by Ilija Injac on Thursday, May 23, 2013 9:04 AM
Hi Suprotim :) Thanks for this very pragmatic and practical approach! Nice one!
Comment posted by ryan on Monday, December 2, 2013 5:26 PM
Thanks Suprotim for the great article. When I tried the source code, the multiple upload to blob storage seems to be much slower than the single file upload. is this expected?
Comment posted by Suprotim Agarwal on Thursday, December 5, 2013 6:36 PM
@Ryan: are you transferring these files in parallel? There could be a number of reasons. Even after you are transferring files in chunks, a slow internet connection will lead to timeouts. Make sure your ops are async and that you are using concurrent threads relative to the CPU cores you have. Also if it is a big file, experiment with the chunk size depending on your connection speed and cores you have. Async is the key here.

Post your comment
Name:  
E-mail: (Will not be displayed)
Comment:
Insert Cancel