Uploading Multiple Files in Chunks from ASP.NET MVC app to Azure Blob Storage

In this third part of our Azure Blob Storage series we will overcome a shortcoming that we saw in the previous ‘chunked’ file upload solution. Previously we were not able to utilize Html5 File Upload input dialog’s multiple file capability. This time, we’ll use it and see how we can send chunks of files in parallel to the server and upload multiple files at the same time.

We will not touch upon the Azure blob storage part or how chunking is working. We urge you to go through our previous article for those details.

Upload Multiple Files to Azure Storage and Manage them from an ASP.NET MVC 4 Application – Part I

Uploading Big files to Azure Storage from ASP.NET MVC – Part II

The JavaScript Client

Building a View Model

Since we have multiple files to upload, we must create a view model object for each file and store the details like, file, size, number of blocks it’s broken into, it’s position (index) in the list of files and so on. Towards this, we create the following JS object.

var chunkedFileUploader =
{
maxRetries: 3, // Number of retries before upload is aborted
blockLength: 1048576, // Size of chunk 1 MB
numberOfBlocks: 1, // The total number of chunks for the file
currentChunk: 1, // The current chunk being uploaded
retryAfterSeconds: 3, // Seconds after which to retry failed upload
fileToBeUploaded: null, // The File object from the HttpRequest
size: 0, // Size of the file
fileIndex: 0, // The index of the file in the list that is to be uploaded
name: "", // The file Name
init: function (file, index)
{
  this.fileToBeUploaded = file;
  this.size = file.size;
  this.name = file.name;
  this.fileIndex = index;
},
…
}

One object of chunkedFileUploader is initialized for every file selected with the File Input dialog. All these instances are stored in a global Knockout View Model object called uploaders. Uploaders has a single property called uploaderCollection and an uploadAll method that fires off the Upload process for each file. The uploaders object is as follows

var uploaders = {
uploaderCollection: ko.observableArray([]),
uploadAll: function ()
{
  for (var i = 0; i < this.uploaderCollection().length; i++)
  {
   var cful = this.uploaderCollection()[i];
   cful.uploadMetaData();
  }
}
}

The uploaders object is initialized when user selects the files and clicks on the Upload button. In the Upload button’s click handler we loop through all the files present selected and create a chunkedFileUploader instance and add them to the uploaderCollection.

$(document).ready(function ()
{
$(document).on("click", "#fileUpload", beginUpload);
ko.applyBindings(uploaders);
});

var beginUpload = function ()
{
var fileControl = document.getElementById("selectFile");
if (fileControl.files.length > 0)
{
  uploaders.uploaderCollection.removeAll();
  for (var i = 0; i < fileControl.files.length; i++)
  {
   cful = Object.create(chunkedFileUploader);
   cful.init(fileControl.files[i], i);
   uploaders.uploaderCollection.push(cful);
  }
  $(".progressBar").progressbar(0);
  uploaders.uploadAll();
}
}

The uploader collection is bound to a <ul> template. The template renders as many progress bars as the files selected. Once all the files have been loaded, we fire up the uploadAll function in the KO ViewModel. This loops through each object in the uploaderCollection and calls the uploadMetaData method.

uploadMetaData: function ()
{
this.numberOfBlocks = Math.ceil(this.size / this.blockLength);
this.currentChunk = 1;
$.ajax({
  type: "POST",
  async: true,
  url: "/Home/SetMetadata?blocksCount=" + this.numberOfBlocks
   + "&fileName=" + this.name
   + "&fileSize=" + this.size
   + "&fileIndex=" + this.fileIndex,
  }).done(function (state)
  {
   if (state.success == true)
   {
    var cufl = uploaders.uploaderCollection()[state.index]
    cufl.displayStatusMessage(cufl, "Starting Upload");
    cufl.sendFile(cufl);
   }
  });
},

The uploadMetaData method calculates the numberOfBlocks the file is going to be split into and sends the fileName, fileSize and the fileIndex to the server. The server stores this information in the session and returns success. Once we get success, we fire the displayStatusMessage(…) saying we are starting upload, and immediately call the sendFile method of the chunkedFileUploader object. This happens for each file independently. This is no shared state, each object is self-contained.

The sendFile function internally calls the sendNextChunk function which slices up the file based on the current position and sends it to the server.

When the server returns a success, the chunk position is updated, the start and end positions re-calculated, and sendNextChunk method is called recursively till all chunks are uploaded successfully.

sendFile: function (uploader)
{
var start = 0,
  end = Math.min(uploader.blockLength, uploader.fileToBeUploaded.size),
  retryCount = 0,
  sendNextChunk, fileChunk;
  this.displayStatusMessage(uploader,"");

var cful = uploader;
sendNextChunk = function ()
{
  fileChunk = new FormData();
  if (uploader.fileToBeUploaded.slice)
  {
   fileChunk.append('Slice', uploader.fileToBeUploaded.slice(start, end));
  }
  else if (uploader.fileToBeUploaded.webkitSlice)
  {
   fileChunk.append('Slice', uploader.fileToBeUploaded.webkitSlice(start, end));
  }
  else if (uploader.fileToBeUploaded.mozSlice)
  {
   fileChunk.append('Slice', uploader.fileToBeUploaded.mozSlice(start, end));
  }
  else
  {
   displayStatusMessage(cful, operationType.UNSUPPORTED_BROWSER);
   return;
  }
  jqxhr = $.ajax({
   async: true,
   url: ('/Home/UploadChunk?id=' + uploader.currentChunk + "&fileIndex=" +  
uploader.fileIndex),
   data: fileChunk,
   cache: false,
   contentType: false,
   processData: false,
   type: 'POST'
  }).fail(function (request, error)
  {
   if (error !== 'abort' && retryCount < maxRetries)
   {
    ++retryCount;
    setTimeout(sendNextChunk, retryAfterSeconds * 1000);
   }
   if (error === 'abort')
   {
    displayStatusMessage(cful, "Aborted");
   }
   else
   {
    if (retryCount === maxRetries)
    {
     displayStatusMessage(cful, "Upload timed out.");
     resetControls();
     uploader = null;
    }
    else
    {
     displayStatusMessage(cful, "Resuming Upload");
    }
   }
   return;
  }).done(function (state)
  {
   if (state.error || state.isLastBlock)
   {
    cful.displayStatusMessage(cful, state.message);
    return;
   }
   ++cful.currentChunk;
   start = (cful.currentChunk - 1) * cful.blockLength;
   end = Math.min(cful.currentChunk * cful.blockLength, cful.fileToBeUploaded.size);
   retryCount = 0;
   cful.updateProgress(cful);
   if (cful.currentChunk <= cful.numberOfBlocks)
   {
    sendNextChunk();
   }
  });
}
sendNextChunk();
},

As each chunk is uploaded the displayStatusMessage is called, this updates the respective progress bar and status labels.

Once the files are loaded, a success message is displayed as below

The Server Side

The server side code is almost the same except that we pass the File Index of every file, as we send the Chunk. This is because on the server side, each file’s metadata is keyed into the session using the File Index.

As we can see below, the CloudFile object has two new properties FileKey and FileIndex. The Key is created by concatenating the string “CurrentFile” and the fileIndex passed from the client. Once the FileKey is created, we store CloudFile object in the session using it.

[HttpPost]
public ActionResult SetMetadata(int blocksCount, string fileName, long fileSize, int fileIndex)
{
    var container = CloudStorageAccount.Parse(
        ConfigurationManager.AppSettings["StorageConnectionString"]).CreateCloudBlobClient()
        .GetContainerReference(ConfigurationManager.AppSettings["CloudStorageContainerReference"]);
    container.CreateIfNotExists();
    var fileToUpload = new CloudFile()
    {
        BlockCount = blocksCount,
        FileName = fileName,
        Size = fileSize,
        BlockBlob = container.GetBlockBlobReference(fileName),
        StartTime = DateTime.Now,
        IsUploadCompleted = false,
        UploadStatusMessage = string.Empty,
        FileKey = "CurrentFile" + fileIndex.ToString(),
        FileIndex = fileIndex
    };
    Session.Add(fileToUpload.FileKey, fileToUpload);
    return Json(new { success = true, index = fileIndex });
}

The CloudFile method stores in session, when the client calls UploadChunk method with the fileIndex. We restore the Storage Blob pointer from the Session and add the new chunk to the blob

[HttpPost]
[ValidateInput(false)]
public ActionResult UploadChunk(int id, int fileIndex)
{
HttpPostedFileBase request = Request.Files["Slice"];
byte[] chunk = new byte[request.ContentLength];
request.InputStream.Read(chunk, 0, Convert.ToInt32(request.ContentLength));
JsonResult returnData = null;
string fileSession = "CurrentFile" + fileIndex.ToString();
if (Session[fileSession] != null)
{
  CloudFile model = (CloudFile)Session[fileSession];
  returnData = UploadCurrentChunk(model, chunk, id);
  if (returnData != null)
  {
   return returnData;
  }
  if (id == model.BlockCount)
  {
   return CommitAllChunks(model);
  }
}
else
{
  returnData = Json(new
  {
   error = true,
   isLastBlock = false,
   message = string.Format(CultureInfo.CurrentCulture,
    "Failed to Upload file.", "Session Timed out")
  });
  return returnData;
}
return Json(new { error = false, isLastBlock = false, message = string.Empty, index = fileIndex });
}

Once all the chunks for a file has been saved, they are committed to Blob Storage as a Single blob via the CommitAllChunks method.

private ActionResult CommitAllChunks(CloudFile model)
{
model.IsUploadCompleted = true;
bool errorInOperation = false;
try
{
  var blockList = Enumerable.Range(1, 
   (int)model.BlockCount).ToList<int>().ConvertAll(rangeElement =>
   Convert.ToBase64String(Encoding.UTF8.GetBytes(
    string.Format(CultureInfo.InvariantCulture, "{0:D4}", rangeElement))));
  model.BlockBlob.PutBlockList(blockList);
  var duration = DateTime.Now - model.StartTime;
  float fileSizeInKb = model.Size / 1024;
  string fileSizeMessage = fileSizeInKb > 1024 ?
  string.Concat((fileSizeInKb / 1024).ToString(CultureInfo.CurrentCulture), " MB") :
  string.Concat(fileSizeInKb.ToString(CultureInfo.CurrentCulture), " KB");
  model.UploadStatusMessage = string.Format(CultureInfo.CurrentCulture,
   "File uploaded successfully. {0} took {1} seconds to upload",
   fileSizeMessage, duration.TotalSeconds);
}
catch (StorageException e)
{
  model.UploadStatusMessage = "Failed to Upload file. Exception - " + e.Message;
  errorInOperation = true;
}
finally
{
  Session.Remove(model.FileKey);
}
return Json(new
{
  error = errorInOperation,
  isLastBlock = model.IsUploadCompleted,
  message = model.UploadStatusMessage,
  index = model.FileIndex
});
}

The rest of the server side code is same as the Single File upload implementation we saw in our previous article.

Conclusion

We were able to implement multiple file upload in chunks to Azure Blob Storage. The client side implementation has a gotcha as in it uses recursion which can slow things down when uploading too many files at the same time. We can mitigate that by restricting the number of uploads possible at one time.

Download the entire source code of this article (Github)

This article has been editorially reviewed by Suprotim Agarwal.

C# and .NET have been around for a very long time, but their constant growth means there’s always more to learn.

We at DotNetCurry are very excited to announce The Absolutely Awesome Book on C# and .NET. This is a 500 pages concise technical eBook available in PDF, ePub (iPad), and Mobi (Kindle).

Organized around concepts, this Book aims to provide a concise, yet solid foundation in C# and .NET, covering C# 6.0, C# 7.0 and .NET Core, with chapters on the latest .NET Core 3.0, .NET Standard and C# 8.0 (final release) too. Use these concepts to deepen your existing knowledge of C# and .NET, to have a solid grasp of the latest in C# and .NET OR to crack your next .NET Interview.

Click here to Explore the Table of Contents or Download Sample Chapters!

Author

Suprotim Agarwal, MCSD, MCAD, MCDBA, MCSE, is the founder of DotNetCurry, DNC Magazine for Developers, SQLServerCurry and DevCurry. He has also authored a couple of books 51 Recipes using jQuery with ASP.NET Controls and The Absolutely Awesome jQuery CookBook.

Suprotim has received the prestigious Microsoft MVP award for Sixteen consecutive years. In a professional capacity, he is the CEO of A2Z Knowledge Visuals Pvt Ltd, a digital group that offers Digital Marketing and Branding services to businesses, both in a start-up and enterprise environment.

Get in touch with him on Twitter @suprotimagarwal or at LinkedIn