Faster blob copying in Azure Blob Storage

I generally try to avoid copying blobs in Azure and just reference a single copy, but there are some situations where you do need to make a copy. The technique you choose can make a big difference to how long that copy takes.

For example, the "naive" approach of downloading the source and re-uploading it to the destination is conceptually simple but likely to be very slow. On my laptop, a 10GB copy using this technique took about one hour, but that same copy can be done server side in minutes, and if you use a parallel block copy approach (similar to that used by AzCopy) you can copy a file of this size in seconds.

In this post, I'll explain three approaches to blob copying and share some timings.

Download and re-upload (Avoid!)

The "naive" approach is to open the source as a stream and then upload it straight back up:

await using var src = await source.OpenReadAsync();
await using var dst = await target.OpenWriteAsync(overwrite: true);
await src.CopyToAsync(dst);

The problems here are first that the speed is bound by your local connection (about 25 Mbps in my tests), and you're paying for egress bandwidth.

The only situation where this approach makes sense is if for some reason you need to transform the data during the copy (in which case it's not really a copy anyway!)

Server-side copy (StartCopyFromUriAsync)

There's a much better approach offered by the Azure Blob storage SDK, where you generate a SAS URI for the source (typically a short-lived user-delegation SAS) and it performs the copy for you server-side. This operation is asynchronous and you are expected to poll until it completes.

var copy = await target.StartCopyFromUriAsync(sourceSasUri);
await copy.WaitForCompletionAsync();   // poll until Azure finishes

There are several benefits here - you're not streaming any data through your machine, and you're not paying egress cost. It's also robust - the copy continues even if your process goes down after initiating the copy.

This approach was my preferred technique for all blob copies until I noticed that it doesn't perform nearly as fast as AzCopy when copying across storage accounts.

Parallel block-from-URL copy

If you really want the fastest cross-storage account copying speed, the technique I've used is to mimic how AzCopy behaves and split the source blob into ranges and use a technique that allows you to stage blocks from a URL. This means that you can instruct Azure to copy many blocks in parallel. Once all the blocks are staged, you then commit the list of blocks.

You will need to decide what block size to use. A block blob can have up to 50,000 blocks, so if you're copying vast files you need to use sufficiently large block sizes (say 100MiB if you want to upload multi-TiB files). In the example below I also show how to limit the number of blocks that are staged at a time.

const long blockSize = 100 * 1024 * 1024; // 100 MiB

var length = (await source.GetPropertiesAsync()).Value.ContentLength;

// pre-compute an ordered (blockId, range) list — block IDs must be equal-length strings
var blocks = new List<(string Id, HttpRange Range)>();
for (long offset = 0; offset < length; offset += blockSize)
{
    var blockId = Convert.ToBase64String(
        Encoding.UTF8.GetBytes($"block-{blocks.Count:D6}"));
    blocks.Add((blockId, new HttpRange(offset, Math.Min(blockSize, length - offset))));
}

// stage up to 8 blocks at a time
await Parallel.ForEachAsync(blocks,
    new ParallelOptions { MaxDegreeOfParallelism = 8 },
    async (block, ct) =>
        await target.StageBlockFromUriAsync(sourceSasUri, block.Id,
            new StageBlockFromUriOptions { SourceRange = block.Range },
            cancellationToken: ct));

// commit in order (the list is already ordered)
await target.CommitBlockListAsync(blocks.Select(b => b.Id));

With this approach, again the bytes are never downloaded to your machine, and this gives a great speed-up. Of course, unlike with StartCopyFromUriAsync you'll need to make sure that you handle resilience - if you're interrupted half-way through, you'll need to finish staging blocks and committing the block list once they're all done.

Copying within the same storage account

An important caveat is that if the source and destination blobs are in the same storage account, you should just use the server-side copy technique with StartCopyFromUriAsync. It's essentially instantaneous regardless of blob size. For example, in my tests, a 10 GB same-account server-side copy completed in under a second, compared with ~17 s for the parallel copy technique (which still has to transfer the entire file as blocks).

Example timings

I tried each of these three techniques for files of various sizes (100MB, 1GB, 10GB), and between storage accounts in the same and different regions, as well as within a storage account.

Scenario	Size	Server-side	Parallel	Naive
Same region (cross-account)	100 MB	31.5 s	2.7 s	35.9 s
Same region (cross-account)	1 GB	65.2 s	2.4 s	5m 56s
Same region (cross-account)	10 GB	4m 04s	9.2 s	72m 02s
Cross region (cross-account)	100 MB	20.9 s	2.4 s	31.0 s
Cross region (cross-account)	1 GB	92.1 s	2.9 s	5m 00s
Cross region (cross-account)	10 GB	4m 39s	21.2 s	53m 33s
Same account	100 MB	0.7 s	1.5 s	32.6 s
Same account	1 GB	0.4 s	3.0 s	4m 41s
Same account	10 GB	0.4 s	16.8 s	54m 55s

As you can see, the parallel copy technique is by far the fastest in all situations except same-account copies.

Important caveats: I only did one timing for each scenario, and of course the naive copy is more a measure of how fast my local internet connection was working than anything to do with how fast Azure blob storage works. I also did the parallel copy without constraining the number of blocks in parallel at all (which could have been about 100 for the largest file).

Is it an exact copy?

I validated that the parallel copy technique resulted in the same file hash, so yes, this technique does faithfully replicate the original file contents. But of course there is more to a blob than just its contents.

Azure blobs can also have metadata, tags, and properties. Each of the three techniques will allow you to also copy these across, but you must opt in (and ensure that any readable SAS URI for the source file has permissions to access this metadata). See my earlier tutorial here for guidance on copying metadata.

Azure blobs are also stored in an access tier (e.g. hot, cold, cool, archive). Unless you explicitly specify a target tier, you'll just use the account default. And of course if your source blob is in archive, it will need to be rehydrated first, which could be a very slow process.

Block blobs are made up of a series of blocks. With the parallel technique, we ignore the block structure of the source file, and recreate a new blob with (typically fewer) blocks than the source blob. This rarely matters but is worth being aware of.

Summary

If you need to copy blobs between storage accounts, the fastest technique I'm aware of is the parallel block-from-URL copy. This keeps the data off your machine and is by far the quickest for anything non-trivial. The key exception is whenever you copy within a single storage account, where you should just use the plain server-side copy as it's effectively instantaneous. And avoid the download and re-upload approach to blob copying.