Perhaps you’ve been considering Windows Azure Media Services. Then, looking back at your asset library, or your weekly volume, and you thought:
How am I going to get 5 Terabytes of media up to the cloud?
Good question, you’re not the first to ask.
You have three options:
- Threading the upload of one asset at a time. HTTP bandwidth limited.
- Using our Bulk Ingest .NET Library. HTTP bandwidth limited.
- Using a fast UDP upload client.
I will cover all three of these methods in this and future blog posts, but for now, let’s look at the problem of moving 5TB.
We have been working very hard with Aspera to integrate their fasp(tm) technology in to Azure. While Media Services is not the only driver for massive or rapid transfers, our customers have a lot to gain and we’re proud to welcome Aspera to the ecosystem.
Aspera upload is an opt-in component on the Azure Marketplace. The rest of this post is dedicated to walking through the installation of the tools, creating accounts, and connecting to your Azure Storage account. In my next post, I will look at how to create a media asset using Aspera for upload.
What do I need?
First some background information, start by reading Aspera’s FAQ:
1. You’ll need an Aspera transfer client.
To get your files from your disk to your storage account using Aspera, you’ll need an Aspera Transfer Client. For simplicity, in this blog post, we will look at the installation and configuration of the desktop client. You will probably want to look at the SDK or server solutions to see if these are a better fit for your automated work flow needs. Of course, you can use any of these Aspera clients for your other data-transfer needs: Nothing here is media-specific.
On the Aspera desktop-client page, click the Evaluation Request link and fill in contact-information table. Aspera will contact you with download instructions. If you are planning on working out an enterprise agreement with Aspera, now would be the time to ask for a Promotion Code.
Install the client:
Read the information about local or domain accounts, and click through to the end of the installer, it will launch the desktop client. Enter your license information, and you’ll get the main user interface.
As you can see, there is not much in the way of server connections, let’s address that next.
2. You’ll need to enable Aspera On Demand for Windows Azure
Next, let’s look at the Windows Azure Marketplace, in their words:
The Windows Azure Marketplace helps connect companies seeking innovative cloud based solutions with partners who have developed solutions that are ready to use.
That’s exactly what you’re looking for: a solution for the fast, secure, transfer of lots of data into your Azure Storage Account. To find Aspera, you can simply enter them in the search bar and choose “Aspera On Demand for Azure“. Read those terms and conditions and follow the link to the Aspera website for additional pricing and details.
Go ahead and sign-in in the top-right with that Microsoft Account you associated to your Windows Azure account, as explained here. If this is your first time logging in to the Windows Azure Marketplace, you’ll be asked some additional information and to agree to the terms and conditions regarding the 3rd party vendors, such as Aspera.
Once you’re done, use the search to get back to the Aspera page. Click ‘Buy’ on the most appropriate offer.
If you’ve contacted the nice folks at Aspera and have a promo-code, now’s the time to enter it. Enter your payment information, agree to the terms and hit ‘Sign up’.
Ok, all set, you’re ready to use it. Or almost. .
Click that ‘Use it!’ link to go to the credentials manager on AsperaOnDemand.com site.
You need to add credentials that you will be able to use to connect to the Windows Azure Aspera servers. Go ahead and click ‘New User Credential’.
NOTE: If you are adding this feature during the month of October 2012, you are still in the beta trial, good pricing, but also only available in “West US”. This means that you need to select West US in this dialog box, and have a Media Services account linked to an Azure Storage account in West US. See here about creating new accounts.
Enter the name you want to see in the list, the region in which your storage account is located, and copy the Host, User and Password information somewhere for you to use in your Aspera Client.
NOTE: As of this writing, the correct Host is: “west-us.azure.asperaondemand.com”
Click ‘Save’ and you are back in the user credentials list. If you need those credentials again, there is an icon in the list which you can click to see them again.
So at this point you should have:
- An Aspera Client to work with.
- An Aspera On Demand for Windows Azure account and credentials.
3. Use the Aspera Client to access your Azure Storage account
Open the Aspera Desktop Client, it still looks like step 1, but now we’re ready to add a server connection.
Click the Connections icon, click the ‘+’ to add a new server, and select Windows Azure as the Storage Type:
We’re going to need to fill in some blanks here.
- The top half is from your Aspera Credentials Manager Host, User and Password.
- The bottom is from your Azure Storage Account Name and Key.
We already covered how to get the Aspera credentials earlier in this blog post, so let’s review where we get this from our storage account.
Choose Storage icon on the right, choose the storage account you want to upload to, click ‘Manage Keys’ icon at the bottom and use your Storage Account Name and your Primary Access Key to fill in the connection information:
Click the ‘Advanced’ button and change the SSH Port (TCP) to 33001. Click OK and use the ‘Test Connection’ button to verify your settings.
Click OK to return to the transfer screen, select the server from the list and click Connect.
You can now copy files from your local file system up into Azure Storage. It is not a good practice to copy files to the root folder of your storage account. Create an ‘uploads’ folder and move some files to and from to test your connectivity.
To check out the full capability, I clicked ‘Preferences’ and brought the ‘Initial Target Upload Rate’ to 600Mbps, which is higher than I expect my network to be able to provide.
Then I started an upload of a 522Mb MP4 of Sintel and clicked over to the details view:
At 291Mbps average and sub 20 seconds I’m happy with the performance. Try your favorite Azure Storage upload tool for comparison. Mine never hit above 12 Mbps, your results will vary.
My next post will discuss how all this applies to creating “media assets” in Windows Azure Media Services.