A simple scenario: Upload, Encode and Package, Stream

What’s going on when I use the Window Azure Portal to encode a job?

The Windows Azure Portal is built on top of the various REST APIs of the underlying Azure components. The presentation layer is HTML5, and uses a mix of REST and the .Net SDKs on the server-side — you could build your own portal to manage your media workflow.
In the next few weeks, under the Media Services tab, we will update the pages to include some simple code snippets that walk you through this simple scenario:

  • Create an Asset and Upload a file
  • Encode to Smooth and Package to HLS
  • Stream to both Smooth and HLS

Here is what is going on in those snippets:

Click through these slowly and try to match the various arrows to the lines of code below.

Main Program:

// Create .Net console app
// Set project properties to use the full .Net Framework (not Client Profile)
// With NuGet Package Manager, install windowsazure.mediaservices
// add: using Microsoft.WindowsAzure.MediaServices.Client;

var context = new CloudMediaContext("your_media_account", "your_media_account_key");

//Slide 1:
string inputAssetId = CreateAssetAndUploadFile(context);

//Slide 2:
IJob job = EncodeAndPackage(context, inputAssetId);

var smoothAsset = job.OutputMediaAssets.FirstOrDefault();
var hlsAsset = job.OutputMediaAssets.LastOrDefault();

//Slide 3:
string smoothStreamingUrl = GetStreamingUrl(context, smoothAsset.Id);
string hlsStreamingUrl = GetStreamingUrl(context, hlsAsset.Id);

Console.WriteLine("\nSmooth Url: \n" + smoothStreamingUrl); Console.WriteLine("\nHLS Url: \n" + hlsStreamingUrl); Console.ReadKey();
//

First slide:

private static string CreateAssetAndUploadFile(CloudMediaContext context) {

var inputFilePath = @"C:\demo\bing_social_search.mp4";

var assetName = Path.GetFileNameWithoutExtension(inputFilePath);

var inputAsset = context.Assets.Create(assetName, AssetCreationOptions.None);

var assetFile = inputAsset.AssetFiles.Create(Path.GetFileName(inputFilePath));

assetFile.UploadProgressChanged += new EventHandler<UploadProgressChangedEventArgs>(assetFile_UploadProgressChanged);
assetFile.Upload(inputFilePath);

return inputAsset.Id;
}

//Monitor progress:
static void assetFile_UploadProgressChanged(object sender, UploadProgressChangedEventArgs e) {
Console.WriteLine(string.Format("{0}   Progress: {1:0}   Time: {2}",
((IAssetFile)sender).Name, e.Progress, DateTime.UtcNow.ToString(@"yyyy_M_d__hh_mm_ss")));
}
//

Second slide:

private static IJob EncodeAndPackage(CloudMediaContext context, string inputAssetId) {

var inputAsset = context.Assets.Where(a => a.Id == inputAssetId).FirstOrDefault();
if (inputAsset == null)
throw new ArgumentException("Could not find assetId: " + inputAssetId);

var encodingPreset = "H264 Smooth Streaming SD 16x9"; // <a href="http://msdn.microsoft.com/en-us/library/windowsazure/jj129582.aspx#H264Encoding">http://msdn.microsoft.com/en-us/library/windowsazure/jj129582.aspx#H264Encoding</a>

IJob job = context.Jobs.Create("Encoding " + inputAsset.Name + " to " + encodingPreset + " and Packaging to HLS");

IMediaProcessor latestWameMediaProcessor = (from p in context.MediaProcessors where p.Name == "Windows Azure Media Encoder" select p).ToList().OrderBy(wame => new Version(wame.Version)).LastOrDefault();

ITask encodeTask = job.Tasks.AddNew("Encoding", latestWameMediaProcessor, encodingPreset, TaskOptions.None);
encodeTask.InputAssets.Add(inputAsset);
encodeTask.OutputAssets.AddNew(inputAsset.Name + " as " + encodingPreset, AssetCreationOptions.None);

var packagingToSmoothConfig = @"<taskDefinition xmlns=""<a href="http://schemas.microsoft.com/iis/media/v4/TM/TaskDefinition#&quot;&quot;><name>Smooth">http://schemas.microsoft.com/iis/media/v4/TM/TaskDefinition#""><name>Smooth</a> Streams to Apple HTTP Live Streams</name><description xml:lang=""en""/><inputDirectory/><outputFolder/><properties namespace=""<a href="http://schemas.microsoft.com/iis/media/AppleHTTP">http://schemas.microsoft.com/iis/media/AppleHTTP</a>#"" prefix=""hls""><property name=""maxbitrate"" value=""10000000"" /><property name=""segment"" value=""10"" /><property name=""encrypt"" value=""false"" /><property name=""pid"" value="""" /><property name=""codecs"" value=""false"" /><property name=""backwardcompatible"" value=""false"" /><property name=""allowcaching"" value=""true"" /><property name=""passphrase"" value="""" /><property name=""key"" value="""" /><property name=""keyuri"" value="""" /><property name=""overwrite"" value=""true"" /></properties><taskCode><type>Microsoft.Web.Media.TransformManager.SmoothToHLS.SmoothToHLSTask, Microsoft.Web.Media.TransformManager.SmoothToHLS, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35</type></taskCode></taskDefinition>";

IMediaProcessor latestPackagerMediaProcessor = (from p in context.MediaProcessors where p.Name == "Windows Azure Media Packager" select p).ToList().OrderBy(wame => new Version(wame.Version)).LastOrDefault();

ITask packagingTask = job.Tasks.AddNew("Packaging to HLS", latestPackagerMediaProcessor, packagingToSmoothConfig, TaskOptions.None);
packagingTask.InputAssets.Add(encodeTask.OutputAssets[0]);
packagingTask.OutputAssets.AddNew(inputAsset.Name + " encoded and packaged to HLS", AssetCreationOptions.None);

job.StateChanged += new EventHandler<JobStateChangedEventArgs>(JobStateChanged);
job.Submit();
job.GetExecutionProgressTask(CancellationToken.None).Wait();

return job;
}

static void JobStateChanged(object sender, JobStateChangedEventArgs e) {
Console.WriteLine(string.Format("{0}\n  State: {1}\n  Time: {2}\n\n",
((IJob)sender).Name, e.CurrentState, DateTime.UtcNow.ToString(@"yyyy_M_d__hh_mm_ss")));
}
//

Third slide:

(provisioning of the origins is done through the management portal scale page)

private static string GetStreamingUrl(CloudMediaContext context, string outputAssetId) {
var daysForWhichStreamingUrlIsActive = 365;

var outputAsset = context.Assets.Where(a => a.Id == outputAssetId).FirstOrDefault();

var accessPolicy = context.AccessPolicies.Create(outputAsset.Name,                                                     TimeSpan.FromDays(daysForWhichStreamingUrlIsActive), AccessPermissions.Read | AccessPermissions.List);

var assetFiles = outputAsset.AssetFiles.ToList();

var assetFile = assetFiles.Where(f => f.Name.ToLower().EndsWith("m3u8-aapl.ism")).FirstOrDefault();
if (assetFile != null) {
var locator = context.Locators.CreateLocator(LocatorType.OnDemandOrigin, outputAsset, accessPolicy);

Uri hlsUri = new Uri(locator.Path + assetFile.Name + "/manifest(format=m3u8-aapl)");
return hlsUri.ToString();
}

assetFile = assetFiles.Where(f => f.Name.ToLower().EndsWith(".ism")).FirstOrDefault();
if (assetFile != null) {
var locator = context.Locators.CreateLocator(LocatorType.OnDemandOrigin, outputAsset, accessPolicy);
Uri smoothUri = new Uri(locator.Path + assetFile.Name + "/manifest");
return smoothUri.ToString();
}

assetFile = assetFiles.Where(f => f.Name.ToLower().EndsWith(".mp4")).FirstOrDefault();
if (assetFile != null) {
var locator = context.Locators.CreateLocator(LocatorType.Sas, outputAsset, accessPolicy);
var mp4Uri = new UriBuilder(locator.Path);
mp4Uri.Path += "/" + assetFile.Name;
return mp4Uri.ToString();
}
return string.Empty;
}
//

Now you put that smooth URL into http://smf.cloudapp.net/healthmonitor or the HLS Url into your iOS device, and you’re ready to go.

Posted in Uncategorized | Leave a comment

Windows Azure Media Services is now live!

As of this morning, Windows Azure Media Services is now live and open for business.

Launched seven months ago in Preview, it has matured and changed based on the great feedback we recieved in the formus, and direct feedback from our customers:  Thank you!

We are actively investing in the platform to make it an ecosystem of offerings, while improving the core components.

You can read more on:

 

 

 

 

Posted in Uncategorized | Leave a comment

Dynamic Packaging and Encoding and Streaming Reserved Units

Today, we got this question on our forums:

Does anyone have any information on the new ‘on the fly converting’ for media services?

I am trying to allow multiple users (hundreds) the ability to upload video files at the same time.  Obviously, currently the queuing process takes too long.  Would like to know if this new way might help.

Thanks.

Things tend to get pushed down quickly in the forums, and since I own this feature, I thought I would cover the answer in a blog post.  But first things first, read this post about architecting a system for user-contributed content.

Yes, that feature is shipping in the coming days, we refer to it as dynamic packaging, you could also call it just-in-time packaging, dynamic muxing, etc.

Specifically, it will offer the ability to:

Transmux from a single source format into two different streaming formats.

What it will NOT do:

Take a single-bitrate source file and produce multi-bitrate streams.  That requires re-encoding is very CPU intensive.  See this post for the difference between muxing and encoding.

Supported input formats:

  • MP4
  • Smooth

Supported output formats:

  • Smooth
  • HLS v4

Supported scenarios:

  • Primarily AAC and H.264 codecs (you can use VC-1, but it will not remux to HLS).
  • A single MP4 to single-bitrate smooth and single-bitrate HLS.
  • Many closed GOP, GOP aligned MP4′s to multi-bitrate smooth or multi-bitrate HLS.
  • Multi-bitrate Smooth to HLS, and of course, Smooth.

Unsupported scenarios at this time:

  • Encrypted content as source files, neither Storage Encryption nor Common Encryption.

Pictures always help:


Cost and setup:

At this time, dynamic packaging will only be offered if you reserve origin capacity in the management portal.  That is, you need to enable and buy an origin reserved unit (~200$/mo charged by 24hr period increments, this is a blog, pricing changes, check the management portal for details).

If you are not using a reserved origin to stream, then you are sharing a small pool of origin servers amongst all the media services users of a datacenter.  We call this the preview-pool, and are not updating it at this time to support dynamic packaging.  There is no Service Level Agreements for the preview pool, if someone’s video goes viral, you’ll be competing with them for bandwidth.  If you’re serious about streaming, get a reserved unit, that’s what they are for.

To reserve on-demand streaming capacity:

Go to the scale page in the management portal and move the slider to 1.  If you are provisioned with a reserved unit, you’re all set.

It takes a few minutes to spin up an origin reserved unit for you, if it’s been a few hours and you don’t see confirmation in the ‘scale’ page of the management UI, then there was not any capacity within our reservation system in the datacenter.  This triggers a request for increased capacity internally, but these need to be provisioned, which takes a while.  We all like to think that ‘the cloud’ is infinity elastic; yes, in theory, but in practice racks of servers are powered down when unused and there are an army of guys around the world with box-cutters and screw drivers racking servers.  There are well over 10,000 subscribers to media services, if everyone asks for 5 reserved units, we’ll be keeping those guys busy.

Back to the original question:

The original poster was looking for the quickest path to putting user contributed content back out there.  That works if the content is ‘just right’.  But it tends to go sideways quickly: all your users are capturing video in different ways, using different codecs and container formats (mov != mp4 except in special cases).  By encoding each of these, even to a very simple (quicker) single-bitrate encoding profile, you create uniformity in the content that you are streaming.  Without uniformity, you will get random failures that will be hard to trace: “this video plays on iOS, but not Android”  “my videos wont stream at all”; and you’ll spend all sorts of hours tracing this down.  Trust me, I do this for customers all week long — we are taking on this burden so you don’t have to.  Skipping the encode step seems like the quickest path, but it will bring you pain.  Pay for encoding reserved units (99$/mo) and you will be able to manage your queue, and mostly, you won’t sit in line with everyone else in the datacenter.

Dynamic Packaging Server Manifests

When you encode to multi-bitrate mp4 with the Windows Azure Media Encoder, the system will produce a server manifest for you.  If you are familiar with the smooth streaming format, this is the .ism file that essentially says: this file is a video track, so is this file, and this file, and this one is an audio track.  When Dynamically Packaging from MP4, you need to explicitly tell the server that these input files are MP4s.  This is done with some metadata in the <head/> section:

 <?xml version="1.0" encoding="utf-8"?>
  <smil xmlns="http://www.w3.org/2001/SMIL20/Language">
  <head>
   <meta name="formats" content="mp4" />
  </head>
  <body>
   <switch>
    <video src="yourFile.mp4" />
    <audio src="yourFile.mp4" />
   </switch>
  </body>
 </smil>

In the case of several MP4 files, each muxed with the same audio track, it would look like this:

<?xml version="1.0" encoding="utf-8"?>
 <smil xmlns="http://www.w3.org/2001/SMIL20/Language">
 <head>
 <meta name="formats" content="mp4" />
 </head>
 <body>
 <switch>
 <video src="yourFile_BR1.mp4" />
 <video src="yourFile_BR2.mp4" />
 <video src="yourFile_BR3.mp4" />
 <video src="yourFile_BR4.mp4" />
 <audio src="yourFile_BR1.mp4" />
 </switch>
 </body>
 </smil>

But is it going to work?

The original poster did not want to wait in a queue before he started streaming.  The trouble is, he will only find out if he has problems once he’s actually streaming.  Unless he sets up a quality check of some sort, he is only going to find out because his users tell him; or may only find out when they abandon his site altogether  — which, unfortunately, is more typical of internet users: “if it doesn’t work, I have better things to do with my time”.

To mitigate this risk, we have set up an MP4 Preprocessor task within the existing Windows Azure Media Packager.  It analyses the input files and checks that they are streamable to Smooth and HLS.  You can make the task fail if the input cannot be streamed to your desired format.  Id rather not maintain code snippets for it here in my blog, but it’s quite easy to use if you’ve built any sort of encoding/packaging workflow against WAMS.  It is documented on MSDN here in the Dynamic Packaging section.
If you’re not going to encode with WAMS, you should at least check your files.  The actual runtime for checking an asset is less than a minute for an asset as large as a few Gigs (set-up time to run the tasks varies proportionally to asset size);  you do need a slot in the queue, however.  The idea is to fail early, before streams are out there on your web property.

So how do I stream?

Just create a locator and append your .ism file and manifest type.
Create the locator:

var accessPolicy = _context.AccessPolicies.Create(assetName, TimeSpan.FromDays(365), AccessPermissions.Read | AccessPermissions.List);
var locator = _context.Locators.CreateLocator(LocatorType.OnDemandOrigin, asset, accessPolicy);

Then for smooth:

UriBuilder ub = newUriBuilder(locator.Path);
ub.Path += “/yourFile.ism/manifest”;
Uri smoothUri = ub.Uri;

or for HLS:

UriBuilder ub = newUriBuilder(locator.Path);
ub.Path += “/yourFile.ism/manifest(format=m3u8-aapl)”;
Uri hlsUri = ub.Uri;

That’s it, it just works.  If you’re interested in the techy details, read on.

At runtime, the server will get the manifest request, it will peek into the .ism file and find the files your have listed, it will then open each and look up the video frame information, it will look for synchronization points across tracks and build the manifest response accordingly.  When it gets fragment requests, it will go back into the video file for that quality level and find the required video frames, it will then build the response from the raw video frames into smooth or HLS, as per the request.

Posted in Windows Azure Media Services | Tagged | Leave a comment

Taking storage and streaming for a spin with Transform Manager

So perhaps you’re thinking:

I’m not ready to move my encoding workflow to the cloud, but I’m interested in exploring storage and streaming alternatives.

.
Ok, I understand where you’re coming from.  I’ve worked with some of the largest content owners and getting it just right is not trivial.  Or perhaps your studio contracts do not allow you to egress media assets unless you’ve applied the stipulated digital rights management encryption (PlayReady, I hope!).

Many large content owners have much more flexibility in their choice of storage, origin service and CDN providers.  In this blog, I’ll touch on leveraging storage and origin services in Windows Azure Media Services without significantly altering your asset creation workflow, or writing any lines of code.

Prior to focusing on the cloud, our Media Services team had all been heads-down on various on-premise products and client frameworks.  I had been working on Transform Manager.  It’s an extensible media workflow tool which also offers a few transmuxing capabilities:  Mp4 to Smooth Streaming, clear Smooth Streaming to PlayReady-protected Smooth Streaming, and (clear or PlayReady) Smooth Streaming to HLS.  It also integrates nicely with Expression Encoder and our HPC Cluster technologies.  While I can’t mention who uses it, I estimate it lights up a significant proportion of on-premise workflows worldwide.  So, perhaps you’re one of the tens of thousands who have downloaded and are using Transform Manager.  Great, but that doesn’t help your storage and streaming needs.

Move storage and streaming to the cloud!

Since TM is an extensible framework, anyone can build additional workflow tasks.  Given my knowledge of the product, I decided to write one myself.

You can find it on codeplex at:  http://createassettask.codeplex.com

What’s great about it?

  • TM is a watch-folder based tool: drop the files in a folder, and the workflow is kicked off.  No coding, no scripting.
  • Just enter your Media Services credentials using the Transform Manager user interface.

What does it do?

  • This task will create and upload a media asset to Windows Azure Media Services.
  • It can also create a streaming URL for that asset at the same time.
  • It outputs a json object with the data you’ll need to use or stream the asset

For existing TM users: you can chain this task at the end of your usual workflow.  Just send a copy of the asset up to the cloud, get a streaming URL and give it to your QA team.  It’s that easy.

For CMS users: you can set your CMS’s media id into the “IAsset.AlternateId” field.  This will allow you to use your CMS id to make queries against your Media Services account to find the asset again.  Otherwise, the task can use the TM job ID as the AlternateId, allowing you to correlate TM jobs to Media Services Assets.

The output of the task is a json object with the

  • Asset.Id
  • Asset.AlternateId
  • PrimaryFile.Name
  • PrimaryFileUri (if you asked for one)

So you can pick up the json object with a web-page and start playing it, or read it with your CMS to pipe this information back into your system.

You do have to build it yourself in visual studio, I haven’t released it as a binary.  All the build instructions are in the main CreateAssetTask.cs file at the top.  The parameters are explained in the CreateAssetTask.xml, and will be visible in the Transform Manager user interface.

It would be easy to modify this task to run encode jobs in the cloud after uploading, but that would be a whole other blog post.

If you just need some sample code to build your own TM task, or to see how to upload an asset or get an origin url, go ahead and use it for that too.

Posted in Windows Azure Media Services | Tagged | Leave a comment

So you want to build your own YouTube?

Here is a reference architecture for building out a user-contributed video gallery using Windows Azure Media Services.

Not a lot of time to blog this week, perhaps I’ll have time in the future to really work this scenario over with full examples including the web and worker roles, azure tables and such.

So there’s the presentation and below is some code to provide that basic interaction with Media Services.

Note, when using a browser client to PUT directly to storage, you will need a client framework such as Flash or Silverlight and a crossdomain.xml / clientaccesspolicy.xml in the $root folder of your storage account to avoid CORS issues with the simpler XMLHttpRequest.send().

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.IO;
using System.Configuration;

/// Microsoft.WindowsAzure.MediaServices.Client
///
/// Reference:
///  C:\Program Files (x86)\Microsoft WCF Data Services\5.0\bin\.NETFramework\Microsoft.Data.Edm.dll
///  C:\Program Files (x86)\Microsoft WCF Data Services\5.0\bin\.NETFramework\Microsoft.Data.OData.dll
///  C:\Program Files (x86)\Microsoft WCF Data Services\5.0\bin\.NETFramework\Microsoft.Data.Services.dll
///  C:\Program Files (x86)\Microsoft WCF Data Services\5.0\bin\.NETFramework\Microsoft.Data.Services.Client.dll
///  C:\Program Files (x86)\Microsoft WCF Data Services\5.0\bin\.NETFramework\System.Spatial.dll
/// Download from: http://www.microsoft.com/en-us/download/details.aspx?id=29306
///
///  C:\Program Files (x86)\Microsoft SDKs\Windows Azure Media Services\Services\v1.0\Microsoft.WindowsAzure.MediaServices.Client.dll
/// Download from: http://www.microsoft.com/en-us/download/details.aspx?id=30153
///
///  C:\Program Files\Windows Azure SDK\v1.6\bin\Microsoft.WindowsAzure.StorageClient.dll
/// Download from WebPI: http://go.microsoft.com/fwlink/?linkid=255386
///                      On the Products tab select All, then find and install the
///                      Windows Azure SDK 1.6 for Visual Studio 2010 (November 2011).
///
/// General Support:
///  Windows Azure Media Services Forums: http://social.msdn.microsoft.com/Forums/da-dk/MediaServices/threads
///
using Microsoft.WindowsAzure.MediaServices.Client;
using Microsoft.WindowsAzure.StorageClient;
using Microsoft.WindowsAzure;

namespace WindowsAzureMediaServices.Helper
{
    public class MediaHelper
    {
        private static string thumbnailToken = "_thumbnail";
        private static string smoothToken = "_smooth";
        private static string windowsAzureMediaEncoderName = "Windows Azure Media Encoder";

        ///<summary>
        /// Call this once per thread to get the context for any further calls. Keep the object and use it for all calls from that thread.
        /// </summary>
        /// context object to be passed to other helper functions for this thread.
        public static CloudMediaContext GetContext(HelperConfigInfo info)
        {
            if (!info.IsInitialized)
                throw new ArgumentNullException("info");

            CloudMediaContext context = null;

            if (!String.IsNullOrEmpty(info.scope))
            {
                context = new CloudMediaContext(new Uri(info.apiServer), info.wamsAccountName, info.wamsAccountKey, info.scope, info.acsBaseAddress);
            }
            else if (!String.IsNullOrEmpty(info.apiServer))
            {
                context = new CloudMediaContext(new Uri(info.apiServer), info.wamsAccountName, info.wamsAccountKey);
            }
            else
            {
                context = new CloudMediaContext(info.wamsAccountName, info.wamsAccountKey);
            }
            return context;
        }

        ///<summary>
        /// Gets the base uri with SAS token for uploads
        /// </summary>
        ///
        ///
        /// Append the filename to the path using a UriBuilder  http://server/path?SasToken
        public static Uri GetUploadUri(HelperConfigInfo info, Guid idForThisMedia)
        {
            if (info == null)
                throw new ArgumentNullException("info");
            if(!info.IsInitialized)
                throw new ArgumentException("make sure info is initialized");
            if (idForThisMedia == Guid.Empty)
                throw new ArgumentException("Provide a non-empty Guid for idForThisMedia");

            CloudMediaContext context = GetContext(info);
            if (context == null)
                throw new ApplicationException("failed to get context");

            // Check if this asset exists:
            IAsset duplicate = GetAssetByName(context, idForThisMedia.ToString());
            if (duplicate != null)
            {
                throw new ArgumentException("idForThisMedia is being re-used.");
            }

            Uri uploadUri = null;

            //
            // Create an empty asset
            //
            IAsset inputAsset = context.Assets.CreateEmptyAsset(idForThisMedia.ToString(), AssetCreationOptions.None);

            //
            // Get a SAS url:
            //
            IAccessPolicy writePolicy = context.AccessPolicies.Create("Policy For Copying", TimeSpan.FromDays(info.accessPolicyDurationInDays), AccessPermissions.Write | AccessPermissions.List);
            ILocator destinationLocator = context.Locators.CreateSasLocator(inputAsset, writePolicy, DateTime.UtcNow.AddMinutes(info.locatorStartTimeInMinutes));

            uploadUri = new Uri(destinationLocator.Path);

            return uploadUri;
        }

        ///<summary>
        /// Disables all SAS locators on the asset
        /// </summary>
        ///
        ///
        ///
        public static void DisableUploadUri(HelperConfigInfo info, Guid idForThisMedia)
        {
            if (info == null)
                throw new ArgumentNullException("info");
            if (!info.IsInitialized)
                throw new ArgumentException("make sure info is initialized");
            if (idForThisMedia == Guid.Empty)
                throw new ArgumentException("Provide a non-empty Guid for idForThisMedia");

            CloudMediaContext context = GetContext(info);
            if (context == null)
                throw new ApplicationException("failed to get context");

            // Check if this asset exists:
            IAsset asset = GetAssetByName(context, idForThisMedia.ToString());
            if (asset != null)
            {
                foreach (var locator in asset.Locators)
                {
                    if (locator.Type == LocatorType.Sas)
                    {
                        context.Locators.Revoke(locator);
                    }
                }
            }
        }

        public static void RunJob(HelperConfigInfo info, Guid idForThisMedia)
        {
            if (info == null)
                throw new ArgumentNullException("info");
            if (!info.IsInitialized)
                throw new ArgumentException("make sure info is initialized");
            if (idForThisMedia == Guid.Empty)
                throw new ArgumentException("Provide a non-empty Guid for idForThisMedia");

            CloudMediaContext context = GetContext(info);
            if (context == null)
                throw new ApplicationException("failed to get context");

            //
            // Get Input Asset
            //
            IAsset inputAsset = GetAssetByName(context, idForThisMedia.ToString());
            if (inputAsset == null)
                throw new ApplicationException("failed to get asset");

            //
            // Publish it
            //
            //This is required to search for and add files that were added by the StorageClient copy/upload calls.
            inputAsset.Publish(); //May throw if there are no files.

            //
            // Check file-count, should be 1
            //
            inputAsset = GetAssetByName(context, idForThisMedia.ToString());
            if (inputAsset.Files.Count() != 1)
                throw new ApplicationException("Asset should have one file, idForThisMedia:" + idForThisMedia); //This will throw at the thread callback level, catch at app level.

            //
            // Create a job, name it idForThisMedia
            //
            IJob job = context.Jobs.Create(idForThisMedia.ToString());

            //
            // Get the media processor
            //
            IMediaProcessor windowsAzureMediaEncoder = (from a in context.MediaProcessors
                                                        where a.Name == windowsAzureMediaEncoderName
                                                        select a).First();

            //
            // Add the Encode task.
            //
            ITask encodeTask = job.Tasks.AddNew(idForThisMedia.ToString() + smoothToken,
                                                    windowsAzureMediaEncoder,
                                                    info.windowsAzureMediaEncoder_EncodeConfig,
                                                    TaskCreationOptions.None);
            encodeTask.InputMediaAssets.Add(inputAsset);
            encodeTask.OutputMediaAssets.AddNew(idForThisMedia.ToString() + smoothToken, true, AssetCreationOptions.None);

            //
            // Add the Thumbnail task.
            //
            ITask thumbnailTask = job.Tasks.AddNew(idForThisMedia.ToString() + thumbnailToken,
                                                    windowsAzureMediaEncoder,
                                                    info.windowsAzureMediaEncoder_ThumbnailConfig,
                                                    TaskCreationOptions.None);
            thumbnailTask.InputMediaAssets.Add(inputAsset);
            thumbnailTask.OutputMediaAssets.AddNew(idForThisMedia.ToString() + thumbnailToken, true, AssetCreationOptions.None);

            //
            // Add this job to the job queue
            //
            job.Submit();

            //
            // The ouput assets are not actually created util the job is submitted.
            // At that point, they can be refreshed and updated.
            //
            IAsset smoothAsset = job.OutputMediaAssets[0];
            smoothAsset = RefreshAsset(context, smoothAsset.Id);
            smoothAsset.Name = idForThisMedia.ToString() + smoothToken;
            context.Assets.Update(smoothAsset);
            IAsset thumbnailAsset = job.OutputMediaAssets[1];
            thumbnailAsset = RefreshAsset(context, thumbnailAsset.Id);
            thumbnailAsset.Name = idForThisMedia.ToString() + thumbnailToken;
            context.Assets.Update(thumbnailAsset);
        }

        ///<summary>
        /// Creates or returns an existing SAS Url for the thumbnail.jpg. Usable immediately.
        /// </summary>
        ///
        ///
        /// Usable URL or null
        public static Uri GetThumbnailUri(HelperConfigInfo info, Guid idForThisMedia)
        {
            if (info == null)
                throw new ArgumentNullException("info");
            if (!info.IsInitialized)
                throw new ArgumentException("make sure info is initialized");
            if (idForThisMedia == Guid.Empty)
                throw new ArgumentException("Provide a non-empty Guid for idForThisMedia");

            CloudMediaContext context = GetContext(info);
            if (context == null)
                throw new ApplicationException("failed to get context");

            Uri thumbnailUri = null;

            //
            // Get the asset
            //
            IAsset asset = GetAssetByName(context, idForThisMedia + thumbnailToken);
            if (asset == null)
                throw new ApplicationException("Could not find asset using idForThisMedia: " + idForThisMedia);

            //
            // Find the .jpg in the asset
            //
            IFileInfo jpgFile = (from f in asset.Files
                                 where f.Name.EndsWith("2.jpg")
                                 select f).FirstOrDefault();
            if (jpgFile == null)
                throw new ApplicationException("Could not find a .jpg file in the asset.id: " + asset.Id);

            //
            // Look for an existing locator
            //
            ILocator locator = null;
            var locators = from rows in asset.Locators where rows.Type == LocatorType.Sas orderby rows.ExpirationDateTime select rows;
            if (locators != null && locators.Count() > 0)
            {
                //Get the one that expires last:
                locator = locators.LastOrDefault();
            }

            //
            // Check the existing locator
            //
            //TODO: Test this logic
            if (locator != null)
            {
                //Check if it will expire within _locatorRecreationThresholdDays
                if (locator.ExpirationDateTime = maxLocators)
                    {
                        var earliest = locators.FirstOrDefault();
                        if (earliest != null)
                        {
                            context.Locators.Revoke(earliest);
                        }
                    }
                }
            }

            //
            // Create a locator if required.
            //
            if (locator == null)
            {
                var accessPolicyTimeout = TimeSpan.FromDays(info.accessPolicyDurationInDays);
                IAccessPolicy readPolicy = context.AccessPolicies.Create("Read Policy " + idForThisMedia + thumbnailToken, accessPolicyTimeout, AccessPermissions.Read);
                var startTime = DateTime.UtcNow.AddMinutes(info.locatorStartTimeInMinutes);
                locator = context.Locators.CreateSasLocator(asset, readPolicy, startTime);
            }

            //
            // Build Uri
            //
            if (locator != null)
            {
                UriBuilder ub = new UriBuilder(locator.Path);
                ub.Path += "/" + jpgFile.Name;
                thumbnailUri = ub.Uri;
            }

            return thumbnailUri;
        }

        ///<summary>
        /// Creates or returns an existing Origin Url for the smooth asset. Usable after 30 seconds.
        /// </summary>
        ///
        ///
        /// Usable URL or null
        public static Uri GetSmoothStreamingUri(HelperConfigInfo info, Guid idForThisMedia)
        {

            if (info == null)
                throw new ArgumentNullException("info");
            if (!info.IsInitialized)
                throw new ArgumentException("make sure info is initialized");
            if (idForThisMedia == Guid.Empty)
                throw new ArgumentException("Provide a non-empty Guid for idForThisMedia");

            CloudMediaContext context = GetContext(info);
            if (context == null)
                throw new ApplicationException("failed to get context");

            Uri smoothUri = null;

            //
            // Get the asset
            //
            IAsset asset = GetAssetByName(context, idForThisMedia + smoothToken);
            if (asset == null)
                throw new ApplicationException("Could not find asset using idForThisMedia: " + idForThisMedia);

            //
            // Find the .ism in the asset
            //
            IFileInfo ismFile = (from f in asset.Files
                                 where f.Name.EndsWith(".ism")
                                 select f).FirstOrDefault();
            if (ismFile == null)
                throw new ApplicationException("Could not find a .ism file in the asset.id: " + asset.Id);

            //
            // Look for an existing locator
            //
            ILocator locator = null;
            var locators = from rows in asset.Locators where rows.Type == LocatorType.Origin orderby rows.ExpirationDateTime select rows;
            if (locators != null && locators.Count() > 0)
            {
                //Get the one that expires last:
                locator = locators.LastOrDefault();
            }

            //
            // Check the existing locator
            //
            //TODO: Test this logic
            if (locator != null)
            {
                //Check if it will expire within _locatorRecreationThresholdDays
                if (locator.ExpirationDateTime = maxLocators)
                    {
                        var earliest = locators.FirstOrDefault();
                        if (earliest != null)
                        {
                            context.Locators.Revoke(earliest);
                        }
                    }
                }
            }

            //
            // Create a locator if required.
            //
            if (locator == null)
            {
                var accessPolicyTimeout = TimeSpan.FromDays(info.accessPolicyDurationInDays);
                IAccessPolicy readPolicy = context.AccessPolicies.Create("Read Policy " + idForThisMedia + smoothToken, accessPolicyTimeout, AccessPermissions.Read);
                var startTime = DateTime.UtcNow.AddMinutes(info.locatorStartTimeInMinutes);
                locator = context.Locators.CreateWindowsAzureCdnLocator(asset, readPolicy, startTime);
            }

            //
            // Build Uri
            //
            if (locator != null)
            {
                UriBuilder ub = new UriBuilder(locator.Path);
                ub.Path +=  ismFile.Name + "/manifest";
                smoothUri = ub.Uri;
            }

            return smoothUri;
        }

        ///<summary>
        /// Query for a fresh reference to a job during threaded operations
        /// </summary>
        ///
        ///
        /// The job or null
        public static IJob GetJobByName(CloudMediaContext context, string jobName)
        {
            // Use a Linq select query to get an updated reference by Id.
            IJob theJob = (from j in context.Jobs
                           where j.Name == jobName
                           select j).FirstOrDefault();
            return theJob;
        }

        ///<summary>
        /// Deletes the associated assets.
        /// </summary>
        ///
        ///
        /// "Deleted" or "Input=DeleteResult, Smooth=DeleteResult, Thumbnail=DeleteResult"
        public static string DeleteAllRelatedAssets(CloudMediaContext context, Guid idForThisMedia)
        {
            string result = string.Empty;

            //
            // Validate input params:
            //
            if (context == null)
                throw new ArgumentNullException("context");
            if (idForThisMedia == Guid.Empty)
                throw new ArgumentException("Provide a non-empty Guid for idForThisMedia");

            String inputResult = DeleteAssetByName(context, idForThisMedia.ToString());
            String smoothResult = DeleteAssetByName(context, idForThisMedia + smoothToken);
            String thumbnailResult = DeleteAssetByName(context, idForThisMedia + thumbnailToken);

            if (inputResult == "Deleted" &&
                smoothResult == "Deleted" &&
                thumbnailResult == "Deleted")
            {
                //Worked
                result = "Deleted";
            }
            else
            {
                //Something failed, give full results:
                result = "Input=" + inputResult + ", Smooth=" + smoothResult + ", Thumbnail=" + thumbnailResult;
            }

            return result;
        }

        #region Internal Helper functions

        ///<summary>
        /// Deletes the asset and the associated locators.
        /// </summary>
        ///
        ///
        /// Deleted or error.
        internal static string DeleteAssetByName(CloudMediaContext context, string name)
        {
            IAsset asset = GetAssetByName(context, name);

            if (asset == null)
                return "NotFound";

            try
            {
                foreach (ILocator locator in asset.Locators)
                {
                    context.Locators.Revoke(locator);
                }
                int numContentKeys = asset.ContentKeys.Count();
                for (int i = 0; i < numContentKeys; i++)
                {
                    asset.ContentKeys.RemoveAt(i);
                }
                context.Assets.Delete(asset);
            }
            catch (Exception e)
            {
                return "Failed: " + e.Message;
            }

            return "Deleted";
        }

        ///<summary>
        /// Query for a fresh reference to an asset during threaded operations.
        /// </summary>
        ///
        ///
        /// The asset or null.
        internal static IAsset GetAssetByName(CloudMediaContext context, string name)
        {
            // Use a Linq select query to get an updated reference by Id.
            IAsset theAsset = (from a in context.Assets
                               where a.Name == name
                               select a).FirstOrDefault();
            return theAsset;
        }

        ///<summary>
        /// Query for a fresh reference to an asset during threaded operations.
        /// </summary>
        ///
        ///
        /// The asset or null.
        internal static IAsset RefreshAsset(CloudMediaContext context, string assetId)
        {
            // Use a Linq select query to get an updated reference by Id.
            IAsset theAsset = (from a in context.Assets
                              where a.Id == assetId
                              select a).FirstOrDefault();
            return theAsset;
        }

        #endregion

    }
}

Here is a bit of code which uses the above in a command line app.
The command line app consolidates the client, upload web-role, the RunJob worker and the Publishing worker.
There are no cross domain hurdles when using a full .Net client for the PUT to storage — as noted above, you need another strategy in-browser.

Dictionary<Guid, string> files = new Dictionary<Guid, string>();
files.Add(Guid.NewGuid(), @"D:\MS\CONTENT\1.MOV");  
files.Add(Guid.NewGuid(), @"D:\MS\CONTENT\2.MOV");
files.Add(Guid.NewGuid(), @"D:\MS\CONTENT\3.MOV");

foreach (var file in files)
{
    Console.WriteLine(file.ToString()); //Track these guids in your content management system.
}

System.Threading.Tasks.Parallel.ForEach(files, file =>
{
    try
    {

        //Get the upload Url
        Uri uploadUrl = MediaHelper.GetUploadUri(info, file.Key);

        //Use it.  In this test, we just upload a file using a PUT.
        //But you could pass it down to an app so that it can handle the upload.
        if (!DoSomethingWithUploadUrl(info, uploadUrl, file.Value))
            return;

        //Disable the upload Url:
        MediaHelper.DisableUploadUri(info, file.Key);

        //Now we verify the asset and kick off the encode:
        MediaHelper.RunJob(info, file.Key);

        //Wait for the job:
        bool success = false;
        {
            CloudMediaContext threadContext = MediaHelper.GetContext(info);
            success = CheckJobProgress(threadContext, file.Key.ToString());
        }

        if (success)
        {
            Uri jpgUrl = MediaHelper.GetThumbnailUri(info, file.Key);
            Console.WriteLine("Jpg: " + jpgUrl.ToString());
            DoSomethingWithJpgUrl(jpgUrl);

            Uri smoothUrl = MediaHelper.GetSmoothStreamingUri(info, file.Key);
            Console.WriteLine("Smooth: " + smoothUrl.ToString());
            System.Threading.Thread.Sleep(30000); //Wait 30s for the streaming servers to get the locator table updates.
            DoSomethingWithStreamingUrl(smoothUrl);
        }
        else
        {
            Console.WriteLine("Main thread simple test failed.");
        }
    }
    catch (Exception e)
    {
        Console.WriteLine(e.Message);
    }
});

And for completeness, here are the functions used in the above:

        #region TestApp Specific
        
        //Derive from WebClient so that you can override base class settings:
        public class WebUpload : WebClient
        {
            protected override WebRequest GetWebRequest(Uri address)
            {
                WebRequest request = (WebRequest)base.GetWebRequest(address);
                // Perform any customizations on the request.
                request.Timeout = 600000; //Set upload timeout to 10min
                return request;
            }

        }
        private static bool DoSomethingWithUploadUrl(HelperConfigInfo info, Uri uploadUrl, string mediaFile)
        {
            string fileName = Path.GetFileName(mediaFile);

            UriBuilder ub = new UriBuilder(uploadUrl);
            ub.Path += "/" + fileName;

            Uri fullUploadUrl = ub.Uri;

            bool uploadOK = false;
            WebUpload wc = new WebUpload();
            try
            {
                Console.WriteLine(mediaFile + " upload start " + DateTime.Now);
                byte[] responce = wc.UploadFile(fullUploadUrl, "PUT", mediaFile);
                Console.WriteLine(mediaFile + " upload done " + DateTime.Now);
                UTF8Encoding enc = new UTF8Encoding();
                string resp = enc.GetString(responce);
                Console.WriteLine(resp);
                uploadOK = true;
            }
            catch (Exception e)
            {
                Console.WriteLine(e.Message);
            }
            return uploadOK;
        }


        private static void DoSomethingWithJpgUrl(Uri jpgUrl)
        {
            WebClient wc = new WebClient();
            byte[] responceData = null;
            try
            {
                responceData = wc.DownloadData(jpgUrl);
            }
            catch (Exception e)
            {
                Console.WriteLine("Failed to download manifest: " + e.Message);
            }
            if (responceData == null || responceData.Length == 0)
            {
                Console.WriteLine("Failed to download manifest: no data.");
            }
            else
            {
                FileStream fs = new FileStream("c:\\temp\\output\\mainThreadUserMediaGuid.jpg", FileMode.Create);
                BinaryWriter br = new BinaryWriter(fs);
                br.Write(responceData);
                br.Close();
                fs.Close();
            }
        }


        private static void DoSomethingWithStreamingUrl(Uri smoothUrl)
        {
            WebClient wc = new WebClient();
            byte[] responceData = null;
            try
            {
                responceData = wc.DownloadData(smoothUrl);
            }
            catch (Exception e)
            {
                Console.WriteLine("Failed to download manifest: " + e.Message);
            }
            if (responceData == null || responceData.Length == 0)
            {
                Console.WriteLine("Failed to download manifest: no data.");
            }
            else
            {
                FileStream fs = new FileStream("c:\\temp\\output\\mainThreadUserMediaGuid.ismc", FileMode.Create);
                BinaryWriter br = new BinaryWriter(fs);
                br.Write(responceData);
                br.Close();
                fs.Close();
            }
        }
        #endregion


        /// <summary>
        /// Expected polling interval in milliseconds.  Adjust this interval as needed based on estimated job completion times. 
        /// </summary>
        const int _JobProgressInterval = 20000;

        /// <summary>
        /// Check the job progress and wait for completion or failure.  If job does not exist, this will wait in the queued state: for async logic.
        /// </summary>
        /// <param name="context"></param>
        /// <param name="jobName"></param>
        /// <returns></returns>
        public static bool CheckJobProgress(CloudMediaContext context, string jobName)
        {
            // Flag to indicate when job state is finished. 
            bool jobCompleted = false;
            bool success = false;
            JobState state = JobState.Queued;

            double loops = 0;
            while (!jobCompleted)
            {
                // Get state:
                IJob job = MediaHelper.GetJobByName(context, jobName);
                if (job == null)
                {
                    state = JobState.Queued; //Force to Queued
                    Console.WriteLine("Job not found, force to Queued State");
                }
                else
                {
                    state = job.State;
                }

                // Report:
                Console.WriteLine(jobName + " " + state + " elapse time: " + loops * _JobProgressInterval / 1000.0);
                loops++;

                // Check:
                switch (state)
                {
                    case JobState.Finished:
                        jobCompleted = true;
                        success = true;
                        break;
                    case JobState.Queued:
                    case JobState.Scheduled:
                    case JobState.Processing:
                        //Do nothing.
                        break;
                    case JobState.Error:
                        jobCompleted = true;

                        // Dig into the main MediaServices.Client for error handling:
                        if (job != null)
                        {
                            foreach (var task in job.Tasks)
                            {
                                var ed = task.ErrorDetails;
                                if (ed != null)
                                {
                                    foreach (var item in ed)
                                    {
                                        Console.WriteLine(String.Format("Job failed while building the asset. Error code: {0} Message: {1}", item.Code, item.Message));
                                    }
                                }
                            }
                        }
                        break;
                    default:
                        // Not normal to be here!
                        jobCompleted = true;
                        break;
                }

                // Wait for the specified job interval before checking state again.
                if (!jobCompleted)
                    System.Threading.Thread.Sleep(_JobProgressInterval);
            }
            return success;
        }

Posted in Windows Azure Media Services | Tagged | 6 Comments

Creating a simple media asset

First things first, lets get a file into Windows Azure Media Services so that we can leverage all that cloud power.

Today we look at the following steps using the Media Services Client .Net SDK:

  • Getting a CloudMediaContext object.
  • Creating an Asset

That’s it, two easy steps.

But before you learn to walk, there’s a bit of crawling to do.  You need to have three bits of software installed to build against the Media Services .Net SDK:

WCF Data Services 5.0

Reference:
 C:\Program Files (x86)\Microsoft WCF Data Services\5.0\bin\.NETFramework\Microsoft.Data.Edm.dll
 C:\Program Files (x86)\Microsoft WCF Data Services\5.0\bin\.NETFramework\Microsoft.Data.OData.dll
 C:\Program Files (x86)\Microsoft WCF Data Services\5.0\bin\.NETFramework\Microsoft.Data.Services.dll
 C:\Program Files (x86)\Microsoft WCF Data Services\5.0\bin\.NETFramework\Microsoft.Data.Services.Client.dll
 C:\Program Files (x86)\Microsoft WCF Data Services\5.0\bin\.NETFramework\System.Spatial.dll
Download from: http://www.microsoft.com/en-us/download/details.aspx?id=29306

Azure Storage Client

Reference:
 C:\Program Files\Windows Azure SDK\v1.6\bin\Microsoft.WindowsAzure.StorageClient.dll
Download from WebPI: http://go.microsoft.com/fwlink/?linkid=255386  
                     On the Products tab select All, then find and install the 
                     Windows Azure SDK 1.6 for Visual Studio 2010 (November 2011).
Yes, we ARE moving toward 1.7 SP1.

Media Services SDK

Reference:
 C:\Program Files (x86)\Microsoft SDKs\Windows Azure Media Services\Services\v1.0\Microsoft.WindowsAzure.MediaServices.Client.dll
Download from: http://www.microsoft.com/en-us/download/details.aspx?id=30153

Here is a little bit more about each step of building that first media asset:

1. What is a CloudMediaContext?

The .Net SDK does a number of things for you so you don’t need to be an expert at making REST calls.  In addition to surfacing all the REST functionality, it also simplifies things by managing connections, redirects, uploads, downloads, provides storage encryption prior to uploads, securely transfers configuration information and keys, it gives you the ability to enumerate your media objects:

  • Assets and their associated objects: Files, Locators, ContentKeys
  • Jobs and their associated objects: Tasks, Configurations, Assets
  • AccessPolicies and Media Processors

To make all this happen, the CloudMediaContext creates a state which holds your user credentials, connectivity information, and the state or data of some of the recently queried objects.  It is important to understand that this state or data can become stale if other threads or the Media Services act upon the objects.  In some cases, you can simply re-query the data, in others, you are best to discard the Context and request a new one.

To get a CloudMediaContext, you need your Media Services credentials.  You can retrieve these using the Azure Management Portal.  In this blog post, you created your account.  Now lets go back and get the credentials you need to use it: login to the Azure Management Portal.

Click on the Media Services icon on the far left.  Choose your Media Service account, in my case ndrouineast, and click Manage Keys at the bottom.

Take note of the Media Service name and the Primary Media Service Access Key, you will need those in the simple call:

CloudMediaContext context = new CloudMediaContext(
             "ndrouineast",
             "H6sdfsdfsdfsdfsdfsdfsdfsdfsdfsdfsdfsdfsdff4=");

Now that you have the context, you can do all sorts of things.

2. Create a new Asset using context.AssetCollection

Lots of ways to create an asset:

  • Create a new empty asset then add files to it.
  • Use some of the convenience features of the SDK to both create and upload at once.
  • Run a Job in which a Task creates an Asset as it’s output.
  • Use the BulkIngest SDK.

We’ll just cover creating an asset and uploading a single file.

IAsset asset = context.Assets.Create(“drive:path\filename.ext”, AssetCreationOptions.None);

That’s it a one-liner again.

If that’s all you need right now, you can stop reading.

Under the hood, the call above will: create an IAsset and add it to the assets collection; create an IFileInfo to put in the asset; create an AccessPolicy with write permission, request a SAS locator for the asset container in Azure Storage using the AccessPolicy, upload the file into your storage account, and finally revoke the SAS url.

Fair warning: we’re probably pulling this one-liner version of create, we feel it is just a little too simple: we need to make all sorts of assumptions in the SDK for that under-the-hood work that just don’t have the granularity that advanced users will need.  There is such a thing as too easy.  I’ll update the blog post when the new SDK is out, we’re looking at maybe 5 lines of code instead.  Meanwhile enjoy the simplicity.

So I hope you’ve had a chance to read about uploading into Azure Storage with Aspera.  So how does that tie in here?  Well, the first thing you’ll notice is that the above is a blocking call because of the upload.  You can also create an asset in three steps:

  1. Create an Empty Asset
  2. Upload a file to the storage container
  3. Update the asset to included the added files.

To create an empty asset, it’s a simple call:

IAsset inputAsset = context.Assets.CreateEmptyAsset(“assetName”, AssetCreationOptions.None);

At this point, you’ll want to add a file to it.  If you are not using our SDK to do the uploads as part of the create call, you can just get a SAS url for the asset container, add your file name to it and do a PUT on it.  The Azure Storage REST Api will process that and create a file for you.

Here’s how to get that SAS UploadUri:

IAccessPolicy writePolicy = context.AccessPolicies.Create(“Policy For Copying”, TimeSpan.FromMinutes(estimatedUploadMaxTime), AccessPermissions.Write | AccessPermissions.List);

ILocator destinationLocator = context.Locators.CreateSasLocator(asset, writePolicy, DateTime.UtcNow.AddMinutes(-5));

Uri uploadUri = newUri(destinationLocator.Path);

Now you can craft your own HttpRequest from the uploadUri, add you filename and you’re in full control.  Skip over the Aspera bit if that’s not for you, but you do need to do a final step, so read on.

So where does Aspera come in?

Well that uploadUri’s first Uri segment will be the asset contianer name, in the form: asset_guid.  You can now browse to the asset container name using the Aspera Desktop Client and upload that 3Gig file at blazing fast speeds.  Soon you’ll be using the Aspera SDK to script all that and you’ll really be off to the races.

Another upload option is the Azure Storage CloudBlob class, which can upload for you.

Once the file is in storage, either with an HttpRequest or Aspera, or the Azure Storage SDK, you need to let the Media Services know that you’ve added files that it isn’t aware of.  To Media Services, that is still an empty asset.

asset.Publish();

Under the hood, this is enumerating the files in the asset container, creating IFileInfo objects for each file and adding these to the asset in our databases.

That .Publish call is another tricky call which we’ve found doesn’t cover all the user scenarios properly.  I’ll blog about how to ‘do this right’ when the next Media Services SDK comes out.

We covered how to add an asset and populate it with a file in this post.  There are several variations on doing this, which make more sense when they are taken in context of a larger application or workflow.  I’m just trying to build a base that I can refer back to later.

Posted in Windows Azure Media Services | Tagged | 3 Comments

Yes, but I have a LOT of content.

Perhaps you’ve been considering Windows Azure Media Services.  Then, looking back at your asset library, or your weekly volume, and you thought:

How am I going to get 5 Terabytes of media up to the cloud?

Good question, you’re not the first to ask.

You have three options:

  • Threading the upload of one asset at a time.  HTTP bandwidth limited.
  • Using our Bulk Ingest .NET Library.  HTTP bandwidth limited.
  • Using a fast UDP upload client.

I will cover all three of these methods in this and future blog posts, but for now, let’s look at the problem of moving 5TB.

We have been working very hard with Aspera to integrate their fasp(tm) technology in to Azure.  While Media Services is not the only driver for massive or rapid transfers, our customers have a lot to gain and we’re proud to welcome Aspera to the ecosystem.

Aspera upload is an opt-in component on the Azure Marketplace.  The rest of this post is dedicated to walking through the installation of the tools, creating accounts, and connecting to your Azure Storage account.  In my next post, I will look at how to create a media asset using Aspera for upload.

What do I need?

First some background information, start by reading Aspera’s FAQ:

http://cloud.asperasoft.com/aspera-on-demand/aspera-on-demand-for-windows-azure-beta-faq/

1. You’ll need an Aspera transfer client.

To get your files from your disk to your storage account using Aspera, you’ll need an Aspera Transfer Client.  For simplicity, in this blog post, we will look at the installation and configuration of the desktop client.  You will probably want to look at the SDK or server solutions to see if these are a better fit for your automated work flow needs.  Of course, you can use any of these Aspera clients for your other data-transfer needs: Nothing here is media-specific.

On the Aspera desktop-client page, click the Evaluation Request link and fill in contact-information table.  Aspera will contact you with download instructions.  If you are planning on working out an enterprise agreement with Aspera, now would be the time to ask for a Promotion Code.

Install the client:

Read the information about local or domain accounts, and click through to the end of the installer, it will launch the desktop client.  Enter your license information, and you’ll get the main user interface.

As you can see, there is not much in the way of server connections, let’s address that next.

2. You’ll need to enable Aspera On Demand for Windows Azure

Next, let’s look at the Windows Azure Marketplace, in their words:

The Windows Azure Marketplace helps connect companies seeking innovative cloud based solutions with partners who have developed solutions that are ready to use.

That’s exactly what you’re looking for:  a solution for the fast, secure, transfer of lots of data into your Azure Storage Account.  To find Aspera, you can simply enter them in the search bar and choose “Aspera On Demand for Azure“.  Read those terms and conditions and follow the link to the Aspera website for additional pricing and details.

Go ahead and sign-in in the top-right with that Microsoft Account you associated to your Windows Azure account, as explained here.  If this is your first time logging in to the Windows Azure Marketplace, you’ll be asked some additional information and to agree to the terms and conditions regarding the 3rd party vendors, such as Aspera.

Once you’re done, use the search to get back to the Aspera page.  Click ‘Buy’ on the most appropriate offer.

If you’ve contacted the nice folks at Aspera and have a promo-code, now’s the time to enter it.  Enter your payment information, agree to the terms and hit ‘Sign up’.

Ok, all set, you’re ready to use it.  Or almost.  .

Click that ‘Use it!’ link to go to the credentials manager on AsperaOnDemand.com site.

You need to add credentials that you will be able to use to connect to the Windows Azure Aspera servers.  Go ahead and click ‘New User Credential’.

NOTE:  If you are adding this feature during the month of October 2012, you are still in the beta trial, good pricing, but also only available in “West US”.  This means that you need to select West US in this dialog box, and have a Media Services account linked to an Azure Storage account in West US.  See here about creating new accounts.

Enter the name you want to see in the list, the region in which your storage account is located, and copy the Host, User and Password information somewhere for you to use in your Aspera Client.

NOTE: As of this writing, the correct Host is: “west-us.azure.asperaondemand.com

Click ‘Save’ and you are back in the user credentials list.  If you need those credentials again, there is an icon in the list which you can click to see them again.

So at this point you should have:

  1. An Aspera Client to work with.
  2. An Aspera On Demand for Windows Azure account and credentials.

Next step:

3. Use the Aspera Client to access your Azure Storage account

Open the Aspera Desktop Client, it still looks like step 1, but now we’re ready to add a server connection.

Click the Connections icon, click the ‘+’ to add a new server, and select Windows Azure as the Storage Type:

We’re going to need to fill in some blanks here.

  • The top half is from your Aspera Credentials Manager Host, User and Password.
  • The bottom is from your Azure Storage Account Name and Key.

We already covered how to get the Aspera credentials earlier in this blog post, so let’s review where we get this from our storage account.

Choose Storage icon on the right, choose the storage account you want to upload to, click ‘Manage Keys’ icon at the bottom and use your Storage Account Name and your Primary Access Key to fill in the connection information:

Click the ‘Advanced’ button and change the SSH Port (TCP) to 33001.  Click OK and use the ‘Test Connection’ button to verify your settings.

Click OK to return to the transfer screen, select the server from the list and click Connect.

You can now copy files from your local file system up into Azure Storage.  It is not a good practice to copy files to the root folder of your storage account. Create an ‘uploads’ folder and move some files to and from to test your connectivity.

To check out the full capability, I clicked ‘Preferences’ and brought the ‘Initial Target Upload Rate’ to 600Mbps, which is higher than I expect my network to be able to provide.

Then I started an upload of a 522Mb MP4 of Sintel and clicked over to the details view:

At 291Mbps average and sub 20 seconds I’m happy with the performance.  Try your favorite Azure Storage upload tool for comparison.  Mine never hit above 12 Mbps, your results will vary.

My next post will discuss how all this applies to creating “media assets” in Windows Azure Media Services.

Posted in Windows Azure Media Services | Tagged , | 1 Comment

Sounds great. What next?

This attempts to answer the question:

what do I need to do, exactly, to start using Widows Azure Media Services.

1. You need accounts.

First and foremost, you need a Microsoft Account, formerly known as a Windows Live Id.  These are credentials on one of MSFT web properties, Hotmail, MSN, Live, etc.  I’ll assume that you either have one, or will have no trouble creating one at live.com.

With this Microsoft Acount, you create an account on Windows Azure.  This is linked to your Microsoft Account and has some billing information.  If you’re shy about billing, go ahead and use one of those 20$ pre-paid Visa cards, can’t break the bank with that!

Start here to create your Azure account: http://www.windowsazure.com/en-us/

Now that you have an account, access the Azure portal with your Microsoft Id., you can start adding the items that you’ll need on Azure.  You’ll need two types, just click the +New button at the bottom of the Azure portal to add:

So what’s it all for?

Your storage account will hold your media assets.  It is your repository for your data.  You control it and own it.  It is fully segregated from anyone else’s media.

Your Media Services account is what gives you access to the Media Platform as a Service.  This is a deployment of servers which work together to make managing your media seamless and painless.  Our servers are replicated in our data-centers around the world and provide redundancy, scalability and elasticity to the Media Services PaaS.  It is these servers that will receive, queue and process requests from you to act on your media assets.  It is these servers that will read the information in your storage account to process or stream your assets.  It is these servers that will provide you with secure uploading and downloading URLs to the media in your storage account.

When you created these accounts, you were asked to associate them to a region and link the Media Services account back to your storage account.  This did three things: it physically placed your storage account in a particular data center; it provided your storage account credentials (securely) to Media Services; and finally it set the affinity of your Media Services API calls to the region of your choice.  We do offer managed geo-replication of both storage and processing to enterprise partners, but let’s keep things simple for now.

2. You need to upload some content and do something with it.

Now that you can see both a Storage account and a Media Services instance associated to your Azure portal, go ahead an use it.

The easiest way will depend on your background.

  • If you are a Java person and love working with json, perfect, go ahead and use the REST API directly.
  • If you are a C# person, go ahead and get going with the .Net samples, these wrap the REST api and take alot of the heavy lifting out of the equation.
  • If you are a media person and both Java and C# make you cringe, then I might be able to help your there: stay tuned to this blog. [Edit: see here for a 'drop files here and start streaming in minutes' solution using Transform Manager.]
Posted in Windows Azure Media Services | Tagged | Leave a comment

An introduction to Windows Azure Media Services

You can find information about how to use and work with Windows Azure Media Services (WAMS) here:

http://www.windowsazure.com/en-us/develop/net/how-to-guides/media-services

In this blog, you’ll find some tips, tricks and code samples to use and work with WAMS. I will concentrate on the automation of large content libraries, but will also include several code snippets that can be used in more general cases.

In essence, WAMS is a media platform as a service (PAAS).  It is not simply infrastructure: servers, network and storage. It offers value-added services in the media space.  You can create media assets, change their format, or derive new formats, set up and control streaming, or download your assets back to your own servers for streaming.  It offers first rate security, massively scalable solutions and a growing list of features.

WAMS is also an ecosystem.  We work with partners to increase the value offered by the platform.  We acknowledge that media is a vast space and that we can’t offer everything under the sun.  By working with leading industry partners, we are bring together their expertise, integrating, standardizing and simplifying how users can interact with a multitude of vendors.

This is just the beginning.  By using simple open standards to communicate with the platform, there are opportunities for anyone to build on top of the platform.  This means that as a content owner, you can build a simple and secure service to manage your own media assets.  As an integrated services vendor, you can build complete content management solutions for your customers.  You’ll leverage the components offered by the underlying platform, brand specific solutions for your customers, or even sell a value-added service to the public.

I look forward to contributing to the body of knowledge that will drive the success of Windows Azure Media Services.

If you have any questions, don’t hesitate to contact me, or post in the forums.


Media Services

Posted in Windows Azure Media Services | Tagged | Leave a comment

A little bit about me

I work for Microsoft in the Windows Azure Media Services team as a Program Manager.

Prior to this, I worked for CAE Inc. for 11 years, the last few of which I spent working on a video capture project to support learning in healthcare education.

At Microsoft, I worked on releasing Transform Manager, and I am now focused on transmuxing workflows in the cloud.

 

Posted in Uncategorized | Tagged , | 4 Comments