Stream #0 recently started looking at
Amazon's EC2 computing offering. We created
our first public AMI, based on Debian Squeeze, including FFmpeg and x264 pre-installed. Now that we can easily start instances with the necessary basics installed, it is time to compare the relative merits of the different instance sizes that Amazon offers.
These 64-bit instances are listed on the EC2 website in the following way:
• Large Instance 7.5 GB of memory, 4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each), 850 GB of instance storage, 64-bit platform
• Extra Large Instance 15 GB of memory, 8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute Units each), 1690 GB of instance storage, 64-bit platform
• High-CPU Extra Large Instance 7 GB of memory, 20 EC2 Compute Units (8 virtual cores with 2.5 EC2 Compute Units each), 1690 GB of instance storage, 64-bit platform
• High-Memory Quadruple Extra Large Instance 68.4 GB of memory, 26 EC2 Compute Units (8 virtual cores with 3.25 EC2 Compute Units each), 1690 GB of instance storage, 64-bit platform
We'll take a closer look later at the in-depth specifications of each below.
Our test file was 5810 frames (a little over 4 minutes and 285MB) of the
HD 1920x1080 MP4 AVI version of Big Buck Bunny. The FFmpeg transcode would convert this to H264 using the following 2-pass command:
>ffmpeg -y -i big_buck_bunny_1080p_surround.avi -pass 1 -vcodec libx264 -vpre fastfirstpass -s 1920x1080 -b 2000k -bt 2000k -threads 0 -f mov -an /dev/null && ffmpeg -deinterlace -y -i big_buck_bunny_1080p_surround.avi -pass 2 -acodec libfaac -ab 128k -ac 2 -vcodec libx264 -vpre hq -s 1920x1080 -b 2000k -bt 2000k -threads 0 -f mov big_buck_bunny_1080p_stereo_x264.mov
Setting Threads to zero should mean that FFmpeg automatically takes advantage of the entire number of CPU cores available on each EC2 instance.
FFmpeg revealed the following information about the transcode:
Input #0, avi, from 'big_buck_bunny_1080p_surround.avi':
Duration: 00:09:56.48, start: 0.000000, bitrate: 3968 kb/s
Stream #0.0: Video: mpeg4, yuv420p, 1920x1080 [PAR 1:1 DAR 16:9], 24 tbr, 24 tbn, 24 tbc
Stream #0.1: Audio: ac3, 48000 Hz, 5.1, s16, 448 kb/s
[libx264 @ 0x6620f0]using SAR=1/1
[libx264 @ 0x6620f0]using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE4.1 Cache64
[libx264 @ 0x6620f0]profile High, level 4.0
Output #0, mov, to 'big_buck_bunny_1080p_stereo_x264.mov':
Stream #0.0: Video: libx264, yuv420p, 1920x1080 [PAR 1:1 DAR 16:9], q=10-51, pass 2, 2000 kb/s, 24 tbn, 24 tbc
Stream #0.1: Audio: aac, 48000 Hz, 2 channels, s16, 128 kb/s
Stream mapping:
Stream #0.0 -> #0.0
Stream #0.1 -> #0.1
Ignore the duration, as that's read from the file header, and we only uploaded part of the overall file.
Now to look at how each EC2 instance performed.
m1.large
(Large Instance 7.5 GB of memory, 4 EC2 Compute Units)
Firstly, querying the machine capacity (cat /proc/cpuinfo) returns the following information:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU E5430 @ 2.66GHz
stepping : 6
cpu MHz : 2659.994
cache size : 6144 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu tsc msr pae mce cx8 apic mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm constant_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr dca lahf_lm
bogomips : 5322.41
clflush size : 64
cache_alignment : 64
address sizes : 38 bits physical, 48 bits virtual
power management:
There's 2 of these cores available. RAM is confirmed as 7.5GB (free -g).
The FFmpeg transcode showed the following:
H264 1st Pass = 11fps - 18 fps, 5 minutes 30 seconds
H264 2nd Pass = 4-5fps, 18 minutes 38 seconds
Total Time: 24 minutes, 8 seconds
m1.xlarge
Extra Large Instance 15 GB of memory, 8 EC2 Compute Units
CPU Info:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU E5430 @ 2.66GHz
stepping : 10
cpu MHz : 2666.760
cache size : 6144 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu tsc msr pae mce cx8 apic mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm constant_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr dca lahf_lm
bogomips : 5336.15
clflush size : 64
cache_alignment : 64
address sizes : 38 bits physical, 48 bits virtual
power management:
There's 4 of these cores available. RAM is confirmed at 15GB.
The FFmpeg transcode showed the following:
H264 1st Pass = 11fps - 14 fps, 5 minutes 30 seconds
H264 2nd Pass = 6-7fps, 14 minutes 19 seconds
Total Time: 19 minutes, 49 seconds
c1.xlarge
High-CPU Extra Large Instance 7 GB of memory, 20 EC2 Compute Units
CPU Info:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU E5410 @ 2.33GHz
stepping : 10
cpu MHz : 2333.414
cache size : 6144 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu tsc msr pae mce cx8 apic mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm constant_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr dca lahf_lm
bogomips : 4669.21
clflush size : 64
cache_alignment : 64
address sizes : 38 bits physical, 48 bits virtual
power management:
There's 8 of these cores available. RAM confirmed at 7GB.
The FFmpeg transcode showed the following:
H264 1st Pass = 24-29fps, 3 minutes 24 seconds
H264 2nd Pass = 11-13fps, 7 minutes 8 seconds
Total Time: 10 minutes, 32 seconds
m2.4xlarge
High-Memory Quadruple Extra Large Instance 68.4 GB of memory, 26 EC2 Compute Units
CPU Info:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5550 @ 2.67GHz
stepping : 5
cpu MHz : 2666.760
cache size : 8192 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu tsc msr pae mce cx8 apic mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm constant_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr dca popcnt lahf_lm
bogomips : 5338.09
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
There's 8 of these cores available. RAM confirmed at 68GB.
The FFmpeg transcode showed the following:
H264 1st Pass = 35-38fps, 2 minutes 47 seconds
H264 2nd Pass = 12-15fps, 6 minutes 30 seconds
Total Time: 9 minutes, 17 seconds
What can be revealed from these figures? As expected, the High-Memory Quadruple Extra Large Instance performed best, but not by much. Certainly all the additional RAM didn't make much of an impact, and the time saving is probably really down to the slightly increased CPU specifications. Obviously, over a larger file set this time saving would be more evident.
Let's look at which EC2 instance gives best value for money for this test. Amazon charges per CPU hour, shown below:
m1.large: $0.40/hour
m1.xlarge: $0.80/hour
c1.xlarge: $0.80/hour
m2.4xlarge: $2.40/hour
These are US Dollars and for a US based instance (European instances are slightly more expensive). Amazon has also revealed that there will be a price reduction in effect from November 1st 2009.
Looking at the time taken to transcode our test file, on each instance, reveals the following:
m1.large
Total Time: 24 minutes, 8 seconds
Total Cost: $0.16 ((($0.40/60)/60) x 1448 seconds)
Cost per GB: $0.57 ((1024MB/285MB) x $0.16)
m1.xlarge
Total Time: 19 minutes, 49 seconds
Total Cost: $0.26 ((($0.80/60)/60) x 1189 seconds)
Cost per GB: $0.93 ((1024MB/285MB) x $0.26)
c1.large
Total Time: 10 minutes, 32 seconds
Total Cost: $0.14 ((($0.80/60)/60) x 632 seconds)
Cost per GB: $0.50 ((1024MB/285MB) x $0.14)
m2.4xlarge
Total Time: 9 minutes, 17 seconds
Total Cost: $0.37 ((($2.40/60)/60) x 557 seconds)
Cost per GB: $1.33 ((1024MB/285MB) x $0.37)
Clearly the c1.large instance represents the best value for money, although I was surprised how close behind the m1.large costs were. The additional RAM, and slightly better CPU specifications for the m2.4xlarge instance do not outweigh the much more expensive per hour cost, at least when it comes to video transcoding.
A typical HD file used for broadcast or high end post production purposes is around 85GB for 60 minutes (DnxHD at 185Mbps). Obviously the time taken to transcode this file, to an H264 at 2Mbps, could vary from the actual source content we used, but from the figures above we can estimate that it would cost $42.50 and take approximately 53.62 hours!
Taking into account that these figures may vary for different input and output files, the above should represent a worst case scenario. For example, I would expect an SD MPEG2 50Mbps file to take proportionally much less effort to transcode than a DNxHD 185Mbps HD file. Only a further test will tell......
Is Amazon's EC2 offering worth considering for high end video file transcoding? Compared to the prices charged by Post-Production facilities it is certainly a lot cheaper, as long as you have time to wait for the end result. However, that's the beauty of cloud based computing power - if you're in a hurry just scale up! Keep in mind though, content still needs to be uploaded to EC2 before transcoding can begin, that's going to take additional time and add further cost.
