Archive

Archive for October, 2009

H264 Video Encoding on Amazon’s EC2

October 28, 2009 2 comments
Stream #0 recently started looking at Amazon’s EC2 computing offering. We created our first public AMI, based on Debian Squeeze, including FFmpeg and x264 pre-installed. Now that we can easily start instances with the necessary basics installed, it is time to compare the relative merits of the different instance sizes that Amazon offers.
EC2 Instances come in a variety of sizes, with different CPU and RAM capacities. We tested the 64-bit offerings, including the recently announced High-Memory Quadruple Extra Large instance.
These 64-bit instances are listed on the EC2 website in the following way:
Large Instance 7.5 GB of memory, 4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each), 850 GB of instance storage, 64-bit platform
Extra Large Instance 15 GB of memory, 8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute Units each), 1690 GB of instance storage, 64-bit platform
High-CPU Extra Large Instance 7 GB of memory, 20 EC2 Compute Units (8 virtual cores with 2.5 EC2 Compute Units each), 1690 GB of instance storage, 64-bit platform
High-Memory Quadruple Extra Large Instance 68.4 GB of memory, 26 EC2 Compute Units (8 virtual cores with 3.25 EC2 Compute Units each), 1690 GB of instance storage, 64-bit platform
We’ll take a closer look later at the in-depth specifications of each below.
Our test file was 5810 frames (a little over 4 minutes and 285MB) of the HD 1920×1080 MP4 AVI version of Big Buck Bunny. The FFmpeg transcode would convert this to H264 using the following 2-pass command:
>ffmpeg -y -i big_buck_bunny_1080p_surround.avi -pass 1 -vcodec libx264 -vpre fastfirstpass -s 1920×1080 -b 2000k -bt 2000k -threads 0 -f mov -an /dev/null && ffmpeg -deinterlace -y -i big_buck_bunny_1080p_surround.avi -pass 2 -acodec libfaac -ab 128k -ac 2 -vcodec libx264 -vpre hq -s 1920×1080 -b 2000k -bt 2000k -threads 0 -f mov big_buck_bunny_1080p_stereo_x264.mov
Setting Threads to zero should mean that FFmpeg automatically takes advantage of the entire number of CPU cores available on each EC2 instance.
FFmpeg revealed the following information about the transcode:
Input #0, avi, from ‘big_buck_bunny_1080p_surround.avi’:
Duration: 00:09:56.48, start: 0.000000, bitrate: 3968 kb/s
Stream #0.0: Video: mpeg4, yuv420p, 1920×1080 [PAR 1:1 DAR 16:9], 24 tbr, 24 tbn, 24 tbc
Stream #0.1: Audio: ac3, 48000 Hz, 5.1, s16, 448 kb/s
[libx264 @ 0x6620f0]using SAR=1/1
[libx264 @ 0x6620f0]using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE4.1 Cache64
[libx264 @ 0x6620f0]profile High, level 4.0
Output #0, mov, to ‘big_buck_bunny_1080p_stereo_x264.mov’:
Stream #0.0: Video: libx264, yuv420p, 1920×1080 [PAR 1:1 DAR 16:9], q=10-51, pass 2, 2000 kb/s, 24 tbn, 24 tbc
Stream #0.1: Audio: aac, 48000 Hz, 2 channels, s16, 128 kb/s
Stream mapping:
Stream #0.0 -> #0.0
Stream #0.1 -> #0.1
Ignore the duration, as that’s read from the file header, and we only uploaded part of the overall file.
Now to look at how each EC2 instance performed.
m1.large
(Large Instance 7.5 GB of memory, 4 EC2 Compute Units)
Firstly, querying the machine capacity (cat /proc/cpuinfo) returns the following information:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU E5430 @ 2.66GHz
stepping : 6
cpu MHz : 2659.994
cache size : 6144 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu tsc msr pae mce cx8 apic mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm constant_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr dca lahf_lm
bogomips : 5322.41
clflush size : 64
cache_alignment : 64
address sizes : 38 bits physical, 48 bits virtual
power management:
There’s 2 of these cores available. RAM is confirmed as 7.5GB (free -g).
The FFmpeg transcode showed the following:
H264 1st Pass = 11fps – 18 fps, 5 minutes 30 seconds
H264 2nd Pass = 4-5fps, 18 minutes 38 seconds
Total Time: 24 minutes, 8 seconds
m1.xlarge
Extra Large Instance 15 GB of memory, 8 EC2 Compute Units
CPU Info:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU E5430 @ 2.66GHz
stepping : 10
cpu MHz : 2666.760
cache size : 6144 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu tsc msr pae mce cx8 apic mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm constant_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr dca lahf_lm
bogomips : 5336.15
clflush size : 64
cache_alignment : 64
address sizes : 38 bits physical, 48 bits virtual
power management:
There’s 4 of these cores available. RAM is confirmed at 15GB.
The FFmpeg transcode showed the following:
H264 1st Pass = 11fps – 14 fps, 5 minutes 30 seconds
H264 2nd Pass = 6-7fps, 14 minutes 19 seconds
Total Time: 19 minutes, 49 seconds
c1.xlarge
High-CPU Extra Large Instance 7 GB of memory, 20 EC2 Compute Units
CPU Info:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU E5410 @ 2.33GHz
stepping : 10
cpu MHz : 2333.414
cache size : 6144 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu tsc msr pae mce cx8 apic mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm constant_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr dca lahf_lm
bogomips : 4669.21
clflush size : 64
cache_alignment : 64
address sizes : 38 bits physical, 48 bits virtual
power management:
There’s 8 of these cores available. RAM confirmed at 7GB.
The FFmpeg transcode showed the following:
H264 1st Pass = 24-29fps, 3 minutes 24 seconds
H264 2nd Pass = 11-13fps, 7 minutes 8 seconds
Total Time: 10 minutes, 32 seconds
m2.4xlarge
High-Memory Quadruple Extra Large Instance 68.4 GB of memory, 26 EC2 Compute Units
CPU Info:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5550 @ 2.67GHz
stepping : 5
cpu MHz : 2666.760
cache size : 8192 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu tsc msr pae mce cx8 apic mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm constant_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr dca popcnt lahf_lm
bogomips : 5338.09
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
There’s 8 of these cores available. RAM confirmed at 68GB.
The FFmpeg transcode showed the following:
H264 1st Pass = 35-38fps, 2 minutes 47 seconds
H264 2nd Pass = 12-15fps, 6 minutes 30 seconds
Total Time: 9 minutes, 17 seconds
What can be revealed from these figures? As expected, the High-Memory Quadruple Extra Large Instance performed best, but not by much. Certainly all the additional RAM didn’t make much of an impact, and the time saving is probably really down to the slightly increased CPU specifications. Obviously, over a larger file set this time saving would be more evident.
Let’s look at which EC2 instance gives best value for money for this test. Amazon charges per CPU hour, shown below:
m1.large: $0.40/hour
m1.xlarge: $0.80/hour
c1.xlarge: $0.80/hour
m2.4xlarge: $2.40/hour
These are US Dollars and for a US based instance (European instances are slightly more expensive). Amazon has also revealed that there will be a price reduction
in effect from November 1st 2009.
Looking at the time taken to transcode our test file, on each instance, reveals the following:
m1.large
Total Time: 24 minutes, 8 seconds
Total Cost: $0.16 ((($0.40/60)/60) x 1448 seconds)
Cost per GB: $0.57 ((1024MB/285MB) x $0.16)
m1.xlarge
Total Time: 19 minutes, 49 seconds
Total Cost: $0.26 ((($0.80/60)/60) x 1189 seconds)
Cost per GB: $0.93 ((1024MB/285MB) x $0.26)
c1.large
Total Time: 10 minutes, 32 seconds
Total Cost: $0.14 ((($0.80/60)/60) x 632 seconds)
Cost per GB: $0.50 ((1024MB/285MB) x $0.14)
m2.4xlarge
Total Time: 9 minutes, 17 seconds
Total Cost: $0.37 ((($2.40/60)/60) x 557 seconds)
Cost per GB: $1.33 ((1024MB/285MB) x $0.37)
Clearly the c1.large instance represents the best value for money, although I was surprised how close behind the m1.large costs were. The additional RAM, and slightly better CPU specifications for the m2.4xlarge instance do not outweigh the much more expensive per hour cost, at least when it comes to video transcoding.
A typical HD file used for broadcast or high end post production purposes is around 85GB for 60 minutes (DnxHD at 185Mbps). Obviously the time taken to transcode this file, to an H264 at 2Mbps, could vary from the actual source content we used, but from the figures above we can estimate that it would cost $42.50 and take approximately 53.62 hours!
Taking into account that these figures may vary for different input and output files, the above should represent a worst case scenario. For example, I would expect an SD MPEG2 50Mbps file to take proportionally much less effort to transcode than a DNxHD 185Mbps HD file. Only a further test will tell……
Is Amazon’s EC2 offering worth considering for high end video file transcoding? Compared to the prices charged by Post-Production facilities it is certainly a lot cheaper, as long as you have time to wait for the end result. However, that’s the beauty of cloud based computing power – if you’re in a hurry just scale up! Keep in mind though, content still needs to be uploaded to EC2 before transcoding can begin, that’s going to take additional time and add further cost. 

AWS Debian Squeeze AMI with FFmpeg and X264

October 22, 2009 Leave a comment

Stream #0 has now made available our first Amazon Web Services (AWS) AMI. This is based on Eric Hammond’s 64-bit Squeeze AMI: ami-fcf61595.

The first Stream #0 AMI can be found by looking for the following AMI ID in the AWS Management Console: ami-b535d6dc

The following additions have been made over the base Squeeze build:

  • Added Debian Multimedia Repository
  • Updated and Upgraded to October 22nd 2009 latest packages
  • Build x264 from source. r1301
  • Build FFmpeg from source. r20350

FFmpeg has been configured as per the options noted in the How-To here

Ultimately we’re planning to build a few different AMI variations. e.g. Lenny with FFmpeg 0.5 build and x264 from Debian Multimedia Repo as a slightly more stable version of the “Squeeze build everything from source” AMI approach.

The AMI has been made public and Stream0 would really appreciate feedback on this, our first time AMI build.

Everything you need to know about Amazon’s Web Services:

Amazon Web Services
AWS Elastic Compute Cloud (EC2)
AWS Developer Guide
Alestic – listing Debian and Ubuntu AMIs
ec2debian Google Group

How-To Build FFmpeg on Debian Squeeze

October 22, 2009 8 comments

It’s been a long time now since I wrote my original How-To for building FFmpeg on Debian. A lot has changed since then, in both the Debian and FFmpeg world, so it’s definitely time for an update.

This tutorial describes how to build x264 and FFmpeg from scratch, on a base Debian Squeeze system. Throughout this tutorial I will be assuming that you are operating as either root or su, or aware of how to use sudo (make sure you’ve added yourself to the /etc/sudoers list).

First, we need to update the sources list. I use pico as my text editor, as I was a long time Pine mail user way back when. Feel free to use vi or emacs if you prefer.

Go to the Debian Multimedia repository site and download the keyring package. Follow the instructions for unpackaging it about half-way down the front page. Now update your sources list:

>pico /etc/apt/sources.list

Add deb http://www.debian-multimedia.org squeeze main on a new line and save the file.

>aptitude update
>aptitude full-upgrade

Now you’re using the latest sources and packages.

Next, install all the additional libraries we’ll need:

>aptitude install install build-essential subversion git-core yasm libgpac-dev libdirac-dev libgsm1-dev libschroedinger-dev libspeex-dev libvorbis-dev libopenjpeg-dev libdc1394-dev libsdl1.2-dev zlib1g-dev texi2html libfaac-dev libfaad-dev libmp3lame-dev libtheora-dev libxvidcore4-dev libopencore-amrnb-dev libopencore-amrwb-dev

Once that has successfully completed, it’s time to grab the latest x264 code:

>git clone git://git.videolan.org/x264.git
>cd x264
>./configure –enable-shared
>make
>make install

Hopefully all is still going well and you encountered no errors so far. Great, let’s grab FFmpeg from Subversion:

>svn checkout svn://svn.ffmpeg.org/ffmpeg/trunk ffmpeg
>cd ffmpeg

Now to configure FFmpeg. There’s so many options, it’s sometimes hard to know which ones to choose. The list below is my personal preference, but do try ./configure –help to assist in choosing your own.

>./configure –enable-gpl –enable-postproc –enable-pthreads –enable-libfaac –enable-libfaad –enable-libmp3lame –enable-libtheora –enable-libx264 –enable-shared –enable-nonfree –enable-libvorbis –enable-libgsm –enable-libspeex –enable-libschroedinger –enable-libdirac –enable-avfilter –enable-avfilter-lavf –enable-libdc1394 –enable-libopenjpeg –enable-libopencore-amrnb –enable-libopencore-amrwb –enable-version3

After a successful configuration, all the enabled decoders, encoders and muxers will be displayed. There are some configuration dependencies here. If you don’t –enable-gpl things like postproc will fail at build time. Next….

>make
>make install

“Make” will probably take quite a long time.

Optionally you may like to build qt-faststart as well. If you don’t know what this does, use Google, but in short it arranges atoms in QuickTime header files to allow for progressive download delivery.

>make tools/qt-faststart

If you try to use FFmpeg now, by simply typing “ffmpeg” you are likely to encounter an error regarding shared libraries (we did build FFmpeg with –enable-shared). To fix this we do the following:

>pico /etc/ls.so.conf

Add the line “/usr/local/lib” (without quotes) to this file and then save it. Read more about dynamically linked libraries here, specifically the fourth paragraph to explain what we just did.

>ldconfig

That’s it! Finished. Pretty easy, right? Now you just need to learn how to use FFmpeg, but that’s a topic for another day. Very briefly though, here’s a command line for creating a 2-pass H264 file, at 750kbps and 480×360 resolution, in a mov container, with progressive download enabled.

>ffmpeg -y -i inputfile.mpg -pass 1 -vcodec libx264 -vpre fastfirstpass -s 480×360 -b 750k -bt 750k -threads 0 -f mov -an /dev/null && ffmpeg -deinterlace -y -i inputfile.mpg -pass 2 -acodec libfaac -ab 128k -ac 2 -vcodec libx264 -vpre hq -s 480×360 -b 750k -bt 750k -threads 0 -f mov outputfile.mov

>/tools/qt-faststart outputfile.mov outputfilefast.mov

Categories: FFmpeg, Video Tags: , , ,

OpenShot Video Editor’s New Skin

October 21, 2009 Leave a comment

Call me shallow, but in all seriousness the main reason I’ve not tried OpenShot video editor is that I can’t stand the glossy, glassy, bubbly look of their default colour scheme. Dislike KDE? Not that I necessarily do, but OpenShot’s theme was like KDE, but worse, much, much worse.

It seems like that’s all just changed for the better! A new “matt” theme is now available for OpenShot, and at first glance it looks much more appealing to my eyes. This will fit much better with my matt XFCE theme
See the latest blog post over on the OpenShot website, with details of how to change from the default theme, and then have a look at the two screen shots below:
Default (Before):
Matt Theme (After):
I guess that takes away any perceived barrier I had to trying this software. Happy days, I can now temporarily join the OpenShot massive (as long as I don’t visit their website too often).

Kdenlive 0.7.6 Released

October 9, 2009 Leave a comment

Just 3 months after the last release, Kdenlive is out with 0.7.6. They even had a pre-release message on their homepage for the last week or so.


The new features are:

  • Title module: rewrite, now allows for basic animation (title zoom & scroll)
  • Track rename: users can now rename tracks
  • Composite transition: keyframes can now be moved
  • Clip management: image and audio clips are automatically monitored and updated in the timeline whenever they change on disk
  • User interface cleanup: cleaner look for timeline
  • Capture monitor improvements: now shows available disk space & timecode, also allows to choose a name for captured files
  • Project management: project settings dialog now allows you to clear the thumbnails cache and delete all videoclips that are not used in the project
  • Improved clip markers: they are now displayed in the clip monitor ruler, and user can easily go to each marker from the context menu
  • Shutdown computer after rendering

Apparently stability is improved as well, through using the latest 0.4.6 version of the MLT Framework.

Read all the details on the Release Notes page and download the application from the Kdenlive website.

Categories: Kdenlive, Video Tags: ,

Applying the HandBrake – Stopping Short of Video Transcoding

October 2, 2009 Leave a comment

Finding a full featured open source video transcoding solution is no easy task. There is of course FFmpeg, which is fantastic and powerful, but one does need to spend a serious amount of time learning the syntax and various options. This is fine if you’re taking one type of input file and creating one type of output file. Easy to script and I’ve seen some really good results, especially using the x264 library. However, trying to find a nice GUI to do the same thing, so far no joy.

I had really hoped HandBrake might be the right tool for the job. It’s been around for a long time now, it’s not just for Mac anymore and there seems to be an active community around it.

The last stable release was almost 10 months ago, but recently an SVN snapshot build was supplied, with some pre-compiled packages to go with it. The usual caveats apply, but the change log provided some tantalising updates, so I decided to try it out.

My home laptop is running Debian Squeeze, but I decided to try the Ubuntu 9.04 GUI package, as I was lazy and didn’t feel like compiling from source. Maybe this was a mistake….. The installation of the deb package went fine. No errors to speak of and a nice little HandBrake icon appeared in my Sound & Video menu. Starting the application also worked flawlessly.

I hadn’t used HandBrake before, so I didn’t know what to expect. On first glance the software appears to be mainly geared towards H.264 transcoding. While this is currently an important video codec, I was hoping for a much wider range of output options. I couldn’t find any.

Fine, let’s stick with H.264. My source file for testing was 4.2GB of DNxHD – a little over 3 minutes of content at 185Mbps. HandBrake uses FFmpeg in the backend to do the transcode heavy lifting. I know FFmpeg supports DNxHD, so I was not expecting any real problems. Unfortunately, my assumption proved to be incorrect.

Selecting the DNxHD file as source, HandBrake scanned the file for a couple of minutes, then stopped. No error message, no warning, just the UI sitting there waiting for the next input. Fortunately, there’s an Activity Window to select and view the command line output. Here’s what I saw…..

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from ‘/home/phillc/Media/South Pacific Avid DNxHD.mov’:
  Duration: 00:03:11.12, start: 0.000000, bitrate: 189668 kb/s
    Stream #0.0(eng): Video: dnxhd, 1920×1080, PAR 1:1 DAR 16:9, 25 tbr, 25 tbn, 25 tbc
    Stream #0.1(eng): Audio: pcm_s16le, 48000 Hz, mono, s16, 768 kb/s
    Stream #0.2(eng): Audio: pcm_s16le, 48000 Hz, mono, s16, 768 kb/s
    Stream #0.3(eng): Audio: pcm_s16le, 48000 Hz, mono, s16, 768 kb/s
    Stream #0.4(eng): Audio: pcm_s16le, 48000 Hz, mono, s16, 768 kb/s
    Stream #0.5(eng): Audio: pcm_s16le, 48000 Hz, mono, s16, 768 kb/s
    Stream #0.6(eng): Audio: pcm_s16le, 48000 Hz, mono, s16, 768 kb/s
    Stream #0.7(eng): Audio: pcm_s16le, 48000 Hz, mono, s16, 768 kb/s
    Stream #0.8(eng): Audio: pcm_s16le, 48000 Hz, mono, s16, 768 kb/s
[10:14:27] scan: decoding previews for title 1
[10:14:45] scan: could not get a decoded picture
[10:14:58] scan: could not get a decoded picture
[10:15:12] scan: could not get a decoded picture
[10:15:26] scan: could not get a decoded picture
[10:15:40] scan: could not get a decoded picture
[10:15:54] scan: could not get a decoded picture
[10:16:08] scan: could not get a decoded picture
[10:16:23] scan: could not get a decoded picture
[10:16:35] scan: could not get a decoded picture
[10:16:49] scan: could not get a decoded picture
[10:16:49] libhb: scan thread found 0 valid title(s)

What’s it all mean?

It would seem that FFmpeg recognised a valid DNxHD file. It found the video track and 8 tracks of audio, but after this, who knows! Not being able to get a decoded picture is not an error I’ve come across using FFmpeg from the command line before.

EDIT: The root of this problem is actually FFmpeg and not Handbrake. The DNxHD file was 10-bit, but FFmpeg currently only supports 8-bit files of this codec.

That’s as far as I’ve managed to travel in the HandBrake world. I had a look at some of the H.264 file output options, which are comprehensive, but don’t at first glance seem to support the general x264 preset options. HandBrake presets are more geared towards devices – iPod and Apple TV. It appears to be possible to queue multiple jobs using HandBrake, but not set watch folders nor apply rules for output files. E.g Once transcoding is finished, move file to folder X.

HandBrake is probably a good tool for those looking to create web media, from DVD or Camcorder source, but it’s not powerful enough to deal with high end media from the post-production or broadcast domain.

Cinelerra Quietly Releases New Version

October 1, 2009 Leave a comment

Without much ado, Heroine Virtual has quietly released Cinelerra 4.1 earlier this month.

A full list of detailed changes is available on the Cinelerra website.
In the race for who has the most effects in their Open Source editing tool, Cinelerra totals 58 in a list claimed not to be updated, including really useful items such a Reverse Telecine, Deinterlance and Denoise. For me, this trumps OpenShot’s Glow, Water and Echo.
After 10 minutes trawling, I can’t seem to figure out if the CinelerraCV (Community Version) is tracking to the latest code or not. There’s been no News announcements on their website since June 2009.
Follow

Get every new post delivered to your Inbox.