Recently at The Station we’ve decided to enter into digital data backup and archiving. What, you may ask, don’t we do this already? Well, typically in the past at the end of an edit job, the client has been happy to take away, for example, a Digital Betcam or HDCAMSR video tape. However, over the last twelve months, more and more clients are shooting digitally (tapeless) and also asking for digital delivery of final master assets. Therefore, we would like to offer as a service long-term data archiving. Spinning disk is not the best platform for this. Drives can fail, and to set up up a totally secure SAN is an expensive proposition.
Enter the world of data tape. The most recent advance, although at least twelve months old now, is LTO5. This format can store up to 1.5TB of uncompressed data, or 3TB of compressed data on one tape. Single tape, desktop drives are in the region of £2,500 and tapes retail around £60 (I’m sure better prices can be found if one tries). LTO data tape is an excellent, cost-effective medium for long-term archival purposes.
One of the great advantages of LTO5 is that drives can be mounted with LTFS and they simply appear as any other attached storage device. Files can be dragged and dropped to and from the tapes. Transfer rates are allegedly in the region of 140Mbps. We have secured a loan Quantum LTO5 drive from the excellent people at Era UK, via an introduction by our friends at JCA. Of course I was intending to connect this drive to a Linux machine, there was never any question about it. Unfortunately, Quantum only officially supports Red Hat and SuSe (I’m guessing Fedora and CentOS would probably work too), for which they provide compiled binaries. Now, I’m a Debian person and the best available machine had Squeeze installed. You know what’s coming next, and I love a challenge.
Start by downloading the source from Quantum. Unfortunately compiling this source wasn’t as simple as configure, make, make install. After installing the dependencies:
> sudo apt-get install libicu-dev libfuse-dev uuid uuid-dev libxml++2.6-2 libxml++2.6-dev
./configure worked without a problem. However, make was failing with the following error:
make: Entering directory `/home/station/Downloads/qtmltfs-1.2.0′
Making all in messages
make: Entering directory `/home/station/Downloads/qtmltfs-1.2.0/messages’
genrb number of files: 3
sh: ./icu-config: not found
pkgdata: icu-config: Could not read from icu-config. (fix PATH or use -O option)
required parameter is missing: -O is required for static and shared builds.
Run ‘pkgdata –help’ for help.
make: *** [bin_mkltfs_dat.o] Error 1
make: Leaving directory `/home/station/Downloads/qtmltfs-1.2.0/messages’
make: *** [all-recursive] Error 1
make: Leaving directory `/home/station/Downloads/qtmltfs-1.2.0′
make: *** [all] Error 2
Not really knowing how to solve this initially, I decided to see what would happen if I compiled these sources on my little Ubuntu 10.04 NBR netbook. Of course, the sources compiled without error, but a netbook is not really the device I wish to connect an LTO5 drive to. So, it seemed like there was some sort of Debian Squeeze related package error.
I’m not ashamed to say that I posted to LinuxQuestions.org and a user by the name of knudfl came up with the solution. Behold, the power of the Internet. After removing the development files for libicu44, and installing the relevant counterparts from libicu42, following the suggested directions, I was able to finish compiling the driver. Great, here we go…..
Not so fast. The next step is to format a tape. The supplied documentation from Quantum suggests the following:
> mkltfs -d /dev/st0
This returned the following error:
> Error remapping st device /dev/st0 to sg : Unable to open /proc/scsi/sg/devices
More investigation finds that /proc/scsi is a legacy way of dealing with scsi devices, and in the Debian kernel this has been disabled. The good news is that, there is an option within menuconfig to turn this on, but it does mean compiling your own kernel. A new challenge for me as I’d never needed to do this before.
More help from the Interwebs, and using a very helpful tutorial page, I was able to build my own custom Linux kernel (2.6.39 if you’re interested). It wasn’t hard at all.
Booting into this kernel and everything seems to work fine. The final problem is:
LTFS20062E Unsupported cartridge type (LTO3RW)
LTFS11299E Cannot format: unsupported medium
I guess LTO3 tapes don’t work in LTO5 drives.
In summary, to setup a Quantum LTO5 tape drive on Debian Squeeze, the following steps are necessary:
1. Download the sources from Quantum.
2. Install the necessary dependencies, including libicu42-dev files (follow the steps here).
3. Configure, make and make install the source
4. Compile your own kernel, with legacy scsi support by selecting that option in menuconfig (follow the tutorial here).
5. Buy LTO5 tapes and format them.
6. Mount the LTO5 drive.
I travel a great deal in Europe. Specifically Germany and Austria. I also use the Internet excessively. UK mobile data roaming rates are absurdly expensive. T-Mobile UK sends me a friendly text whenever I try to use data roaming informing me that it will cost £1 for 3MB, £5 for 20MB or £10 for 50MB. Crazy, and crazily expensive.
I finally decided to bight the bullet and buy a 3G data dongle in Austria. Originally I had thought to purchase something from 3 (Drei), however it seemed that everything advertised was on a two year contract. That’s not for me. Ultimately I went for a bob breitband package. This includes a ZTE MF180 3G dongle/stick and a SIM card. Buying the package for €50 was the easy bit. The package itself was €50 for the dongle, with 1GB data gratis, and then €9 for 9GB per month, on a 30 day rolling contract. €9 for 9GB isn’t too bad, although if you exceed this limit the cost is €4 per GB, which is bad. The other gotcha is that you need an Austrian bank account to setup a direct debit for the monthly payments. I won’t bore you with the details of how I have access to one of those.
After opening the package, inserting the SIM card and then plugging the dongle into my Mint Debian laptop, nothing happened. That’s right, nothing. The dongle showed a red light and Network Manager didn’t recognise any new modem. Cutting long stories short, I unplugged, plugged in, unplugged, plugged in, fiddled and swore for a good hour with no luck. At the point of the customer support help line answering, magically I obtained a connection. The light on the 3G dongle had turned blue. I had no idea how. All is good for the remainder or the day, except that the connection is slower than anticipated. Using speedtest.net, initially I was seeing a download speed of only 1.6Mbps and an upload of 0.10Mbps. Pretty poor. Later in the evening this increased to 2.6Mbps download and 0.3Mbps upload. Still not great.
This morning, I once again found it impossible to obtain a connection? Why? Why? WHY? I extracted the SIM card from the dongle and inserted it into my Nokia N900 phone. With the marvelous help of bluetooth, I easily obtained a data connection. The connection dropped occasionally, but in general all was good. The problem was not with the SIM.
Now that I had a connection again, I used our Googlian friends to search for “ZTE MF180 Debian” for answers. The answer was easy to find. Apparently, the 3G USB dongle is initially recognised as a CD-ROM drive. Eject the drive and everything works. Network Manager asks for a pin number to unlock the SIM, then one is able to connect. I don’t know why the dongle is recognised as a CD-ROM drive, and I don’t know why ejecting it fixes the problem, but it does. Thanks to Christian Amsuess for the information. If everything doesn’t work as expected after ejecting the CD drive, there is more help on Christian’s page.
The network speed is no better today, but at least I’m back online. I wouldn’t recommend bob breitband, even with the cool name, and I certainly would not recommend ZTE MF180 dongle. I’m sure ejecting the CD drive every time I need a connection will become annoying, but at least it works.
Stream #0 has now made available our first Amazon Web Services (AWS) AMI. This is based on Eric Hammond’s 64-bit Squeeze AMI: ami-fcf61595.
The first Stream #0 AMI can be found by looking for the following AMI ID in the AWS Management Console: ami-b535d6dc
The following additions have been made over the base Squeeze build:
- Added Debian Multimedia Repository
- Updated and Upgraded to October 22nd 2009 latest packages
- Build x264 from source. r1301
- Build FFmpeg from source. r20350
FFmpeg has been configured as per the options noted in the How-To here
Ultimately we’re planning to build a few different AMI variations. e.g. Lenny with FFmpeg 0.5 build and x264 from Debian Multimedia Repo as a slightly more stable version of the “Squeeze build everything from source” AMI approach.
The AMI has been made public and Stream0 would really appreciate feedback on this, our first time AMI build.
Everything you need to know about Amazon’s Web Services:
It’s been a long time now since I wrote my original How-To for building FFmpeg on Debian. A lot has changed since then, in both the Debian and FFmpeg world, so it’s definitely time for an update.
This tutorial describes how to build x264 and FFmpeg from scratch, on a base Debian Squeeze system. Throughout this tutorial I will be assuming that you are operating as either root or su, or aware of how to use sudo (make sure you’ve added yourself to the /etc/sudoers list).
First, we need to update the sources list. I use pico as my text editor, as I was a long time Pine mail user way back when. Feel free to use vi or emacs if you prefer.
Go to the Debian Multimedia repository site and download the keyring package. Follow the instructions for unpackaging it about half-way down the front page. Now update your sources list:
Add deb http://www.debian-multimedia.org squeeze main on a new line and save the file.
Now you’re using the latest sources and packages.
Next, install all the additional libraries we’ll need:
>aptitude install install build-essential subversion git-core yasm libgpac-dev libdirac-dev libgsm1-dev libschroedinger-dev libspeex-dev libvorbis-dev libopenjpeg-dev libdc1394-dev libsdl1.2-dev zlib1g-dev texi2html libfaac-dev libfaad-dev libmp3lame-dev libtheora-dev libxvidcore4-dev libopencore-amrnb-dev libopencore-amrwb-dev
Once that has successfully completed, it’s time to grab the latest x264 code:
>git clone git://git.videolan.org/x264.git
Hopefully all is still going well and you encountered no errors so far. Great, let’s grab FFmpeg from Subversion:
>svn checkout svn://svn.ffmpeg.org/ffmpeg/trunk ffmpeg
Now to configure FFmpeg. There’s so many options, it’s sometimes hard to know which ones to choose. The list below is my personal preference, but do try ./configure –help to assist in choosing your own.
>./configure –enable-gpl –enable-postproc –enable-pthreads –enable-libfaac –enable-libfaad –enable-libmp3lame –enable-libtheora –enable-libx264 –enable-shared –enable-nonfree –enable-libvorbis –enable-libgsm –enable-libspeex –enable-libschroedinger –enable-libdirac –enable-avfilter –enable-avfilter-lavf –enable-libdc1394 –enable-libopenjpeg –enable-libopencore-amrnb –enable-libopencore-amrwb –enable-version3
After a successful configuration, all the enabled decoders, encoders and muxers will be displayed. There are some configuration dependencies here. If you don’t –enable-gpl things like postproc will fail at build time. Next….
“Make” will probably take quite a long time.
Optionally you may like to build qt-faststart as well. If you don’t know what this does, use Google, but in short it arranges atoms in QuickTime header files to allow for progressive download delivery.
If you try to use FFmpeg now, by simply typing “ffmpeg” you are likely to encounter an error regarding shared libraries (we did build FFmpeg with –enable-shared). To fix this we do the following:
Add the line “/usr/local/lib” (without quotes) to this file and then save it. Read more about dynamically linked libraries here, specifically the fourth paragraph to explain what we just did.
That’s it! Finished. Pretty easy, right? Now you just need to learn how to use FFmpeg, but that’s a topic for another day. Very briefly though, here’s a command line for creating a 2-pass H264 file, at 750kbps and 480×360 resolution, in a mov container, with progressive download enabled.
>ffmpeg -y -i inputfile.mpg -pass 1 -vcodec libx264 -vpre fastfirstpass -s 480×360 -b 750k -bt 750k -threads 0 -f mov -an /dev/null && ffmpeg -deinterlace -y -i inputfile.mpg -pass 2 -acodec libfaac -ab 128k -ac 2 -vcodec libx264 -vpre hq -s 480×360 -b 750k -bt 750k -threads 0 -f mov outputfile.mov
>/tools/qt-faststart outputfile.mov outputfilefast.mov
NB: This How-To is now almost totally worthless as it is so out of date. Please click here for an updated version explaining how to install FFmpeg on Debian Squeeze.
I have to say, that the video manipulation
program FFmpeg, while very powerful, is not very user-friendly when it
comes to installation. While many Linux programs can be happily
installed from either a pre-compiled package, or downloading source and
compiling yourself, this isn’t necessarily the case with FFmpeg. The
ease of FFmpeg installation largely depends on how many different video
codecs and containers you want to be able to input or output. The
greater the number, the exponential increase in installation
difficulty. My main need was for FFmpeg to accept a wide range of input
formats, while outputting H.264 encoded QuickTime (MOV) files. Here’s
how I achieved this on a Debian Etch server……..
I’m going to assume that you are familiar with using the Linux
command prompt, moving between directories, editing text files and have
at least some experience compiling programs.
The first thing I would recommend doing is making an addition to your source repository lists.
Add the following line:
deb http://www.debian-multimedia.org stable main
This repository contains some essential libraries for xvid and
x264 (an open source H.264 codec) amongst other things. You’ll need to
install some software from here. The software may well be available
from other repositories too, that are already in your sources.list
file, but add this one to be safe.
Next rebuild your sources:
I would also recommend installing checkinstall. This program can
be used instead of a regular “make install” command and produces a deb
package file that will make re-installation or multiple machine
installs much easier. If checkinstall isn’t already on your machine
download it from:
Maybe navigate here with lynx, maybe use wget once you’ve found
the actual file you need, maybe download it with a GUI based web
browser and then copy it to your desired directory. It’s your choice. I
grabbed the latest .deb package. After the download, execute the following as root:
dpkg -i checkinstall_1.6.1-1_i386.deb
Checkinstall should have happily installed on your system. Now it’s time to really get into FFmpeg.
Build the dependencies:
apt-get build-dep ffmpeg
Next we’re going to install a whole lot more useful software
that will allow FFmpeg to output many more than just the minimal file
apt-get install liblame-dev libfaad-dev libfaac-dev
libxvidcore4-dev liba52-0.7.4 liba52-0.7.4-dev libx264-dev
We’ve also ensured that you have the necessary tools installed
to compile from source (build-essential) and obtain files from the Subversion version control repositories.
We’re ready to checkout FFmpeg itself:
svn checkout svn://svn.mplayerhq.hu/ffmpeg/trunk ffmpeg,
At the time of writing the latest revision was 11212. If you’d
feel more comfortable not using the lastest bleeding edge version of FFmpeg, issue the Subversion command as follows:
svn checkout -r 11212 svn://svn.mplayerhq.hu/ffmpeg/trunk ffmpeg
This will ensure that you are also downloading the 11212 revision. Once downloaded, move into the ffmpeg directory (cd ffmpeg) and configure:
./configure –enable-gpl –enable-pp
–enable-libvorbis –enable-liba52 –enable-libdc1394 –enable-libgsm
–enable-libmp3lame –enable-libfaad –enable-libfaac –enable-pthreads
–enable-libx264 -enable-libxvid –enable-shared
So, what have we done here……
- gpl – allow use of GPL code, the resulting libav* and ffmpeg will be under GPL
- pp – enable GPLed postprocessing support
- libvorbis – enable Vorbis encoding via libvorbis (http://www.vorbis.com/)
- liba52 – enable GPLed liba52 support (http://liba52.sourceforge.net/)
- libdc1394 – enable IIDC-1394 grabbing using libdc1394 and libraw1394 (http://sourceforge.net/projects/libdc1394/)
- libgsm – enable GSM support via libgsm (http://packages.debian.org/testing/sound/libgsm-tools)
- libmp3lame – enable MP3 encoding via libmp3lame (http://lame.sourceforge.net/download.php)
- libfaad – enable FAAD support via libfaad (http://www.audiocoding.com/faad2.html)
- libfaac – enable FAAC support via libfaac (http://www.audiocoding.com/faac.html)
- pthreads – use pthreads (http://en.wikipedia.org/wiki/POSIX_Threads)
- libx264 – enable H.264 encoding via x264 (http://www.videolan.org/developers/x264.html)
- libxvid – enable Xvid encoding via xvidcore, native MPEG-4/Xvid encoder exists (http://www.xvid.org/)
The essence of his information, and many more options, can be found by typing ./configure –help first.
(You might also consider including libtheora in your configuration, but I forgot at the time)
We’re now ready to make the installation files so at the command prompt:
If something goes wrong, and you need to start again, a useful command to know is:
Make sure you do this first and then run the configure command again.
You will be asked a few questions, which should be
straightforward enough to answer – yes to creating the documentation,
choose a name, select D for Debian package, lastly select number 3 and
type a version name that means something to you. Mine was
svn11212-etch-20071213. Checkinstall will now create a Debian package
of FFmpeg, bespoke for your system with the configuration options
you’ve selected earlier. Checkinstall WILL NOT install the package, so
don’t forget to do that:
dpkg -i ffmpeg_svn11212-etch-20071213-1_i386.deb
With some small amount of luck, you should now have a working
version of FFmpeg installed on your Debian Etch server. You will be
able to output H.264 encoded files in a variety of containers.
Now the fun part really begins as you spend days tinkering with
commands to output the best possible files. Documentation for using
FFMPEG can be found at:
(Credit for getting me started in the right direction goes to Paul Battley and his FFmpeg Ubuntu Feisty install how-to)