4 May
2010
4 May
'10
4:12 p.m.
greetings!
i'm looking at my pile of external audio drives and backup drives, and wondering about
better solutions.
a friend has recently mentioned multiple-drive bays, which allow you to swap raw drives
in/out at will, eliminating all the individual enclosures. we also had a discussion about
bays which implement hardware RAID for realtime mirroring.
i'm wondering if any of you have experience with these sorts of devices in the context
of linux multitrack audio work. specific concerns might be things like:
1. reduction of total throughput in a combined-disk device relative to separate drive
enclosures connected to their own USB ports
2. reduction of total throughput caused by the RAID mirroring process, even when it is
done by dedicated hardware
3. linux compatibility
4. expense
any experience and/or recommendations about doing this sort of thing would be most
welcome! thanks in advance.
cheers!
--
.pltk.
4 May
4 May
10:01 p.m.
(1) Use Linux Software Raid and not "FakeRaid" -- marketing and florid prose
might tempted you into thinking FakeRaid was hardware assisted raid, it
isn't; it will cause you a world of woe if you ever need to recover a disk
-- all the recovery tools end up requiring a windows partition had already
been installed.... For details,
https://raid.wiki.kernel.org/index.php/Linux_Raid == LSR.
In fact a dual-boot w/ windows is the only reason to use fakeraid (
https://help.ubuntu.com/community/FakeRaidHowto ):
Why not use a linux software raid?
If you have arrived here after researching this topic
on the Internet, you
know that a common response to this question is, "I don't know if you can
actually do that, but why bother -- Linux has built-in softRAID capability."
Also, it's not clear that there is any performance gain using hardware
fakeRAID under Linux instead of the built-in softRAID capability; the CPU
still ends up doing the work. The most common reason for using fakeRAID is
in a dual-boot environment, where both Linux and Windows must be able to
read and write to the same RAID partitions.
...
Chances are, if it's cheap, it's fakeraid and closed by virtue of the
closed-ness of your motherboard manufacturer or the manufacturer of the
particular "raid solution in a box."
And also consider that if your special RAID hardware dies some years in the
future, there may be no easy way to get that hardware controller again, and
your disks, written with it's own special protocol, will be useless. With
LSR that
won't be a problem -- chances are there'll still be some still-working
branch of the open-source code that'll work with your old data on your old
drives, far into the future. As long as there's SATA ports on motherboards
and drives, you'll be able to recover and read your LSR-created drives, even
between distros.
<https://raid.wiki.kernel.org/index.php/Linux_Raid>(2) seems like many
motherboards have at least two e-sata ports (one internal and one external)
which
could be routed into an external drive-bay container like this:
http://www.directron.com/ms2tplus.html
(3) Although I'm using fixed disks, i've been using Linux Sofware Raid 1 on
Fedora 11&12 -- and it works great -- as protection against disk failure.
Don't use Raid 0... buy a faster or bigger disk instead. LSR's performance
impact on modern desktop processors seems negligible. For situations where
you're really pounding the disks (internet server, database server, video
server, or multichannel 192k audio) plug in a hardware PCI or PCIe Raid
controller card and don't use LSR.
(4) The only thing that I could see as dangerous w/ an external setup --
make sure there's no way of "unpairing" Raid-1 sets, including having one
drive not powered on, or properly plugged in, etc... Because your system
will happily come up using only one disk of a raid-set. Then you get to
spend some time syncing disks, and hoping (or furiously reading manual
pages) you can figure out a way to make sure it syncs the newer disk over
the old one, and not vice-versa.
By the way, LSR is very nice, even while rebuilding... you'll notice your
load higher and your disk IO up... but otherwise, you can continue working
(maybe not enough i/o leftover for heavy video or audio editing, but enough
that you'll still be able to use your computer). The amount of bandwidth for
disk checking and rebuilding can be adjusted & configured. My cron checks my
disks weekly noting the configured disk-usage bandwidth constraints:
May 2 04:10:01 gnulem kernel: md: data-check of RAID array md1
May 2 04:10:01 gnulem kernel: md: minimum _guaranteed_ speed: 1000
KB/sec/disk.
May 2 04:10:01 gnulem kernel: md: using maximum available idle IO bandwidth
(but not more than 200000 KB/sec) for data-check.
May 2 04:10:01 gnulem kernel: md: using 128k window, over a total of
485107136 blocks.
May 2 04:10:01 gnulem kernel: md: delaying data-check of md0 until md1 has
finished (they share one or more physical units)
Niels
http://nielsmayer.com
PS: Fedora 13 is going to have something even cooler -- BTRFS -- you'lll be
able to "roll back" to a the copy of the filesystem before you overwrote the
only copy you had of an original source track...
http://www.phoronix.com/scan.php?page=article&item=fedora_13_btrfs&… ...
http://www.phoronix.com/scan.php?page=news_item&px=NzcyNA .... Note a
missing feature/bug from ReiserFS (
http://en.wikipedia.org/w/index.php?title=Comparison_of_file_systems&ol…
).
:-/
5 May
5 May
1:40 a.m.
Yo, back on Tuesday 04 May 2010 plutek was all like:
greetings!
i'm looking at my pile of external audio drives and backup drives, and wondering
about better solutions.
a friend has recently mentioned multiple-drive bays, which allow you to swap raw drives
in/out at will, eliminating all the individual enclosures. we also had a discussion about
bays which implement hardware RAID for realtime mirroring.
i'm wondering if any of you have experience with these sorts of devices in the
context of linux multitrack audio work. specific concerns might be things like:
1. reduction of total throughput in a combined-disk device relative to separate drive
enclosures connected to their own USB ports
2. reduction of total throughput caused by the RAID mirroring process, even when it is
done by dedicated hardware
3. linux compatibility
4. expense
any experience and/or recommendations about doing this sort of thing would be most
welcome! thanks in advance.
cheers!
If you want to use hardware RAID. Make sure you get a real hardware raid, not one of those
soft RAID controllers built into the BIOS jobs. They are worthless and actually slower the
Linux software raid. (I set one up once and figured out what it was doing was creating a
FAT partition on each drive, each containing a disk image file. Stuuuuuupid.)
Don't use USB or firewire raid units. *cough* Glyph *cough* It's like the kid
next door that bolts a spoiler on his 1998 Grand Am. It's not going to make it any
faster.
If you just want backup, a software raid 1 will do fine.
If you are looking for serious throughput with lots of tracks, consider a hardware SATA2
RAID 5 card (meaning that there is a hardware chip calculating the parity bits, and it has
on onboard cache) If it's less that $100 it's probably not what you are looking
for.
If you want a super fast RAID, use SSD drives.
If you want swappable drives, make sure you get a quality back-plane. I had a cheap one
once that fried 4 motherboards before I figured out what was going on.
-Reuben
10:14 a.m.
Hi,
On Tuesday 04 May 2010 16:12:39 plutek wrote:
i'm looking at my pile of external audio drives
and backup drives, and
wondering about better solutions.
a friend has recently mentioned multiple-drive bays, which allow you to
swap raw drives in/out at will, eliminating all the individual enclosures.
we also had a discussion about bays which implement hardware RAID for
realtime mirroring.
i'm wondering if any of you have experience with these sorts of devices in
the context of linux multitrack audio work. specific concerns might be
things like:
1. reduction of total throughput in a combined-disk device relative to
separate drive enclosures connected to their own USB ports
I think the bottleneck here is usb, not disks...
2. reduction of total throughput caused by the RAID
mirroring process, even
when it is done by dedicated hardware
3. linux compatibility
4. expense
any experience and/or recommendations about doing this sort of thing would
be most welcome! thanks in advance.
Here is my advice:
Don't do hardware-raid! Neither the real nor the "soft" hardware raid help
you
that much. What do you do when the controller fails?
With a hardware-raid you have to have a second of the same kind in stock to
get back the data on your disks. Don't even think about not having a spare
controller and buying one when yours fails.
Murphies law says the controller will fail at exactly the time you have a
_very_ important project going on and the controller to use is not sold
anymore.
With a software-raid you just plug the disks in another machine/controller and
return to working on your project.
The "reduced" throughput of a software-raid is worth the ease of use. And its
not that "reduced" at all.
Oh, and use only raid1 or combinations of 0 and 1. For all the others see
http://baarf.com.
And if you want more freedom for your disks and partitions, use lvm. Just this
month I had a disk in an array of 5 that announced its failing in smart.
Marked the raid1-copy of the systempartition on that disk as faulty (hot-spare
kicked in), did lvmove to move all the logical volume off that disk and carried
the disk to my dealer for an rma. Two weeks later I shoved in the
replacement... Server uptime: not interrupted. Admins sanity: not influenced.
Of course that only works if you have *some* free space on the physical
volumes.
Arnold
10:39 a.m.
Arnold,
Here is my advice:
Don't do hardware-raid! Neither the real nor the "soft" hardware raid help
you
that much. What do you do when the controller fails?
Well, this is a stupid advice, unless you tell plutek also to get
everything else in his studio twice.
A hardware raid does all the computation for you and just passes along
the data you need. It also eats away a lot of stress through huge
buffers. With an Areca 1280 for example, I get 3GB/s for the first
second and ~1.5GB/s sustained on a RAID6.
With software raid, you have to transfer all data twice through the
system, you fuck up you caches, and the CPU has to do all the
computations. If you only have two drives, it doesn't matter that much.
So it really depends on what you want to do and what your budget is. If
you have to avoid IO stress, go and get a decent RAID controller.
With a hardware-raid you have to have a second of the
same kind in stock to
get back the data on your disks. Don't even think about not having a spare
controller and buying one when yours fails.
This is only true if the on-disk format is not specified somewhere. If
you buy cheap shit, that may be and you may have to fiddle around to
extract the data. But it can be done.
The "reduced" throughput of a software-raid
is worth the ease of use. And its
not that "reduced" at all.
Again, it depends on the use case.
Oh, and use only raid1 or combinations of 0 and 1. For
all the others see
http://baarf.com.
Again, it depends on the use case. As a general rule, this is just wrong.
Flo
--
Machines can do the work, so people have time to think.
public key DA43FEF4 x-hkp://wwwkeys.eu.pgp.net
11:02 a.m.
Hi,
On Wednesday 05 May 2010 10:39:32 Florian Faber wrote:
Imagine how plutek runs to a friends computer (or just breaks out his second
machine) when the first fails during the session?
And now imagine plutek checking ebay for days to hunt down the same hw-
controller as the one that just fried up...
Oh, most people have two or more audio-interfaces. And two or more headphones
and speaker-pairs. And two or more guitars (if they are guitar-players).
But with the disks storing the results of the work, he should be careless???
Again, the question is probably not so much the "if" or "how" but the
"how
fast" and "how pricey".
I know business you have lost raid5-systems (and all the data on it) because a
second disk broke under the stress of reconstructing a first failed disk.
But be my guest, I (*) work in a company helping such firms after disaster.
Note that we also write invoices for our work.
Have fun,
Arnold
(*) insert: almost
Don't do
hardware-raid! Neither the real nor the "soft" hardware raid
help you that much. What do you do when the controller fails?
Well, this is a
stupid advice, unless you tell plutek also to get
everything else in his studio twice. A hardware raid does all the computation for you and
just passes along
the data you need. It also eats away a lot of stress through huge
buffers. With an Areca 1280 for example, I get 3GB/s for the first
second and ~1.5GB/s sustained on a RAID6.
With software raid, you have to transfer all data twice through the
system,
No, the only place where it feeds through twice is the part memory->disk-
controller (which is pretty fast thanks to dma). Before that its just one
stream, after that its one dedicated bus per device (given this centuries
sata/sas).
you fuck up you caches,
If the linux kernel fucks up its caches when its doing its own raid, something
is _horribly_ wrong. And I doubt it is.
and the CPU has to do all the
computations. If you only have two drives, it doesn't matter that much.
What computations? - Ah, you still don't know that any [345]-raid is bad...
So it really depends on what you want to do and what
your budget is. If
you have to avoid IO stress, go and get a decent RAID controller.
If you want to avoid user-stress, do software-raid.
With a
hardware-raid you have to have a second of the same kind in stock
to get back the data on your disks. Don't even think about not having a
spare controller and buying one when yours fails.
This is only true if the on-disk
format is not specified somewhere. If
you buy cheap shit, that may be and you may have to fiddle around to
extract the data. But it can be done. The
"reduced" throughput of a software-raid is worth the ease of use. And
its not that "reduced" at all.
Again, it depends on the use case.
Oh, and use only raid1 or combinations of 0 and
1. For all the others see
http://baarf.com.
Again, it depends on the use case. As a general rule, this is
just wrong.
11:41 a.m.
Arnold,
For 0/1, yes.
If you only look at 0/1, of course not.
Arnold, keep cool. There's more than 01345. And for good reasons.
It all depends on what you are doing. There is no general rule of thumb.
If you are dealing with professional video for example, you have very
high demands on bandwidth and reliability. If you want to achieve the
same reliability and performance you can easily get with a RAID 6(0)
with any combination of 0 and 1, you are wasting a huge pile of money.
Yes. The only thing that got me started was your general advise not to
use a hardware RAID. I'm all with you to avoid cheap hardware RAIDs. But
a proper hardware RAID is pure bliss.
And now imagine plutek checking ebay for days to hunt
down the same hw-
controller as the one that just fried up...
I am not arguing pro cheap shit. I am arguing against a general advice
to use software RAID, as there a numerous things that just don't work
with a software RAID - I'm talking about actually getting things done,
not theory.
Oh, most people have two or more audio-interfaces. And
two or more headphones
and speaker-pairs. And two or more guitars (if they are guitar-players).
What if his mainboard breaks, and he is using a PCI or Firewire
soundcard? And there's no mainboard with PCI slot or Firewire around
anymore?
Your argument 'if you buy a hardware RAID controller, buy a second one'
is simply not true. Yes, there are a lot of cheap solutions out there
and a lot of people got burned. But in general, this is as true as
'firewire is dead' :)
With software
raid, you have to transfer all data twice through the
system,
No, the only place where it feeds through twice is the part
memory->disk-
controller (which is pretty fast thanks to dma). Before that its just one
stream, after that its one dedicated bus per device (given this centuries
sata/sas). you fuck up
you caches,
If the linux kernel fucks up its caches when its doing its own raid,
something
is _horribly_ wrong. And I doubt it is. and the CPU
has to do all the
computations. If you only have two drives, it doesn't matter that much.
What
computations? - Ah, you still don't know that any [345]-raid is bad... So it really
depends on what you want to do and what your budget is. If
you have to avoid IO stress, go and get a decent RAID controller.
If you want to
avoid user-stress, do software-raid. This is only
true if the on-disk format is not specified somewhere. If
you buy cheap shit, that may be and you may have to fiddle around to
extract the data. But it can be done.
Again, the question is probably not so much
the "if" or "how" but the "how
fast" and "how pricey". I know business you have lost raid5-systems (and all
the data on it) because a
second disk broke under the stress of reconstructing a first failed disk.
We don't have to argue about the stupidity of RAID 5, specially with
nowadays disk sizes, or lazy administrators, that don't do integrity
checks on a regularly basis. For above mentioned setups, I use RAID 6(0)
with hot spares.
But be my guest, I (*) work in a company helping such
firms after disaster.
Note that we also write invoices for our work.
That is very fortunate for you. In that case of course you know that it
is almost always the user (administrator) that fucked up. Either by
chosing a wrong setup, or by making a minor error worse when taking
wrong actions.
Flo
--
Machines can do the work, so people have time to think.
public key DA43FEF4 x-hkp://wwwkeys.eu.pgp.net
6:50 p.m.
Below are some numbers on performance that correspond to the more normal
"musician" usage of RAID, which would be raid-1 to protect against data-loss
due to disk failure. Other levels of RAID end up being pretty unworkable,
esp in a situation where you're using removable Raid-pairs in media-bays to
store your media. And what's the point of doing Raid-5 (and wasting all that
disk space and hardware controllers) in a situation where the disk isn't
permanently mounted and needing to be online 24/7 -- aka a "server".
So it's important to characterize what you're trying to do before jumping to
wild conclusions about what is "stupid." Stupid for a server room might be
smart for a home recording setup. And far too often, i've watched supposedly
professional sysadmins screw up their fancy RAID setups and lose everybody's
data (they also forgot to check that the backups were actually valid) --
something that probably woulnd't have happened nearly as often if they'd
have just gotten a single SATA disk and hooked it up without RAID, perhaps
even on a standard "consumer" computer instead of their special overpriced
"server" equipment (don't even start me on when
<large-comp-co-killed-by-linux-and-now-acquired> would force their open
source projects to use their crap "server" equipment that performed worse
and were less reliable than some snot-nose gamer kid's rig.)
Anyways, enough talk, some numbers:
https://raid.wiki.kernel.org/index.php/Performance#Performance_of_raids_wit…
RAID type sequential read random read sequential write
random write
Ordinary disk 82 34 67 56
RAID0 155 80 97 80
RAID1 80 35 72 55
RAID10,n2 79 56 69 48
RAID10,f2 150 79 70 55
You'll note that RAID-1 performs about the same as an ordinary disk. In some
cases of reading, RAID-1 can perform better than ordinary disks.
The other question to ponder, in this day and age of fast processors.....
how much are you actually getting out of your expenditure for a hardware
RAID controller. For example, the Areca 1280 was mentioned earlier (
http://www.google.com/products/catalog?q=Areca+1280&um=1&ie=UTF-8&a…
)
-- $1,169 new!! For that price you could buy a whole other computer and let
its general purpose CPU's and LSW do all the RAID computations -- aka a disk
server. For that matter, why not just get a new 6-core processor (
http://www.anandtech.com/show/3674/amds-sixcore-phenom-ii-x6-1090t-1055t-re…
) instead
of a 4-core (or a 3 or 4 core instead of a 2 core) and when the additional
cores are done doing your RAID computations, you'll be able to use them for
more important and tangible things -- audio and graphics.
Isn't that a better use of potentially limited cash and space resources?
Niels
http://nielsmayer.com
5317
days inactive
5318
days old
linux-audio-user@lists.linuxaudio.org
7 comments
5 participants
participants (5)
-
Arnold Krille -
Florian Faber -
Niels Mayer -
plutek -
Reuben Martin