Problem: I have about 50 GB of irreplaceable data, that I would like to have regular access to, over the Internet.
For the past couple years I have been using a Linux box at home as NAS, running a RAID-5 array with daily incremental backup to Amazon Glacier (an extremely cheap and reliable storage service that has a retrieval latency of around 5 hours). That gives me fast access at home (over my gigabit LAN), but accessing over the Internet has always been problematic. I usually use SFTP/SCP/SSHFS, but that really only work for small files. For big files it gets a little ridiculously slow even though I have pretty good 10 MBps up at home. It’s even more painful from places where ping to the server is high, like across the Pacific Ocean. With 300ms ping, directory browsing is almost impossible.
Since unreliable storage is so cheap now (that is, harddrives in PCs), it feels to me like cloud-based storage (for the lack of a non-buzzword-term) sync-ed locally to each computer is the way to go. Storing multiple copies of data may even be an advantage when it comes to reliability.
The easiest way to do cloud-based storage, is probably to just sign up for a Dropbox account, and use that. For me it would be $10/month, which is quite reasonable, but there are a few reasons why I am reluctant to put all my data on Dropbox, and decided to build my own cloud instead –
- What if someone hacks into Dropbox, and deletes everything? Dropbox uses Amazon S3 for backend storage, which has very high reliability (Amazon says S3 standard redundancy storage can sustain simultaneous data loss in 2 data centers, and multiple copies of data is stored at each data center), but backup systems usually do not guard against intentional sabotaging, and if someone gets Dropbox’s access keys (eg. a disgruntled ex-employee?), it’s very easy to wipe out their entire database, including all backups on Amazon (and judging by the amount of data they have, I highly doubt they have their own backup). What if my own cloud gets hacked? Well that’s definitely possible, but it would have to be an attack directed specifically at me, and I have far fewer enemies than Dropbox for sure.
- There’s only 30 days history. They charge a lot of money for infinite history (which is reasonable considering the potential for abuse). I can get infinite history for a lot less, because I know I am not going to abuse it.
- Dropbox lies. I believe they have changed their wording since then, but they once said their employees cannot access your data since it’s encrypted and they don’t have the key. Well, turned out that’s not true. They do have the encryption key. Data is stored with AES-256 encryption on their site, and they manage all the keys. That also means any attacker that gets hold of the keys can decrypt all your data.
- Dropbox already has a few very severe security breaches, in their short existence.
- Slow. I have seen performance drop to 100-200KBytes/s syncing big files. That’s very slow.
Ultimately, it’s just not a company I want to deal with, if I have other choices. And it turned out I do.
I also looked into Google Drive, which looks like one of the only 2 worthy competitors to Dropbox, but their Windows client does this weird thing with random file conversions to try to force you to open all your Office files using Google Doc… so that’s out. I want a simple file storage solution. Not something that tries to decide how I should do things and what programs I should use. Also, no Linux client… which is really surprising.
The other one is Microsoft SkyDrive – most of the same problems as Dropbox, and not have a Linux client in addition.
I’m not interested in smaller providers because they get in and get out of the market in hordes. I don’t want to have to migrate my data between providers every couple months (assuming they don’t just disappear with all my data).
So then I looked into building my own solution.
I have a Linode VPS that I have been using for 5-6 years now, and it already runs my mail server and web server, so I’d like the solution to be based on that.
I also evaluated Amazon EC2 (micro instance), and that turned out to be a joke, mostly because of the very aggressive CPU throttling and weird “bursting” system, but also pathetic disk IO performance. A micro-instance and a ~50GB EBS (elastic block storage, the Amazon term for “harddrive”) would cost about as much as Linode, and a Linode is much more powerful.
Linode hosts are 8-core (16-threads) Sandy Bridge 2.6 GHz Xeons shared between several servers, but any server can use all 8-cores if no one else is using them. I’ve never had to fight anyone for CPU. The host has always been idle every time I needed to use the CPU (that’s not guaranteed of course… if you are unlucky you may get a host with someone using Linode for compute, but that’s unlikely, because Linodes aren’t very cost effective for that). BTW, if you have never tried “make -j 10”, you should. It’s a pretty liberating experience. You can compile the Linux kernel in 1 minute 41 seconds. Their network performance is pretty awesome, too. Usually 100mbps up and down, though it’s hard to find other servers that can talk to you that fast.
They also have extremely good support. I’ve opened a few support tickets over the years, and they have always been answered within 1-2 MINUTES, even on Sunday nights, by people who actually know what they are talking about, and no BS.
Highly recommended. They may not be the absolute cheapest, but the support alone is worth it.
However I don’t want my data to actually be hosted on Linode, because –
- Linode storage is @#$@#%ing expensive for some reason. $20/48GB.
- They don’t offer offsite backup. There was another VPS provider a while ago that lost a bunch of customer data because their data center got flooded (literally, by water). Definitely need offsite backup.
Linode doesn’t provide affordable reliable storage (I’m using Amazon’s definition of reliable – simultaneous loss of 2 facilities, or at least 4 storage devices), so it looks like I’d have to use network storage, which is not ideal, but we’ll see how that goes.
I decided to use S3 because the only worthy competitor is Google Storage, and I don’t like it for a variety of reasons. S3 is also much better supported in terms of available 3rd-party tools.
S3 is available in several regions (where the data centers are located), and it turned out which region you choose and where your server is has GREAT impact on what kind of network performance you’ll get.
I did a few benchmarks using Linode trial instances at a few data centers, to all the S3 regions in North America.
I found the best match to be Linode’s Fremont location, coupled with S3 North California region, where I got 19 MBytes/s up (to S3) and 25 MBytes/s down (from S3). It’s definitely worth the time to do this experiment – many other combinations produced <1MBytes/s speeds in upload or download. I asked Linode to transfer my server to their Fremont location. It was painless, and only took an hour and half. I wouldn’t be surprised if Amazon’s S3 server is actually in the same data center as Linode, or at least on the same ISP.
For the actual hosting application, I tried all the popular open source offerings – SparkleShare, Owncloud, and Seafile.
SparkleShare was just too hard to set up and sometimes didn’t detect file changes… it’s too buggy.
Owncloud was very nice and polished with tons of cool features… but with just one fatal flaw – it’s @#@#%ing slow. I don’t know how they came to the conclusion that it’s a good idea to build a cloud storage platform ENTIRELY based on PHP. Obviously that didn’t work out very well. I have many small files, and it takes about 1 second per file. I believe it just does 1 WebDAV upload request per file with no threading. Ugh. I would get 10-20 Kbytes/s from my house (10 MBps upload). I also ran into a bug when I paused syncing to play a game of Age of Empires – it decided it just doesn’t want to start again afterwards.
Seafile, while doesn’t have a fancy interface, is by far the most functional. The frontend may not be very pretty, but it’s definitely good enough, and I haven’t been able to find anything wrong with the backend, and I have been doing some pretty weird stuff to test it out. It’s also blazingly fast. I get an average of about 700-800 KBytes/s uploading a bunch of small files, which is pretty close to my theoretical maximum. I also talked to the developers about a few things (not problems with the program), and they were very responsive and helpful, this is in contrast to Owncloud where most bug reports show no sign of having ever been looked at by a developer.
Seafile has a closed-source version that is free for personal use and supports S3 as storage backend directly, but I decided against using that, and decided to use S3QL instead, because that would allow me to also put random things on there for backup (home directories, mail, mysql dump, etc), and keep things simple. One of the developers actually suggested it for people that don’t want to use the proprietary version. They say it’s slower, but it still doesn’t bottleneck my upload or download speed (still at least couple MBytes/s), so I’m not worried about that. I did disable S3QL’s encryption, and have it use zlib instead of lzma for compression (lzma would definitely limit S3QL throughput on faster networks).
One of the many cool things S3QL does is data de-duplication. It computes SHA-256 hash of all blocks, and will only store identical blocks once. That makes backing up stuff extremely simple. I just have to keep copying stuff in, and if they haven’t been changed since the last backup, they won’t take up space on S3.
Since Seafile keeps all file revisions already, by now the biggest risk of data loss is application error corrupting my data. It can happen, for example, if there is a bug in Seafile, or S3QL. S3 has a very nice way of protecting against that – versioning. I turned on versioning on my S3 bucket for S3QL, which means if data corruption happens, I can always just revert the entire bucket to an earlier time, and since S3QL stores all its FS states in the bucket, the filesystem will be as good as it was at the time I reverted the bucket to. Since the S3 North California data center provides immediate consistency, I don’t have to worry about weird problems with eventual consistency. Objects in versioned buckets cannot be easily permanently deleted. A special command needs to be sent to delete them, so there is no chance of that happening accidentally.
That is my entire private cloud storage stack. Let’s see what happens.
My biggest risks of losing data now –
- I forget to pay Linode or Amazon, and got my accounts terminated. This is the biggest risk, but there’s no way to work around that besides storing all the data myself.
- Intentional malicious attack, which is unlikely because that person would have to target me specifically, and be able to tweak Seafile in a way that all my computers would sync to an empty directory instead of stop syncing, and also clear my S3 versioned bucket… that’s a lot of work. I’ll just try to not make too many very technically savvy enemies with a lot of UNIX experience.
- I delete my own data in my sleep. Unlikely, because I don’t even KNOW what the command is to delete versioned objects. It’s unlikely that I’ll be able to figure it out in my sleep.
If I am even more paranoid, I can enable 2-factor authentication for permanent versioned object deletion on S3, but I think I’m ok with not having that right now. I’ll probably end up losing the authentication device, and not be able to delete that bucket… forever.
Note that high availability is not very important to me, that’s why I don’t have things like automatic-fallback, etc. System design would have to be a lot more complicated if high availability is also a requirement. I don’t generally mind if I can’t sync for a day or 2, as long as the data is safe. For example, if Linode goes down, I will just set up another VPS or a dedicated server somewhere, and set up S3QL to mount the S3 filesystem, and Seafile to continue serving my data.
Summary of my current setup:
{home directory backup, emails backup, Seafile} -> S3QL -> Amazon S3
Amazon S3 is where everything goes in the end, and Amazon is responsible for data reliability against hardware failures and natural disasters. I am responsible for data reliability against application errors in Seafile and S3QL, and I’m combating that using S3 versioning. I am also responsible for, for example, a virus or “rm -rf” on a sync-ed computer, and I’m combating that using Seafile’s infinite revision history (in addition to S3 versioning, which shouldn’t be used unless Seafile corrupts its own database).
In the extremely unlikely event that S3 data is lost (eg. due to forgetting to pay Amazon), I’ll still have recent copies of my data on my sync-ed computers, and master copy of home directory and emails on my Linode server.
Cost: $4.75/month for 50GB ($20/month for Linode, but I needed that anyways, for other stuff).