GlusterFS / Async Writes
Over the weekend i decided to experiment and setup a glusterfs between my root server(plexus) and my fileserver at home(hydra).
I won’t describe how to setup glustefs. There are various good guides "available":http://www.gluster.org/community/documentation/index.php/QuickStart.
gluster volume create gv0 replica 2 hydra:/opt/gfs/brick1/gv0 plexus:/gfs/brick1/gv0 gluster volume set gv0 network.ping-timeout 5 gluster volume start gv0 root@plexus:/gfs# mount -t glusterfs localhost:/gv0 gv0/ root@hydra:/gfs# mount -t glusterfs localhost:/gv0 gv0/
At first glance i hoped that small writes would be cached and asynchronously flushed to the filesystem. I tried to increase the performance.cache-size and performance.write-behind-window-size option but did not succeed in increasing the write performance.
As you can see the write performance is basically predetermined by my max upload speed.
# Write 1MB root@hydra:/gfs/gv0# dd if=/dev/zero of=testfile bs=1K count=1024 1024+0 records in 1024+0 records out 1048576 bytes (1.0 MB) copied, 3.36612 s, 312 kB/s
But luckily there are branching filesystems like "unionfs":http://en.wikipedia.org/wiki/UnionFS that is for example used for live CDs. Such filesystems work in that way that they overlay multiple filesystems into a single virtual one.
I created a new gv0-writecache folder which can be written to. The follow command creates a unionfs filesystem that is mounted to the /gv0/union folder.
mkdir -p /gfs/union unionfs-fuse -o default_permissions -o allow_other /gfs/gv0-writecache=RW:/gfs/gv0=RO /gfs/union/
Succeeding writes to the union folder are now very fast and the files are stored within the /gfs/gv0-writecache folder.
root@hydra:/gfs/union# dd if=/dev/zero of=test2 bs=1M count=10 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0454509 s, 231 MB/s
After that step we only need to make sure to sync those files from the writecache to the glusterfs mount.
root@hydra:/gfs/gv0-writecache# rsync --remove-source-files -av /gfs/gv0-writecache/* /gfs/gv0
I also tried to use aufs instead of unionfs-fuse but i got various strange errors.