GlusterFS / Async Writes

Over the weekend i decided to experiment and setup a glusterfs between my root server(plexus) and my fileserver at home(hydra).

I won’t describe how to setup glustefs. There are various good guides "available":http://www.gluster.org/community/documentation/index.php/QuickStart.

  gluster volume create gv0 replica 2 hydra:/opt/gfs/brick1/gv0 plexus:/gfs/brick1/gv0
  gluster volume set gv0 network.ping-timeout 5
  gluster volume start gv0

  root@plexus:/gfs# mount -t glusterfs localhost:/gv0 gv0/
  root@hydra:/gfs# mount -t glusterfs localhost:/gv0 gv0/

At first glance i hoped that small writes would be cached and asynchronously flushed to the filesystem. I tried to increase the performance.cache-size and performance.write-behind-window-size option but did not succeed in increasing the write performance.

As you can see the write performance is basically predetermined by my max upload speed.

  # Write 1MB
  root@hydra:/gfs/gv0# dd if=/dev/zero  of=testfile bs=1K count=1024
  1024+0 records in
  1024+0 records out
  1048576 bytes (1.0 MB) copied, 3.36612 s, 312 kB/s

But luckily there are branching filesystems like "unionfs":http://en.wikipedia.org/wiki/UnionFS that is for example used for live CDs. Such filesystems work in that way that they overlay multiple filesystems into a single virtual one.

I created a new gv0-writecache folder which can be written to. The follow command creates a unionfs filesystem that is mounted to the /gv0/union folder.

  mkdir -p /gfs/union
  unionfs-fuse  -o default_permissions -o allow_other /gfs/gv0-writecache=RW:/gfs/gv0=RO /gfs/union/

Succeeding writes to the union folder are now very fast and the files are stored within the /gfs/gv0-writecache folder.

  root@hydra:/gfs/union# dd if=/dev/zero of=test2 bs=1M count=10
  10+0 records in
  10+0 records out
  10485760 bytes (10 MB) copied, 0.0454509 s, 231 MB/s

After that step we only need to make sure to sync those files from the writecache to the glusterfs mount.

  root@hydra:/gfs/gv0-writecache# rsync  --remove-source-files -av /gfs/gv0-writecache/* /gfs/gv0

I also tried to use aufs instead of unionfs-fuse but i got various strange errors.