Real-Time Multi-Client Synchronization with Conflict Management

Question

I need to setup a synchronization between my Computers with a NAS in the local network. The setup is somewhat complicated, though, so I am assuming there is not an off-the-shelf product that does what I need.

Details:

Devices I use:

Desktop PC, dual-boot Windows 10 / Arch Linux
Laptop (Arch Linux)
NAS (self-build, FreeNAS running on it)

I've used rsync so far to backup to the NAS (one-way, my dual-boot setup uses the same storage for both OS).

The problem is, I would like to get the Laptop in the sync. Therefore I need to do some kind of two-way sync.

Problems that arise here:

I often save files only seconds before the shutdown, so I need to monitor file changes instead of running a sync tool periodically
- I cannot sync my laptop outside of the local network (and in the local network, I use the desktop PC most of the times); the local (updated) version could be quite out of date and give some conflicts.
I do want to use the local network only (probably the existing network share of the NAS), not an ownCloud or something similar, because we have quite limited bandwidth where I live and I have some rather huge files to keep synced.

My ideas

I found this question: How do I setup rsync and any other scripts to mimic a minimal viable replica of DropBox?, and looked into a solution using lsyncd (Live Syncing Daemon), which synchronizes local directories with remote targets. This could solve problem #1, but I still need to be prepared for conflicts. Furthermore, I could not find any docs about lsyncd on Windows, so I assume it won't work there.

I also looked at the potential to incorporate Unison into a solution; it seems to do two-way syncs. Maybe I could use it together with lsyncd. I'm not sure about the conflict handling, though. I found a hint that with -batch, conflicts are just skipped. But how do I handle conflicted files in the long term (I want to sync them, after all)?

So my question is, how can I accomplish this?

Accepted Answer

Depending on your NAS - I'm pretty fond of BitTorrent sync for this. It runs on most systems (including raspberry pis and mips systems) with support from quite a few consumer NAS systems (like my seagate). You just pick a folder to share, generate a key, and use the key to set up the share on each system you want it on. Its smart enough to detect you're on a LAN, works outside of it. The free version works well enough for my needs - I mainly use it to sync downloads from a system I keep on all the time and backups of many small files that I add to infrequently.

It'll do what you need - one way or two way sync, using the lan if needed. It deals with conflict by keeping a copy of deleted files, (which I turn off for my use cases) and generally runs well enough that I hardly notice it.

Answer 2

So, my self-developed solution requires a bit more scripting than the one of Journeyman Geek. I used unison for the sync-part and zenity to warn for conflicts.

Must Read

Scripts

The bash script (put this into your startup-applications):

#!/bin/bash
if [ -z "$(arp | grep MAC_ADDRESS_OF_NAS)" ]; then
    zenity --warning --text="NAS not found. Aborting."
fi;
unison sync -batch=true -repeat=watch |& while read -r line; do
    if test -n "$(echo $line | grep skipped:)"; then
        zenity --warning --text="$line"; 
    fi;
done

The unison configuration (put this as sync.prf into ~/.unison/):

root=/home/my-username/the-data-i-want-to-backup/
root=/mnt/NAS/some/sub/path/

# follow all symlinks
follow = Path *

# automatically do everything (if there are no conflicts)
auto = true

# for some reason unison fails if I sync the permissions
perms = 0
dontchmod = true

Real-Time Multi-Client Synchronization with Conflict Management

Details:

My ideas

2 Answers

Must Read

Scripts

James H. Sterling