FileSync for Different Device


There are many file synchronization software available. It compares source with target and copy only the changed files. Save time for a full copy.

You may choose online storage like Copy.com and Dropbox. They both have version for undelete. SkyDrive seems only have recycle bin without version. Box.com provides Version History in paid subscription. Unfortunately, Neither SkyDrive nor Box.com have official Linux client.

But for private data, huge and archived files, I will backup to external storage or NAS. Here I am going to explain my preferred software for different devices.

FreeFileSync for Internal or External Device

FreeFileSync screen capture
FreeFileSync screen capture

FreeFileSync supports Windows, Mac OSX, and Linux (Ubuntu, Debian. and OpenSuSE). It is also portable and may be install onto USB flash disk or external hard drive. You may download from FossHub. The interface is clean and conscious: [Compare] then [Synchronize].

After comparison, it will generate a hidden file .sync.ffs_db to detect changes between sync in [Two way] mode.

In my test for syncing a source folder containing 11,839 files for 2.2GB to another folder with 5,839 for 1.1GB. It find out the different 6,000 files and synchronize without any problem. Much better than my previous experience with Always Sync 8.2.5 for U3, 12.2.3, and SyncBack on Windows XP. norems reports a successful comparison of 13TB data with over 21 million files.

LFTP for NAS with FTP Server

lftp in terminal
lftp in terminal

Most NAS comes with ftp service. You need a ftp client tools to upload or download from ftp server. You may use apt-get install lftp in Debian and Ubuntu, or [YaST]→[Software Management] in OpenSuSE.

lftp is a command line ftp client with advanced features. It support mirror for upload or download but not sync in one command. It is fast and compatible with UTF8 which means your Traditional Chinese, Japanese, and etc file name won’t become question marks.

Here are some commands I use frequently:

  • -u: user name and password
  • -p: ftp server port number and site with file access method like ftp, ftps, http, https, hftp, fish, sftp, and file.
  • mirror -c: continue a mirror job if possible
  • mirror -e: delete target files if not present  in source
  • mirror -R: upload
  • bye: exit lftp interactive command mode

For upload and sync syntax:

lftp:~>open -u [user name],[password] -p [port number] [protocol:site]
lftp:~>mirror -c -e -R [/local/source] [/remote/target]

Sample code to upload to sftp server on 192.168.0.104 using port 22, delete target files if not present  in source, as root:

$lftp
lftp:~>open -u root,mypassword -p 22 sftp://192.168.0.104
lftp:~>mirror -c -e -R /home/amigo/test /home
lftp:~>bye

For download and sync syntax:

lftp:~>open -u [user name],[password] -p [port number] [protocol:site]
lftp:~>mirror -c -e [/remote/target] [/local/source]

Sample code to download from sftp server on 192.168.0.104 using port 22, delete target files if not present  in source, as root:

$lftp
lftp:~>open -u root,mypassword -p 22 sftp://192.168.0.104
lftp:~>mirror -c -e /home /home/amigo/test
lftp:~>bye

You may save the open and mirror command as a text file and use $lftp -f command.txt to automatically them.

rsync for NAS with rsync Server

rsync in terminal
rsync in terminal

Most advanced NAS, eg. Synology and QNAP, comes with rsync server. It is widely used for backup among Linux and NAS. Most Linux have rsync build-in in standard setup configuration. You may use apt-get install rsync in Debian and Ubuntu, or [YaST]→[Software Management] in OpenSuSE.

rsync is a command line tool to communicate with rsync server running rsync daemon. It will compare remote and local folder and sync in one command. It is fast and compatible with UTF8 which means your Traditional Chinese, Japanese, and etc file name won’t become question marks.

Here are some commands I use frequently:

  • –checksum: force a full checksum comparison to prevent skipping files with the same time and size but different content.
  • –del or –delete: delete files which are removed in local in remote directory
  • -a: transfer in archive mode which will keep all file attributes.
  • -v: show all message
  • -z: compress during transfer

For sync without delete syntax:

$rsync -avz [/local/source] [user name]@[site]:[remote/target]

Sample code to rsync server on 192.168.0.104 in archive mode, show all message, and use compression, as root:

rsync -avz /home/amigo/test root@192.168.0.104:/home
root@192.168.0.104's password: 
sending incremental file list

sent 223 bytes  received 14 bytes  52.67 bytes/sec
total size is 1292843314  speedup is 5455035.08

For sync with delete syntax:

$rsync -avz --delete [/local/source] [user name]@[site]:[remote/target]

Sample code to rsync server on 192.168.0.104 in archive mode, show all message, and use compression, as root:

rsync -avz --delete /home/amigo/test root@192.168.0.104:/home
sending incremental file list
deleting test/turnkey-zurmo-13.0-wheezy-amd64-ovf.zip
deleting test/TurnKey ZurmoCRM 13.0/turnkey-zurmo-13.0-wheezy-amd64-disk1.vmdk

sent 151 bytes  received 14 bytes  36.67 bytes/sec
total size is 14642  speedup is 88.74

You may find more information with man rsync for the manual and detail explain.

Reference

  1. Copy.com
  2. Dropbox
  3. Paolo Brocco Works: Copy vs. Dropbox: Features comparison
  4. Microsoft: SkyDrive
  5. Box.com: Plan & Pricing
  6. GitHub: noiselabs / box-linux-sync
  7. Wiki: Comparison of file synchronization software
  8. FreeFileSync
  9. FossHub: FreeFileSync
  10. Always Sync
  11. 2BrightSparks: SyncBack
  12. FreeFileSync: Any Limitation on Freefilesync
  13. lftp
  14. Wiki: lftp
  15. Debian
  16. Ubuntu
  17. OpenSuSE
  18. Linux.com: Using lftp to synchronize folders with a FTP account
  19. Synology
  20. QNAP
  21. Samba.org: rsync
  22. Wiki: rsync
  23. Linuxawy.org: howto setup an rsync server (debian/ubuntu)
  24. rsync: Frequently Asked Questions
  25. 中央研究院計算中心:資料備份同步工具簡介— rsync
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s