Backup & Sync Local and Remote Directories Using RSYNC

There are many backup methods that had been brought on Cloud Raya’s Knowledge Base. In this tutorial, we will bring another backup & synchronization topic. Here, we will learn how to synchronize directories and certain files, from our main VM to another. So, it is similar to deploying DRC from our main VM. 

The method we will bring in this tutorial is by using RSYNC. What is RSYNC, the command line, and the example of its application? Let’s find out. 

What is RSYNC?

RSYNC is one of the powerful synchronization tools but quite easy to use. RSYNC runs on devices with Linux Based OS (Ubuntu, CentOS, Debian, etc.). 

By using RSYNC, we can synchronize any file or folder from one device (source) to one or more other devices (destination). 

The file transfer method on RSYNC can be through an SSH (secure shell) or RSH (remote shell) connection. Or even directly via the RSYNC daemon, so there will be an open RSYNC port on our device that will talk directly to an RSYNC client on another host.

However, in this tutorial, we will talk about how to use RSYNC from SSH connection, so that it will be more secure and most importantly, encrypted. 

One of the capabilities of RSYNC is to synchronize which files differ between source and destination. (incremental backup). With this, we don’t need to always copy files in full in every synchronization process. This of course can save more bandwidth and your network, especially if you want to synchronize 1 device and another over the public network.

With RSYNC, we can also synchronize files and folders across OS, as long as they are still in one linux distribution.

RSYNC Command Format

Here is command structure on RSYNC usage:  

rsync [optional modifiers] [source folder location] [destination folder location]

---

There are optional modifiers that function to determine the action of our command. A group of main options from RSYNC includes:

Those are some of the common RSYNC options that we can use. We can choose based on our needs and goals. But, in my personal opinion what essential to our need is -a -v and –progress, so that the files and folders that are synchronized are exactly the same as the source destination, we can check which files will be processed and also see the transfer rate.

Installing RSYNC

RSYNC is generally installed as a default feature on devices with Linux-based OS. However, it is possible that your device doesn’t have it yet. 

If so, you can install RSYNC by running this command: 

apt-get install rsync

In this tutorial, we are going to use 2 VMs from Cloud Raya with Ubuntu as the OS. Don’t worry for you who use OS from other Linux distributions, the operational method will be similar. If there’s a minor difference, it would be just on the RSYNC installation process.

Both of the VMs will be in 2 different regions, Jakarta and Surabaya, to get better experience for the real DRC conditions. 

Implementation

After we discover the command format and how to install RSYNC, let’s work try it and make a sample test.

▶️ Performing Local Backups

As the first test, let’s try to synchronize one directory with another, but still in the same VM. 

Local Host – Source Dir ➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖➖ Files and folder list: RSYNC SYNTAX root@local-host:/usr/trial# rsync -avp source-dir/ destination-dir sending incremental file list ./ file1.txt file2.txt file3.txt file4.txt file5.txt new/ new/testing-file.txt presentation/ presentation/CR-SalesDeck.pdf presentation/CR-presentation.pptx subfolder/ subfolder/file10.txt subfolder/file6.txt subfolder/file7.txt subfolder/file8.txt subfolder/file9.txt sent 5,891,441 bytes received 290 bytes 3,927,820.67 bytes/sec total size is 5,888,936 speedup is 1.00

↳ To synchronize all files and subfolders from the source destination, we can use the command above. In this tutorial, I will use additional options -v, -p to check what files will be synchronizedd, and how’s the progress going. 

You may notice that there is a trailing slash (/) at the end of the first argument in the syntax, that this (/) specifies the contents of source dir. 

Without the trailing slash, source dir, including the directory, would be placed within the destination dir. The outcome would create a hierarchy like the following:

---

You can adjust it based on your project’s needs and goals. 

▶️ Performing Backup to Remote System

Now we can jump to one of the main capabilities of RSYNC, which is doing a backup and synchronization within the local devices towards our other devices (remote system). 

↳ To sync with a remote system by using RSYNC, you just need SSH access configured between your local and remote machines, as well as RSYNC installed on both systems. After you have the SSH access verified between the two machines, you can sync the source-dir folder to a destination-dir folder that is on the remote machine.

---

To avoid entering passwords repeatedly while the SSH access between the local and remote is processing, you can configure the SSH key within the two devices.  

For those of you who may not know how to connect 2 systems using the ssh key, we will talk about it in a different article.  

---

Besides synchronizing the local device to the remote device, we can do the reverse activity as well by synchronizing the remote device with our local device. For the syntax, we can simply reverse it.  

▶️ Using Exclude Options

RSYNC also has the ability to filter which files or folders we want to exclude in the synchronization process. Yes, we can use the –exclude option. 

↳ So, in the sample, we will exclude the type file .txt from the synchronization process.

---

If you want to add another exclude condition, You can use the following array:  

↳ So, in the above sample we will exclude the type file .txt, .jpg, folder presentation, and what’s inside the new folder from the synchronization process.  

▶️ Using Exclude & Include Options

Besides exclude, we also able to filter with 2 direct conditions in the synchronization, include certain condition, and exclude other condition. And yes, we can combine argument –exclude and –include into 1 command. 

If we see the output, the testing-file.txt is still synchronized even though they include in the condition of exclude *.txt dan /new/*. The subfolder is still synchronized even though what’s inside is not synchronized because it’s containing .txt file. 

So, to use this option, make sure that you put the condition that you want to run first. Then, other conditions will be put behind it. (Include some, exclude the rest) 

▶️ Multiple Destinations Synchronization

Because the natural method of RSYNC is to work on point-to-point synchronization, so there is no default command from RSYNC to conduct multiple destination synchronization.  

However, we can do it with manual technique. One of the way to do it is by use the help from the command ‘for’ loop.  

This is how to write Syntax Structure ‘for’ loop in providing RSYNC multiple destination: 

And for the sample implementation can be done as followed: 

Common RSYNC Usage in Real Life Condition

The RSYNC usage in the real-life condition is unlimited, it includes these activities:  

• Backup and DRC for the website/web application asset 
• Backup to disk or usb stick
• Full server migration from 1 VM to another VM. Yes, it is a great potential from RSYNC that is very possible to implement. For those who need help with this migration level, our support team can help you from our Manage Service. Feel free to schedule consultation with us. 
• And many others. You can determine the conditions yourself for activities related to synchronization and backup on Linux based OS.

Conclusion

Now we know how to backup and synchronize local & remote directories with RSYNC along with RSYNC’s benefits and usage. Hopefully, you find this article helpful.  

On my next article, I will continue this series of article which is discussing about the other synchronization tutorial of 2 VMs. My tutorial final goal will be automatically synchronizing our main VM with our backup VM. Also, we will be discussing about dividing our 2 VMs load equally. So, if if any problem happen to one of our VM, the traffic will distributed automatically to our backup VM. With that, our application will be appeared to be always online seamlessly.

Visit our knowledge base page for more tutorial article. Or, if you don’t have Cloud Raya’s VM, You can join us by free register yourself here.