What is Rsync and How to Use it for Backups
Currently maintaining an infrastructure of any size without any support can be considered crazy, a disk can fail at any time, and you could lose valuable information such as the accounting of the company. In this post, we will talk about how to use Rsync, one of the best tools for backups in * nix systems.
We will see from the most basic to more advanced examples with scripts and patterns that you can apply in your work environment.
The need to have the same information in different places has been one that is gaining strength; this is because our workspace is variable and we want to have our tools always with us or even as backup measures.
What is rsync?
It is an open-source tool for transferring files and directories between one location and another . Its advantages are mainly based on the compression of the information to be sent, it allows the transfer to be made through an SSH channel and it transfers only the files and pieces of files that have been modified instead of transferring the complete file again, something similar to what happens when transferring the differences in a file under the control of Git versions.
Among the different uses that are given to rsync are the following:
- Automated backups to disks or remote servers.
- Synchronization of files and remote directories.
- Common transfer of files.
Advantages of Rsync
1. It is faster than scp (Secure Copy) because rsync uses a upload protocol that allows only the difference between the files to be transferred, the first time it transfers the whole file for the second time only the changes that may be there.
2. It is designed for remote backups.
3. It supports the copy of all the permission of the files for example, owner, soft groups and hard link.
4. It uses less bandwidth when using compression while sending files.
Learning to use Rsync
We fully enter the management of the command and the most useful options.
Basic Syntax
It consists of the rsync command at the start, the options to be used, the source file and the destination of the backup.
# rsync options source destination
Rsync options
These are some of the most useful options that can be used with Rsync remember that to see all your options we can visit your man page —> man rsync
-v: verbose ---> Show more information or n of the operation or n -r: recursive ---> is used to n the sub - folders in the operation or n -to: file ---> copy as is preserving permissions , symbolic links , users and groups and keeping modification times or n . -z: compress ---> enables compression or n while being transferred -h: human - readable ----> The units are maintained in MB GB for f to easy reading
Install Rsync
It is installed in almost all distributions, but you can still install it from the repo.
#Red Hat/Centos yum install rsync #Debian/Ubuntu/Linux Mint apt-get install rsync
Using Rsync for Local Backups
Sometimes we must support certain information before making modifications and in case of some error recover from the Backup, we will start with a laboratory doing backup from one folder to another.
Preparing the System
We start verifying that rsync is installed.
which rsync /usr/bin/rsync
We will create 2 folders for the origin and destination of the files.
mkdir {backup1, backup2}
We create several files in the backup1 folder in this case no matter the size of the files.
$ touch backup1/file{1..75}
The directory should have 75 files.
ls backup1/ file1 file14 file19 file23 file28 file32 file37 file41 file46 file50 file55 file6 file64 file69 file73 file10 file15 file2 file24 file29 file33 file38 file42 file47 file51 file56 file60 file65 file7 file74 file11 file16 file20 file25 file3 file34 file39 file43 file48 file52 file57 file61 file66 file70 file75 file12 file17 file21 file26 file30 file35 file4 file44 file49 file53 file58 file62 file67 file71 file8 file13 file18 file22 file27 file31 file36 file40 file45 file5 file54 file59 file63 file68 file72 file9
Backup Files and Directories
To back up the files to the backup2 we use:
rsync -r backup1/backup2
The origin must end in / otherwise it would copy its folder and not the content.
If we enter the backup2 we can see the 75 files we can even check that they are 2 identical directories using diff.
diff backup1 backup2
If you create a softlink from a file in directory 1
ln -s file30 file100
When using rsync with -r will not synchronize the softlink in this case we use:
rsync -a backup1/backup2
If we look this time we will have the synchronized softlink.
Using the Dry Run in Rsync
If we are not sure of what we are going to support, we can use a simulation with -and in conjunction with -v it can be very useful.
$ rsync -anv backup1/backup2 sending incremental file list sent 606 bytes received 12 bytes 1236.00 bytes/sec total size is 6 speedup is 0.01 (DRY RUN)
Copy Files to a Remote Server
It is possible to make backups to external servers in this case we must have access by SSH to the destination server remember to activate the compression for much faster transfers. We must use the access address followed by the directory where the backup will be saved.
[[email protected] ~]$ rsync -avz backup1/ [email protected]:/home/user/ The authenticity of host '172.31.17.163 (172.31.17.163)' can't be established. ECDSA key fingerprint is 0d:0c:b1:1d:e1:cf:6d:9f:51:bf:0f:dc:60:82:a1:73. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '172.31.17.163' (ECDSA) to the list of known hosts. [email protected]'s password: sending incremental file list ./ file1 file10 file100 -> file30 file11 file12 file13 file14 file15 file16 …. sent 3312 bytes received 1443 bytes 559.41 bytes/sec total size is 6 speedup is 0.00
It will ask us for the password of the other server if we do not have it with public keys at the end we will have a replica of the backup1 folder in ServerB.
Copy Files from a Remote Server to a Local Server
In case you want to download the files to a local machine we use the same command but changing the order of the origin and destination following the previous example would be like this.
[[email protected] ~]$ rsync -avz [email protected]:/home/user/backup1/ home/backupo1
Backup using SSH for encrypted data
Security is very important in production environments , customer information can not be compromised and we can encrypt data while they are backed up using SSH, another of its benefits is that credentials can be protected in case we do not use public keys. Its disadvantage is that it can consume more system resources and the transfer may not be as fast.
To use a protocol with rsync we use the option -e followed by the protocol to use, in this we will use ssh and the command is like this.
[[email protected] ~]$ rsync -avzhe ssh [email protected]:/home/user/ /home/backup1
Show Progress with Rsync
Making a single backup can take some time you can take a progress from the console using the –progress option I do not recommend using this option with scripts can give you a very extensive log.
[[email protected] ~]$ rsync -avzhe ssh --progress [email protected]:/home/user/ /home/backup1 sending incremental file list created directory /root/rpmpkgs rpmpkgs/ rpmpkgs/httpd-2.2.3-82.el5.centos.i386.rpm 1.02M 100% 2.72MB/s 0:00:00 (xfer#1, to-check=3/5) rpmpkgs/mod_ssl-2.2.3-82.el5.centos.i386.rpm 99.04K 100% 241.19kB/s 0:00:00 (xfer#2, to-check=2/5) rpmpkgs/nagios-3.5.0.tar.gz 1.79M 100% 1.56MB/s 0:00:01 (xfer#3, to-check=1/5) rpmpkgs/nagios-plugins-1.4.16.tar.gz 2.09M 100% 1.47MB/s 0:00:01 (xfer#4, to-check=0/5) sent 4.99M bytes received 92 bytes 475.56K bytes/sec total size is 4.99M speedup is 1.00
Use -include and -exclude with Rsync
It is possible to use access lists with Rsync with the include and exclude options, we just need to create text files and place the names of the files or folders that should be included in the backup or add the list of what should be ignored make the transfer.
We will create a file called include, and another called exclude
[[email protected] ~] $ touch {include, exclude}
We add what we want in each file and then include the file with rsync
[[email protected] ~]$rsync -avze ssh --include --exclude [email protected]:/home/user/backup1/ /home/media/backup2
Delete files with Rsync
By default we can not delete files between the source and destination unless we use the -delete option , when using it if the destination is a file that is not in the source will be deleted so that the 2 directories are equal.
If we create a file in the backup2.
[[email protected] ~]$ touch backup2/solvetic
Then we synchronize from the backup1
rsync -avz --delete backup1/ backup2
We see how the solvent file is removed.
Maximum quota of files to be transferred
We can set a maximum size of the files to be transferred or synchronized, when using the -max-size option. For example if we only want to transfer files no larger than 200 mb we use -max-size ‘200mb', This can be very useful when we support a web but we want to omit its database.
[[email protected] ~]$ rsync -avzhe ssh --max-size'200mb' /var/www/* [email protected]:/root/backupweb
Delete files automatically after being transferred or synchronized
Suppose we have a web server and a backup server, and we have a daily backup, and we do not want to save backup content on the web server. it is not necessary to delete the files manually rsync I can delete them automatically using the option -remove-source-files.
When the transfer is finished, the files will be immediately deleted from the web server.
[[email protected] ~]$ rsync --remove-source-files -zvh backup.tar /tmp/backups/ backup.tar
sent 14.71M bytes received 31 bytes 4.20M bytes/sec total size is 16.18M speedup is 1.10 [[email protected] ~] ll backup.tar ls: backup.tar: No such file or directory
Limit bandwidth when transferring files with rsync
it is common to make the backups in production servers To avoid that the service to the clients cba diminished For a little bandwidth it is good to limit the backup, using the -bwlimit option we will limit the I / O bandwidth.
[[email protected] ~]rsync --bwlimit=100 -avzhe ssh /var/lib/rpm/ [email protected]:/root/tmprpm/ [email protected]'s password: sending incremental file list sent 324 bytes received 12 bytes 61.09 bytes/sec total size is 38.08M speedup is 113347.05
We have limited the backup to 100 kbps of transfer speed.
Conclusions:
We have seen how to use rsync with the most useful options, together with tools such as cron and scripts in bash, python we will have everything necessary to do our backups without the need of third-party tools, in another post I will talk about the types of backups and how to use them with additional rsync add much more complex scripts.
