10+ lsyncd examples to sync directories real time in CentOS/RHEL 7

In this example I will share some examples with lsyncd to perform real time sync in Linux environment. With lsyncd we can sync directories and files across multiple targets and directories in Linux. I am using RHEL/CentOS 7 as the source and RHEL/CentOS 8 as target node to perform real time sync of directories and files.

Overview on lsyncd

• Lsyncd uses a file system event interface (inotify or fsevents) to watch for changes to local files and directories.
• Lsyncd collates these events for several seconds and then spawns one or more processes to synchronize the changes to a remote file system.
• The default synchronization method is rsync. Thus, Lsyncd is a light-weight live mirror solution.
• Lsyncd is comparatively easy to install and does not require new file systems or block devices.
• Lysncd does not hamper local file system performance.

Why use lsyncd?

Lsyncd is designed to synchronize a slowly changing local directory tree to a remote mirror and is especially useful to sync data from a secure area to a not-so-secure area.

Install lsyncd

lsyncd is not part of default repository of RHEL/CentOS 7, hence you must install this using EPEL repo. You can install epel-repo using

[root@centos-7 ~]# yum -y install epel-release

Next you can install lsyncd rpm on the source machine. Here our source machine is RHEL/CentOS 7 Linux node running on Oracle VirtualBox

[root@centos-7 ~]# yum -y install lsyncd

1. One liner lsyncd examples to real time sync directories and files

To print version number

[root@centos-7 ~]# lsyncd --version
Version: 2.2.2

To sync directories on localhost using rsync with lsyncd

[root@centos-7 ~]# lsyncd -rsync /source/ /target/
20:35:33 Normal: --- Startup, daemonizing ---

Here the process went in the background and is working as a daemon, you can also use lsyncd frontend

[root@centos-7 ~]# lsyncd -nodaemon -rsync /source/ /target/
20:36:36 Normal: --- Startup ---
20:36:36 Normal: recursive startup rsync: /source/ -> /target/
20:36:36 Normal: Startup of /source/ -> /target/ finished.

20:37:21 Normal: Calling rsync with filter-list of new/modified files/dirs
/file3
/
20:37:21 Normal: Finished a list after exitcode: 0
^C20:38:30 Normal: --- INT signal, fading ---

There is a difference in behaviour when running with -nodaemon. Lsyncd will not change its working directory to /, as it does when becoming a daemon.

To sync files and directories across hosts using rsync in the backend

[root@centos-7 ~]# lsyncd -nodaemon -rsync /source/ centos-8:/target/
20:39:04 Normal: --- Startup ---
20:39:04 Normal: recursive startup rsync: /source/ -> centos-8:/target/
20:39:04 Normal: Startup of /source/ -> centos-8:/target/ finished.

To sync files and directories across hosts using rsyncssh as backend

[root@centos-7 ~]# lsyncd -nodaemon -rsyncssh /source centos-8 /target/
20:40:48 Normal: --- Startup ---
20:40:48 Normal: recursive startup rsync: /source/ -> centos-8:/target/
20:40:49 Normal: Startup of "/source/" finished: 0
^C20:40:52 Normal: --- INT signal, fading ---

2. Configure lsyncd config file for default settings

The default lsyncd config file is /etc/lsyncd.conf. But you can also find a bunch of examples lsyncd config file under /usr/share/doc/lsyncd*/examples. I am using a very basic lsyncd config in my setup

For example in the below lsyncd config file,

• the following code will instruct Lsyncd to log into /var/log/lsyncd/lsyncd.log,
• periodically update the file /var/log/lsyncd-status.log with its status
• write the status file at shortest after 20 seconds has passed
• and to not detach as a daemon.

settings {
   logfile    = "/var/log/lsyncd/lsyncd.log",
   statusFile = "/var/log/lsyncd-status.log",
   statusInterval = 20,
   nodaemon   = true
}

For more details of supported directives check official page of lysyncd

3. Sync local directories and files using default.direct

default.direct can be used to keep two local directories in sync with better performance than using default.rsync. Default.direct uses (just like default.rsync) rsync on startup to initially synchronize the target directory with the source directory.

However, during normal operation default.direct uses /bin/cp, /bin/rm and /bin/mv to sync directories.

Below is a sample lsyncd configuration file /etc/lsyncd.conf using defualt.direct

settings {
   logfile    = "/var/log/lsyncd/lsyncd.log",
   statusFile = "/var/log/lsyncd/lsyncd-status.log",
   statusInterval = 20,
   nodaemon   = true,
}

sync {
    default.direct,
    source = "/source",
    target = "/target1",
    delay = 10,
}

Next restart your lsyncd service

[root@centos-7 ~]# systemctl restart lsyncd

Check the status of the service and logs for any error output

[root@centos-7 ~]# systemctl status lsyncd
[root@centos-7 ~]# less /var/log/lsyncd/lsyncd.log

4. Sync local directories and files using default.rsync

The default rsync configuration will aggregate events up to delay seconds or 1000 separate uncollapsible events, which ever happens first. Then it will spawn one rsync with a filter of all files that changed. The filter list is transmitted to rsync trough a pipe.

Below is a sample lsyncd config file using defualt.rsync to sync directories and files:

[root@centos-7 ~]# cat /etc/lsyncd.conf

settings {
settings {
   logfile    = "/var/log/lsyncd/lsyncd.log",
   statusFile = "/var/log/lsyncd/lsyncd-status.log",
   statusInterval = 20,
   nodaemon   = true,
}

sync {
   default.rsync,
   source = "/source",
   target = "/target",
   delay  = 10,
}

Here we have defined a delay wait period of 10 seconds to sync directories. For more list of supported directive check official github page of lsyncd

Verify the directory syncing on localhost. Here we have two directories on localhost

[root@centos-7 ~]# mkdir /source /target

So we have configured lsyncd to sync content of /source into /target directory on the same node.

[root@centos-7 ~]# touch /source/file

After few seconds verify the same content in /target

[root@centos-7 ~]# ls /target/
file

From log file /var/log/lsyncd/lsyncd.log

Sun Jan 19 16:24:21 2020 Normal: --- Startup ---
Sun Jan 19 16:24:21 2020 Normal: recursive startup rsync: /source/ -> /target/
Sun Jan 19 16:24:21 2020 Normal: Startup of /source/ -> /target/ finished.

If the files are modified the same is updated in /target

Sun Jan 19 16:24:35 2020 Normal: Calling rsync with filter-list of new/modified files/dirs
/file
/
Sun Jan 19 16:24:35 2020 Normal: Finished a list after exitcode: 0

5. Sync files and directories from source on localhost to another target host using rsync

We can use the existing configuration file with a little change to sync directories across multiple Linux hosts. I am using CentOS 7 as my source node and CentOS 8 as my destination. Here the target should contain the hostname and the absolute target path

Below is a sample lsyncd config lsyncd.conf file. You can either configure a DNS server to resolve hostname or update /etc/hosts file.

[root@centos-7 ~]# cat /etc/lsyncd.conf
settings {
   logfile    = "/var/log/lsyncd/lsyncd.log",
   statusFile = "/var/log/lsyncd/lsyncd-status.log",
   statusInterval = 20,
   nodaemon   = true,
}

sync {
   default.rsync,
   source = "/source",
   target = "centos-8:/target",
   delay  = 10,
}

Next restart your lsyncd service

[root@centos-7 ~]# systemctl restart lsyncd

Check the status of the service and logs for any error output

[root@centos-7 ~]# systemctl status lsyncd

When you restart the service below logs would be seen in /var/log/lsyncd/lsyncd.log to initiate the directory syncing

Sun Jan 19 16:19:36 2020 Normal: --- TERM signal, fading ---
Sun Jan 19 16:19:36 2020 Normal: --- Startup ---
Sun Jan 19 16:19:36 2020 Normal: recursive startup rsync: /source/ -> centos-8:/target/
Sun Jan 19 16:19:36 2020 Normal: Startup of /source/ -> centos-8:/target/ finished.

Every time a file/dir is modified based on delay value the directories between hosts will automatically get synced real time:

Sun Jan 19 16:20:09 2020 Normal: Calling rsync with filter-list of new/modified files/dirs
/file
/
Sun Jan 19 16:20:09 2020 Normal: Finished a list after exitcode: 0

6. Sync directories and files from source on localhost to different target host using rsyncssh

This configuration differs from the standard rsync configuration in that it uses ssh commands to move files or directories locally at the target host instead of deleting and transferring again. This configuration does spawn rsync processes like default.rsync but additionally will spawn /usr/bin/ssh HOST mv ORIGIN DESTINATION commands.

Different to default.rsync it does not take an uniform target parameter, but needs host and targetdir separated.

Here is a sample lsyncd config file, we use default.rsyncssh under sync section instead of default.rsync

[root@centos-7 ~]# cat /etc/lsyncd.conf
settings {
   logfile    = "/var/log/lsyncd/lsyncd.log",
   statusFile = "/var/log/lsyncd/lsyncd-status.log",
   statusInterval = 20,
   nodaemon   = true,
}

sync {
   default.rsyncssh,
   source="/source",
   host="centos-8",
   excludeFrom="/etc/lsyncd.exclude",
   targetdir="/target",
   delay = 10,
}

7. Enable compress and archive with lsyncd for directory mirroring

You can enable compression and archiving while attempting the file transfer to sync directories by using the highlighted section in your lsyncd config file

[root@centos-7 ~]# cat /etc/lsyncd.conf
settings {
   logfile    = "/var/log/lsyncd/lsyncd.log",
   statusFile = "/var/log/lsyncd/lsyncd-status.log",
   statusInterval = 20,
   nodaemon   = true,
}

sync {
   default.rsync,
   source = "/source",
   target = "/target",
   delay  = 10,
   rsync  = {
     archive  = true,
     compress = true
   }
}

Next restart your lsyncd service

[root@centos-7 ~]# systemctl restart lsyncd

Check the status of the service and logs for any error output

[root@centos-7 ~]# systemctl status lsyncd

8. lsyncd multiple targets and directories from one source

There are two methods to perform lsyncd multiple target in Linux, either by creating separate sync functions or by using for loop as shown below:

Method 1: lsyncd multiple targets by creating separate sync block

You can create separate sync block inside lsyncd config file /etc/lsyncd.conf for respective source and target location for lsyncd multiple targets. For examples below I have two sync functions to sync directories and files locally at two different targets while the third sync function will copy files and directories between localhost and target CentOS 8 node.

[root@centos-7 ~]# vim /etc/lsyncd.conf

settings {
   logfile    = "/var/log/lsyncd/lsyncd.log",
   statusFile = "/var/log/lsyncd/lsyncd-status.log",
   statusInterval = 20,
   nodaemon   = true,
}

sync {
   default.rsync,
   source = "/source",
   target = "/target1",
   delay  = 10,
}

sync {
   default.rsync,
   source = "/source",
   target = "centos-8:/target",
   delay  = 10,
}

sync {
   default.rsync,
   source = "/source",
   target = "/target2",
   delay  = 10,
}

Method 2: lsyncd multiple targets by using for loop

Alternatively for lsyncd multiple targets, instead of creating so many sync functions you can also use for loop to sync directories and files between one source and multiple targets

settings {
   logfile    = "/var/log/lsyncd/lsyncd.log",
   statusFile = "/var/log/lsyncd/lsyncd-status.log",
   statusInterval = 20,
   nodaemon   = true,
}

targets = {
    'centos-8:/targetdir',
    '/target1',
    '/target2',
}

for _, target in ipairs( targets )
do
    sync{ default.rsync, source='/source', target=target }
end

Now after you choose method 1 or 2 for your setup to perform lsync multiple targets, restart your lsyncd service to activate the changes

[root@centos-7 ~]# systemctl restart lsyncd

Check the status of the service and logs for any error output

[root@centos-7 ~]# systemctl status lsyncd

You can verify the the logs /var/log/lsyncd/lsyncd.log to make sure lsyncd multiple targets is working as expected

Sun Jan 19 17:14:17 2020 Normal: Startup of /source/ -> /target1/ finished.
Sun Jan 19 17:14:17 2020 Normal: Startup of /source/ -> /target2/ finished.
Sun Jan 19 17:14:56 2020 Normal: Startup of /source/ -> centos-8:/target/ finished.

9. Execute script before or after rsync operation

We can use binary directive with the absolute path of the script inside sync function to execute a script at post stage. Below is a sample lsyncd config file /etc/lsyncd.conf

settings {
   logfile    = "/var/log/lsyncd/lsyncd.log",
   statusFile = "/var/log/lsyncd/lsyncd-status.log",
   statusInterval = 20,
   nodaemon   = true,
}

sync {
   default.rsync,
   source = "/source",
   target = "/target1",
   delay  = 10,
   rsync = {
        binary = "/tmp/target1_status.sh"
   }
}

Here we have created a sample script which will echo a statement "Completed sync in target1" in a temporary file /tmp/file once the syncing is complete between provided source and target. This can be used to check exit status of directory syncing.

[root@centos-7 ~]# cat /tmp/target1_status.sh
#!/bin/bash
/usr/bin/rsync "$@"
result=$?
(
  if [ $result -eq 0 ]; then
     echo "`date`: Completed sync in target1" >> /tmp/file
  fi
) >/dev/null 2>/dev/null </dev/null

exit $result

Next restart your lsyncd service

[root@centos-7 ~]# systemctl restart lsyncd

Check the status of the service and logs for any error output

[root@centos-7 ~]# systemctl status lsyncd

Verify the status of the execution from the logs /var/log/lsyncd/lsyncd.log

Sun Jan 19 17:49:40 2020 Normal: Startup of /source/ -> /target1/ finished.

10. lsyncd exclude files and directories while performing real time sync

To exclude files and directories from getting synced by lsyncd, two additional parameters can be specified to sync{}:

• excludeFrom = FILENAME - loads exclusion rules from this file, on rule per line
• exclude = LIST - loads exclusion rules from this list of strings

Exclusion rules are modelled after rsync’s exclusion patterns but are a bit simpler. lsyncd supports these features:

Generally if any segment of the pathname of an event matches the text, it is excluded. E.g. the file “/bin/foo/bar” matches the rule “foo”.

• If the rule starts with a slash, it will only be matched at the beginning of the pathname
• If the rule ends with a slash, it will only be matched at the end of a pathname
• ? matches any character that is not a slash.
• * matches zero or more characters that are not a slash
• ** matches zero or more characters, this can be slashes.

In the below example I am excluding any file or directory having "file" in the filename or ending with "tmp"

[root@centos-7 ~]# vim /etc/lsyncd.conf
settings {
   logfile    = "/var/log/lsyncd/lsyncd.log",
   statusFile = "/var/log/lsyncd/lsyncd-status.log",
   statusInterval = 20,
   nodaemon   = true,
}

sync {
    default.direct,
    source = "/source",
    target = "/target1",
    delay = 10,
    exclude = { 'file*' , '*tmp' }
}

Next restart your lsyncd service

[root@centos-7 ~]# systemctl restart lsyncd

Check the status of the service and logs for any error output

[root@centos-7 ~]# systemctl status lsyncd

From the logs /var/log/lsyncd/lsyncd.log you can verify the exclusion

Sun Jan 19 18:28:43 2020 Normal: --- Startup ---
Sun Jan 19 18:28:43 2020 Normal: recursive startup rsync: /source/ -> /target1/ excluding
file*
*tmp
Sun Jan 19 18:28:43 2020 Normal: Startup of /source/ -> /target1/ finished.

11. lsyncd delete files while syncing directories

By default Lsyncd will delete files on the target that are not present at the source since this is a fundamental part of the idea of keeping the target in sync with the source. However, many users requested exceptions for this, for various reasons, so all default implementations take delete as an additional parameter.

Valid values for delete are:

• delete = true Default. Lsyncd will delete on the target whatever is not in the source. At startup and what's being deleted during normal operation.
• delete = false Lsyncd will not delete any files on the target. Not on startup nor on normal operation. (Overwrites are possible though)
• delete = 'startup' Lsyncd will delete files on the target when it starts up but not on normal operation.
• delete = 'running' Lsyncd will not delete files on the target when it starts up but will delete those that are removed during normal operation.

For more information check official page of lsyncd

sync {
    default.direct,
    source = "/source",
    target = "/target1",
    delay = 10,
    delete = false
}

Lastly I hope the steps from the article to sync directories and files using lsyncd config on RHEL/CentOS 7/8 Linux was helpful. So, let me know your suggestions and feedback using the comment section.