Hi All,
Hope you all doing good.

I'm facing some issue while syncing data using lsyncd. I'm working on a project to migrate data from a source S3 bucket to target S3 bucket. Both buckets has been configured via AWS storage gateway and shared to Linux servers as nfs shares.
The data size on below servers are:
Server A : approx 2 TB
Server B : approx 8 TB
Server C : approx 25 TB

NFS shares :
Server A : (src-nfsshare1, tgt-nfsshare2)
Server B : (src-nfsshare3, tgt-nfsshare4)
Server C : (src-nfsshare5, tgt-nfsshare6)

Our approach of data migration is, to mount the source S3 (src nfs share x) and target S3 (tgt nfs share x) on each server.
I've used tar command to copy the initial full copy of data which was quite quicker than rsync.

Code:

ex: cd /src/data/01;tar -c * |tar -xvf - -k -C /tgt/data/01 >> /root/tar_01.log

It took me 2 months to complete the full copy process. Now, I've setup lsyncd to replicate the delta data as soon as the source have new/updated data. I preferred lsyncd preffered over rsync (scheduling via cron).

Now, for "Server A" it works fine. I can see the logs are updating and showing the data being updated.

Logs Server A:

Code:

# tail -f /var/log/lsyncd/lsyncd.log
Mon Sep  2 23:50:25 2019 Normal: --- Startup ---
Mon Sep  2 23:51:27 2019 Normal: recursive startup rsync: /src/data/01/ -> /tgt/data/01/ excluding
*.snapshot
lost+found
Mon Sep  2 23:51:27 2019 Normal: recursive startup rsync: /src/data/01/ -> /tgt/data/01/ excluding
*lost+found
*.snapshot
Mon Sep  2 23:56:22 2019 Normal: Startup of /src/data/01/ -> /tgt/data/01/ finished.

Lsyncd Configuration:

Code:

# cat /etc/lsyncd.conf
----
-- User configuration file for lsyncd.
--
-- Simple example for default rsync, but executing moves through on the target.
--
-- For more examples, see /usr/share/doc/lsyncd*/examples/
--
settings {
logfile = "/var/log/lsyncd/lsyncd.log",
statusFile = "/var/log/lsyncd/lsyncd.status",
statusInterval = 60,
}
sync { default.rsync, source = "/src/data/01", target = "/tgt/data/01", delete = "false", delay = 300, exclude = { 'lost+found', '*.snapshot' }, rsync = {archive = true,compress = false,verbose = true} }
#

- Increased the below value from default to 150000 which worked for me. Reason, getting error "Consider increasing /proc/sys/fs/inotify/max_user_watches" while lsyncd initiating the sync.

# cat /proc/sys/fs/inotify/max_user_watches
150000
#

But for server B and C, though lsyncd doesn't throw any error but I think there is some issue. Actually, the server has been started since yesterday but seems it is not doing anything.

Lsyncd configuration for server B & C is same a above (only the src & tgt paths are different).
Here I set inotify watches a little high because till 500000, it was failing.

Code:

# cat /proc/sys/fs/inotify/max_user_watches
600000
#

Lsyncd logs for Server A & B are same:

Code:

# tail -f /var/log/lsyncd/lsyncd.log
Consider increasing /proc/sys/fs/inotify/max_user_watches
Wed Sep  4 12:10:56 2019 Normal: --- Startup ---
Wed Sep  4 12:30:44 2019 Error: Terminating since out of inotify watches.
Consider increasing /proc/sys/fs/inotify/max_user_watches
Wed Sep  4 12:59:58 2019 Normal: --- Startup ---
Wed Sep  4 18:10:36 2019 Error: Terminating since out of inotify watches.
Consider increasing /proc/sys/fs/inotify/max_user_watches
Wed Sep  4 23:21:16 2019 Normal: --- Startup ---
Thu Sep  5 02:14:52 2019 Normal: --- TERM signal, fading ---
Thu Sep  5 02:14:52 2019 Normal: --- Startup ---

Its more than 25 hours but nothing logged in the log files. /etc/lsyncd.status file also blank. Can anybody shade light on this?
Also, I'm looking to add other rsync option with lsyncd (i.e. --stats, --progress or --progress2=info). How can I do that?
How to redirect the errors which occurred during file syncing operation (i.e. io errors or timeout errors) to a separate log file?