Login or Register to Ask a Question and Join Our Community


UNIX for Beginners Questions & Answers

Lsyncd Configuration

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums UNIX for Beginners Questions & Answers Lsyncd Configuration
# 1  
Old 09-06-2019
Lsyncd Configuration
Hi All,
Hope you all doing good.

I'm facing some issue while syncing data using lsyncd. I'm working on a project to migrate data from a source S3 bucket to target S3 bucket. Both buckets has been configured via AWS storage gateway and shared to Linux servers as nfs shares.
The data size on below servers are:
Server A : approx 2 TB
Server B : approx 8 TB
Server C : approx 25 TB

NFS shares :
Server A : (src-nfsshare1, tgt-nfsshare2)
Server B : (src-nfsshare3, tgt-nfsshare4)
Server C : (src-nfsshare5, tgt-nfsshare6)

Our approach of data migration is, to mount the source S3 (src nfs share x) and target S3 (tgt nfs share x) on each server.
I've used tar command to copy the initial full copy of data which was quite quicker than rsync.
Code:
ex: cd /src/data/01;tar -c * |tar -xvf - -k -C /tgt/data/01 >> /root/tar_01.log

It took me 2 months to complete the full copy process. Now, I've setup lsyncd to replicate the delta data as soon as the source have new/updated data. I preferred lsyncd preffered over rsync (scheduling via cron).

Now, for "Server A" it works fine. I can see the logs are updating and showing the data being updated.

Logs Server A:
Code:
# tail -f /var/log/lsyncd/lsyncd.log
Mon Sep  2 23:50:25 2019 Normal: --- Startup ---
Mon Sep  2 23:51:27 2019 Normal: recursive startup rsync: /src/data/01/ -> /tgt/data/01/ excluding
*.snapshot
lost+found
Mon Sep  2 23:51:27 2019 Normal: recursive startup rsync: /src/data/01/ -> /tgt/data/01/ excluding
*lost+found
*.snapshot
Mon Sep  2 23:56:22 2019 Normal: Startup of /src/data/01/ -> /tgt/data/01/ finished.

Lsyncd Configuration:
Code:
# cat /etc/lsyncd.conf
----
-- User configuration file for lsyncd.
--
-- Simple example for default rsync, but executing moves through on the target.
--
-- For more examples, see /usr/share/doc/lsyncd*/examples/
--
settings {
logfile = "/var/log/lsyncd/lsyncd.log",
statusFile = "/var/log/lsyncd/lsyncd.status",
statusInterval = 60,
}
sync { default.rsync, source = "/src/data/01", target = "/tgt/data/01", delete = "false", delay = 300, exclude = { 'lost+found', '*.snapshot' }, rsync = {archive = true,compress = false,verbose = true} }
#

- Increased the below value from default to 150000 which worked for me. Reason, getting error "Consider increasing /proc/sys/fs/inotify/max_user_watches" while lsyncd initiating the sync.

# cat /proc/sys/fs/inotify/max_user_watches
150000
#

But for server B and C, though lsyncd doesn't throw any error but I think there is some issue. Actually, the server has been started since yesterday but seems it is not doing anything.

Lsyncd configuration for server B & C is same a above (only the src & tgt paths are different).
Here I set inotify watches a little high because till 500000, it was failing.
Code:
# cat /proc/sys/fs/inotify/max_user_watches
600000
#

Lsyncd logs for Server A & B are same:

Code:
# tail -f /var/log/lsyncd/lsyncd.log
Consider increasing /proc/sys/fs/inotify/max_user_watches
Wed Sep  4 12:10:56 2019 Normal: --- Startup ---
Wed Sep  4 12:30:44 2019 Error: Terminating since out of inotify watches.
Consider increasing /proc/sys/fs/inotify/max_user_watches
Wed Sep  4 12:59:58 2019 Normal: --- Startup ---
Wed Sep  4 18:10:36 2019 Error: Terminating since out of inotify watches.
Consider increasing /proc/sys/fs/inotify/max_user_watches
Wed Sep  4 23:21:16 2019 Normal: --- Startup ---
Thu Sep  5 02:14:52 2019 Normal: --- TERM signal, fading ---
Thu Sep  5 02:14:52 2019 Normal: --- Startup ---

Its more than 25 hours but nothing logged in the log files. /etc/lsyncd.status file also blank. Can anybody shade light on this?
Also, I'm looking to add other rsync option with lsyncd (i.e. --stats, --progress or --progress2=info). How can I do that?
How to redirect the errors which occurred during file syncing operation (i.e. io errors or timeout errors) to a separate log file?
# 2  
Old 09-06-2019
FWIW (many very little):

I tried lsyncd (for a short time) before but it seemed a bit overly complex to me, so I just run my own rsync scripts via crontab and have had no problems.
This User Gave Thanks to Neo For This Post:
arun_adm
# 3  
Old 09-06-2019
25 TB seems quite a lot.

Are those like billions of small files perhaps ?
Is content is constantly adding and to what extent ?

I have no experience lsyncd with but from Lsyncd github page :

Quote:
Lsyncd is designed to synchronize a local directory tree with low profile of expected changes to a remote mirror.
This sentence would suggest that mentioned software does 'not like' source changing a lot, someone please correct me Smilie

As for inotify, read the 'Limitations and caveats' part of inotify(7) - Linux manual page

There are some stuff there which might interest you if you have large number of watchers.

Hope that helps
Regards
Peasant.
This User Gave Thanks to Peasant For This Post:
arun_adm
# 4  
Old 09-06-2019
Thanks Peasant for your reply.

I'll surely check the inotify link.

Yes, the count of files in the source directory and subsequent sub-directories are in millions for sure, though I don't know the exact count. Mostly, the file types are jpg,png, pdf and xml. There are 2 scripts used to run from cron, one script used to delete 1 year older data from the source on daily basis. Another one used to pull data from another server on daily basis. So I would say there are lot of changes happen in the source on daily basis. But here I want to mention that both scripts used to run in the midnight, 1st one takes approx 2 hours and the 2nd one takes 4-5 hours (got this info from apps team).

--- Post updated at 12:38 AM ---

Thanks Neo for your reply.

Could you please share the rsync and cron setup you used in your usecase? I can give a try with that.

Here is the rsync command which i used to run manually to sync the data.

Code:
nohup /usr/bin/time --format='Elapsed Time = %e seconds' rsync -avh --exclude="lost+found" --exclude=".snapshot" --info=progress2 --stats /src/data/01/* /tgt/data/01/ > /root/rs-logs/data_01-`date +%Y%m%d-%H%M`.log &

# 5  
Old 09-06-2019
I used lsyncd to migrate(p2v) servers to minimize downtime when migrating. Had lot's of files(1K-1M).

Like plain rsync it's good for a lot of things. It's very bad for syncing big files which change often(-> big database files), because lsyncd will constantly syncing those big files if only one byte changes. ( I had ~100 smaller mysql databases on some server. This worked fine.)

Worked like charm. You can apply rsync args with rsync/_extra:

Code:
sync {
    default.rsync,
    source    = "/",
    target    = "1.2.3.42:/disk/",
    exclude   = {
                "/proc/**",
                "/sys/**",
                "/dev/**"
                },
    rsync     = {
        archive  = true,
        _extra  = { "-v", "-H", "--numeric-ids" }
    }
}

Update:
Since you copy files from nfs to nfs I recommend to check disabling the Delta-Transfer with -W.

If not lsyncd/rsync will read the whole file from the source-nfs before transferring the 5 (Bytes? MBytes? GBytes?) that have changed.

The above is default, when - like in this case - both source and target path are local(begin with /).

If you have so many files and access it via nfs the initial phase is of course taking long. (I had backup issues of servers with 4-8 million files, took 10-30 hours for an rsync backup to take place even if little changed - and that was with a local source). This should be either with lsyncd or rsync. But I'm curious if plain rsync is faster.

If you just have vanishing many files, this should not cause heavy load.

I think there's not so much difference between lsyncd and rsync. Because the former is just like a wrapper of the latter.
Last edited by stomp; 09-07-2019 at 10:09 AM..
# 6  
Old 09-08-2019
Quote:
Originally Posted by arun_adm

Thanks Neo for your reply.

Could you please share the rsync and cron setup you used in your usecase? I can give a try with that.

Here is the rsync command which i used to run manually to sync the data.

Code:
nohup /usr/bin/time --format='Elapsed Time = %e seconds' rsync -avh --exclude="lost+found" --exclude=".snapshot" --info=progress2 --stats /src/data/01/* /tgt/data/01/ > /root/rs-logs/data_01-`date +%Y%m%d-%H%M`.log &

I have my rsync command(s) in a crontab script(s), like so, for example:

Code:
30 23 * * * /usr/bin/nice -n 19 /usr/local/bin/sync_dumps > /dev/null 2>&1

... where my script sync_dumps contains an rsync command set up for my particular case.

If the files have not changed, rsync will log and exit, of course.

This is a classic "polling" versus "pushing" problem, and I prefer to keep this simple with rsync and cron. The frequency of my backups is based on my risk model and my risk model for this instance says that backing up to a remote server once a day is fine. Your risk model may be different and requires a different backup frequency.

FWIW, I never (rarely, is a better word, I guess) use additional processes to perform tasks which I can perform in a more simpler way, so I use rsync and not lsyncd, but that's just me. I like simple. It's easily to maintain, update and troubleshoot if there are problems (or port to a new server when a datacenter change is required).
Login or Register to Ask a Question

Previous Thread | Next Thread

8 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Postfix Configuration

Hi, i have to configure postfix in this mode: - if i receive a mail from a@a.a relay to - if i receive a mail NOT from a@a.a relay the mail to user "test" that is on localhost For the first rule i do this in main.cf: sender_dependent_default_transport_maps = hash:/etc/postfix/sender_check... (0 Replies)
Discussion started by: certo85
0 Replies

2. Red Hat

LDAP configuration

I have installed openldap-servers package in server machine i have edited vi /etc/openldap/ldap.conf i have added following line BASE dc=abcd,dc=com URI ldap://ldap.abcd.com ldap://ldap-master.abcd.com:666 i have restarted the ldap service then client pc i have installed... (0 Replies)
Discussion started by: ainstin
0 Replies

3. SCO

printer configuration

Dear all i have install printer in sco open 5 , i m not getting print out , but printing is showing in que plz help (7 Replies)
Discussion started by: sudhir69
7 Replies

4. AIX

Need help on IP Configuration

I have an RS/6000 running AIX 5.3. I would like to get either dhcp client working or assigning it an ip would be fine as well. I have tried using smit to do it many times trying different things, and I can get an ip assigned but it doesn't communicate with the network or internet. I haven't been... (3 Replies)
Discussion started by: izzzy
3 Replies

5. Linux

configuration

hello, I don't know how to configure servor dns on linux,please help I thanck YOU (1 Reply)
Discussion started by: djest
1 Replies

6. Post Here to Contact Site Administrators and Moderators

configuration

how to configure servor dns on windows servor 2003 (0 Replies)
Discussion started by: djest
0 Replies

7. BSD

X configuration

HEllo, I try to Launch OOO, but I have X configuration problem. I 'm searching in man pages but if someone can help me... Of course I set the DISPLAY as explain in the man page but with no more result: setenv DISPLAY myws:0 Thanks in advance (1 Reply)
Discussion started by: SoulCoder
1 Replies

8. UNIX for Dummies Questions & Answers

Configuration

Can anyone please tell me anything about the following: AIX O/S - IBM RS6000 server I want to know where I can go to check and see how much RAM is intalled in the server, how many Processors are installed in the server, and how I can run a sar command to show me processor statistics? I... (4 Replies)
Discussion started by: Docboyeee
4 Replies
Login or Register to Ask a Question