Rsync in progress, strange results


Login or Register for Dates, Times and to Reply

 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users Rsync in progress, strange results
# 1  
Rsync in progress, strange results

Disclaimer, I've been a Linux admin for a while but don't frequently setup rsysnc jobs.

Here's the command I'm running on CentOS 5.5, rsync 2.6.8:
Code:
rsync -arvz --progress --compress-level=9 /src/ /dest/

/src has 1.5 TB of data, /dest/ is a new destination and started out empy. Oh ya, both /srv and /dest are cifs shares. I know that /src is off a Windows 2008 server and /dest I have no idea. The /src is a local file server connected via 1g ethernet and /dest is remote on the other end of a slow WAN link.

Here's the mount output:
Code:
//remotesf/dest on /dest type cifs (rw,mand)
//localfs/src on /src type cifs (rw,mand)

Here's the part that's got me scratching my head. This transfer has been running for a few hours and seems to be fine. I know it's gonna take a few days and I'm good with that. BUT, I'm watching transfer and as a file completes I'm doing a ls -lah so I can see the completed file but its not there... like all the directories are being created, the copies are taking time and doing their work but when I browse or ls the directory it's empty.

What's the deal?

---------- Post updated at 10:20 PM ---------- Previous update was at 10:18 PM ----------

Bonus question; without stopping the running job anyone got a good way to see how much has transferred? df -h /dest doesn't seem to produce any useful output.

Last edited by Scrutinizer; 03-15-2014 at 12:43 AM.. Reason: code tags
# 2  
DustinT, I don't have any experience with cifs and I don't have any answers to your questions. But I do know what I would do if I were in your situation. I am rather conserative by nature and I don't simply go jump into some unkown pond. I dip my toe in the water first...

I would abort this job since you only have a few hours invested in it. Set up a very small sample .. a few dozen files in a few dozen subdirectories. It should be large enough to consume about 5 to 10 minutes. Rsync that. During the rsync do you see the files appear? When it's done does everything look ok?

If your sample works ok, then start the real job. If the sample doesn't work, the real job probably won't work either. Figure out how to get the sample working. And then do the big job.

If you let the multi-day job continue to run you risk having nothing to show for all those days. Maybe that's not a problem. If you have several weeks to do this and it's just a background sort of thing maybe you can let it roll and see what happens. But even in that case, I would have tried a small sample first. Just saying....

Last edited by Perderabo; 03-15-2014 at 01:57 AM..
# 3  
On an older system, we set up cifs that required authentication: the controlling (parent) process had to be an already authenticated user to a Windows fileserver before any file activity was allowed. Authentication was persistent for the duration of the cifs mount.

Any unauthenticated process that ran in the background simply hung, root included.

Is that the case for your filesystem?
# 4  
Quote:
Originally Posted by Perderabo
If you let the multi-day job continue to run you risk having nothing to show for all those days. Maybe that's not a problem. If you have several weeks to do this and it's just a background sort of thing maybe you can let it roll and see what happens. But even in that case, I would have tried a small sample first. Just saying....
You make some excellent points in your post. I have done some testing of the process, spent a couple hours on it as a matter of fact. Testing from the /src and too a remote linux server as the dest over ssh worked perfectly. For the production work, the /dest was switched to a remote cifs server which I didn't think I needed to test for. Now, it seems obvious that config needed testing too.

Quote:
Originally Posted by jim mcnamara
Any unauthenticated process that ran in the background simply hung, root included.
Both the /src and /dest were mounted with domain credentials. Wouldn't that cover the authentication needs of the rsync process transparently?

On another note, it's interesting because while the file is being copied, I can see it if I browse the remote directory. Once the transfer is completed, the remote version disappears.
# 5  
Quote:
Originally Posted by DustinT
On another note, it's interesting because while the file is being copied, I can see it if I browse the remote directory. Once the transfer is completed, the remote version disappears.
Turns out that error was because I wasn't running rsync with enough permissions. That was an easy fix but I needed one more thing to complete this project.

I ended up having the user onsite at the remote location send the files on a USB drive and then I used rysnc to complete the transfer once the drive was here. It proceeded much more quickly and wrapped up after about 48 hours. The remote user sent more files than I was initially targeting so the final transfer ended up being 2.6 million files.
Login or Register for Dates, Times and to Reply

Previous Thread | Next Thread
Thread Tools Search this Thread
Search this Thread:
Advanced Search

Test Your Knowledge in Computers #910
Difficulty: Easy
In C, you can declare a pointer as follows: data_type * pointer_variable_name;
True or False?

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Strange results from 'strings | sort'

Using the 'strings' command and piping the result to 'sort' is producing strange results. I get block of lines that begin with asterisks, then a block that begins with some text, then more lines that begin with asterisks. The actual content is correct - lines beginning with asterisks is the... (5 Replies)
Discussion started by: edstevens
5 Replies

2. Shell Programming and Scripting

I want to add a variable for the results from the formula of one variable and results of another var

Good morning all, This is the file name in question OD_Orders_2019-02-19.csv I am trying to create a bash script to read into files with yesterdays date on the file name while retaining the rest of the files name. I would like for $y to equal, the name of the file with a formula output with... (2 Replies)
Discussion started by: Ibrahim A
2 Replies

3. UNIX for Beginners Questions & Answers

Strange sort -r results

Hi Folks - I have this file that looks like this: outbox/logs/Client_1042.log outbox/logs/Client_941.log outbox/logs/Client_942.log outbox/logs/Client_943.log outbox/logs/Client_944.log And this is my code: #!/bin/bash _OUTBOX_BIN="outbox/logs/" _NAME="Client" _TEMP="temp.txt"... (9 Replies)
Discussion started by: SIMMS7400
9 Replies

4. Shell Programming and Scripting

Rsync Error: rsync: link_stat failed: No such file or directory (2)

I wish to copy all the files & folder under /web/Transfer_Files/data/ on mymac1 (Linux) to remote server mybank.intra.com (Solaris 10) /tmp/ location I am using Ansible tool synchronize module which triggers the unix rsync command as below:rsync --delay-updates -F --compress --archive --rsh=ssh... (2 Replies)
Discussion started by: mohtashims
2 Replies

5. Shell Programming and Scripting

Find with rm command gives strange results

I want to remove any files that are older than 2 days from a directory. It deletes those files. Then it comes back with a message it is a directory. What am I doing wrong here? + find /mydir -mtime +2 -exec rm -f '{}' ';' rm: /mydir is a directory (2 Replies)
Discussion started by: jtamminen
2 Replies

6. Shell Programming and Scripting

Rsync, compression, progress and logging

Hello. Using Rsync to synchronize files on 2 red hat servers, works fine. Trying to use the compress options, but I don't know if it is working? Nothing logged in the progress messages. Not sure in my case if compression will be helpful in my case, so I would like to see some statistics. Is... (2 Replies)
Discussion started by: JimBurns
2 Replies

7. Shell Programming and Scripting

Can ctag and cscope support recording search results and displaying the history results ?

Hello , When using vim, can ctag and cscope support recording search results and displaying the history results ? Once I jump to one tag, I can use :tnext to jump to next tag, but how can I display the preview search result? (0 Replies)
Discussion started by: 915086731
0 Replies

8. Shell Programming and Scripting

Strange results from FDISK?????

Hi all, I am writing script that returns the size of each disk or partition when called. I am using FDISK -l and parsing the results to get the result I want. When I execute fdisk -l it shows correct results, BUT when I execute the same thing with results to be put in a variable, I get strange... (5 Replies)
Discussion started by: alirezan
5 Replies

9. UNIX for Dummies Questions & Answers

Strange Results

I am using th following to get the percentage and have never used bc before: percent=$(echo "scale=4;(34117/384000)*100" | bc) 8.884600 percent=$(echo "scale=2;(34117/384000)*100" | bc) 8.00 Why do I get the results of 8.00 instead of 8.88 when using a scale of 2. I only want 2 decimal... (2 Replies)
Discussion started by: mariaa33
2 Replies

10. AIX

Strange ls results..

Multipart question.. Can anybody explain why this happens : -rw-rw-r-- 1 fnsw fnusr 1531061 Feb 13 21:45 filename1.log -rw-rw-r-- 1 fnsw fnusr 1760706 Feb 10 22:10 filename2.log -rw-rw-r-- 1 fnsw fnusr 1525805 Aug 16 2005 filename3.log -rw-rw-r-- 1... (3 Replies)
Discussion started by: dbridle
3 Replies

Featured Tech Videos