Shell Programming and Scripting

Gentleman,
Please move if I have chose the incorrect forum section. I am trying to move data that is not backed up from partition 1 to partition 2 on a SAN that has a GFS2 filesystem. Since the data is not backed up I am rsyncing this data and once verified I will delete from the source partition.

The command that I am using is painfully slow. However this is the way that I was taught to use the rsync command and I am having trouble sorting out the options from the man page.

Can someone please recommend a better or faster command set?

Here is the command I am currently using.

Code:

[rubedo@NYOTS01 ~]$ /usr/bin/rsync  -rlvtzgoD /mnt/onetick2/data/taq_nbbo/ /mnt/onetick1/databases/

At the rate that I am currently syncing it will take days to copy the 2TB's that I am trying to move.

jaysunn

What kind of data is this? Many small files, many large ones, or mixed? If it's many small files, it will take longer as rsync has to build an index of all files in both directories.

Generally, you could drop all options and just run rsync -av ..., as -a will work the same as your command line, minus the compression. And compression between 2 disks on the same system is pretty useless anyways.

Hi Pludi,

The files are made up of a flat file database. The data is in a directory for each day of the month and they go back to 2008. The database is also binary so I cannot view the contents. Here is a listing for one day.

Code:

[rubedo@NYOTS01 20101126]$ ls -lah
total 1.3G
drwxr-xr-x   2 rubedo rubedo 3.8K Nov 27 00:27 .
drwxrwxr-x 971 rubedo rubedo 2.0K Nov 27 00:27 ..
-rw-r--r--   1 rubedo rubedo 1.3G Nov 27 00:27 data_unassigned
-rw-r--r--   1 rubedo rubedo 1.5M Nov 27 00:27 index
-rw-r--r--   1 rubedo rubedo  786 Nov 27 00:27 meta_data
-rw-r--r--   1 rubedo rubedo  83K Nov 27 00:27 symbols

Hope this helps with your diagnosing of the correct tool to use. Right now I have been running rsync and we are only in the month of March from 2008 and I have to go to present.

jaysunn

---------- Post updated at 11:22 AM ---------- Previous update was at 10:58 AM ----------

pludi,

Code:

rsync -av ...

This seems to be much faster. Thanks
jaysunn

For the first backup just use copy and when you finish just sync using rsysc

@danmero,

Are you suggesting. I use:

Code:

cp -r /path/to/data /path/to/data2

I was worried that the shear amount of data would not work well with cp command.

Note that this is a GFS2 redhat filesystem on 2 separate partitions.
I am getting great speeds now with the rsync - av command.

Not there is no backup for this data so I wish to be as careful as possible.

jaysunn

Check your cp man page to see if -a argument is supported(should be).

pludi,
A bit off topic, however when I type man cp on a rhel 5 system I am unable to see options. However when I type man cp > cp_man and concatenate the file I can see many options. Very weird IMHO....

Code:

man cp

Code:

man cp > cp_man

My cp command does support the -a option.

Code:

 -a, --archive
              same as -dpR

jaysunn