rsync from partition to partition fastest


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting rsync from partition to partition fastest
# 1  
Old 11-27-2010
rsync from partition to partition fastest

Gentleman,
Please move if I have chose the incorrect forum section. I am trying to move data that is not backed up from partition 1 to partition 2 on a SAN that has a GFS2 filesystem. Since the data is not backed up I am rsyncing this data and once verified I will delete from the source partition.

The command that I am using is painfully slow. However this is the way that I was taught to use the rsync command and I am having trouble sorting out the options from the man page.

Can someone please recommend a better or faster command set?

Here is the command I am currently using.

Code:
[rubedo@NYOTS01 ~]$ /usr/bin/rsync  -rlvtzgoD /mnt/onetick2/data/taq_nbbo/ /mnt/onetick1/databases/

At the rate that I am currently syncing it will take days to copy the 2TB's that I am trying to move.

jaysunn
# 2  
Old 11-27-2010
What kind of data is this? Many small files, many large ones, or mixed? If it's many small files, it will take longer as rsync has to build an index of all files in both directories.

Generally, you could drop all options and just run rsync -av ..., as -a will work the same as your command line, minus the compression. And compression between 2 disks on the same system is pretty useless anyways.
This User Gave Thanks to pludi For This Post:
# 3  
Old 11-27-2010
Hi Pludi,

The files are made up of a flat file database. The data is in a directory for each day of the month and they go back to 2008. The database is also binary so I cannot view the contents. Here is a listing for one day.


Code:
[rubedo@NYOTS01 20101126]$ ls -lah
total 1.3G
drwxr-xr-x   2 rubedo rubedo 3.8K Nov 27 00:27 .
drwxrwxr-x 971 rubedo rubedo 2.0K Nov 27 00:27 ..
-rw-r--r--   1 rubedo rubedo 1.3G Nov 27 00:27 data_unassigned
-rw-r--r--   1 rubedo rubedo 1.5M Nov 27 00:27 index
-rw-r--r--   1 rubedo rubedo  786 Nov 27 00:27 meta_data
-rw-r--r--   1 rubedo rubedo  83K Nov 27 00:27 symbols

Hope this helps with your diagnosing of the correct tool to use. Right now I have been running rsync and we are only in the month of March from 2008 and I have to go to present.

jaysunn

---------- Post updated at 11:22 AM ---------- Previous update was at 10:58 AM ----------

pludi,

Code:
rsync -av ...

This seems to be much faster. Thanks
jaysunn
# 4  
Old 11-27-2010
For the first backup just use copy and when you finish just sync using rsysc
# 5  
Old 11-27-2010
@danmero,

Are you suggesting. I use:

Code:
cp -r /path/to/data /path/to/data2

I was worried that the shear amount of data would not work well with cp command.

Note that this is a GFS2 redhat filesystem on 2 separate partitions.
I am getting great speeds now with the rsync - av command.

Not there is no backup for this data so I wish to be as careful as possible.

jaysunn
# 6  
Old 11-27-2010
Quote:
Originally Posted by jaysunn
@danmero,

Are you suggesting. I use:

Code:
cp -r /path/to/data /path/to/data2

Check your cp man page to see if -a argument is supported(should be).
# 7  
Old 11-27-2010
pludi,
A bit off topic, however when I type man cp on a rhel 5 system I am unable to see options. However when I type man cp > cp_man and concatenate the file I can see many options. Very weird IMHO....

Code:
man cp

Code:
man cp > cp_man


My cp command does support the -a option.


Code:
 -a, --archive
              same as -dpR


jaysunn
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Red Hat

Shrink LVM partition & create new Linux Primary partition

Hello All, I have a Red Hat Linux 5.9 Server installed with one hard disk & 2 Partitions created on it as follows, /boot - Linux Partition & another is LVM - One VG & under that 5-6 Logical volumes(var,opt,home etc). Here my requirement is to take out 1GB of space from LVM ( Any logical... (5 Replies)
Discussion started by: gr8_usk
5 Replies

2. Filesystems, Disks and Memory

Ask concept soft partition vs hard partition

Hi Experts I would like to know different between soft partition concept and hard partition concept on solaris. Here is little explanation between soft partition concept and hard partition concept on solaris. Soft Partition: 1TB total space available in storage in all mapped to the OS to... (2 Replies)
Discussion started by: edydsuranta
2 Replies

3. UNIX for Dummies Questions & Answers

Partition

Hi, Can anyone explain me the concepts of partitioning and mounting in unix. As i am new to unix. Thanks, Manju (1 Reply)
Discussion started by: Kashyap
1 Replies

4. Solaris

Partition overlaps another partition while creating new parition in solaris

hi all while formatting hard disk i am getting following error. Partition 1 ends at 266338338 It must be between 34 and 143374704. label error: EFI Labels do not support overlapping partitions Partition 8 overlaps partition 1. Warning: error writing EFI. Label failed. I have formatted the... (2 Replies)
Discussion started by: nikhil kasar
2 Replies

5. Linux

Partition

Why the hard drive can only have 4 primary partition. If i want do more, what should i do? (3 Replies)
Discussion started by: RaNoNeR
3 Replies

6. UNIX for Dummies Questions & Answers

I've created a partition with GNU Parted, how do I mount the partition?

I've created a partition with GNU Parted, how do I mount the partition? The manual information at http://www.gnu.org/software/parted/manual/parted.html is good, but I am sure about how I mount the partition afterwards. Thanks, --Todd (1 Reply)
Discussion started by: jtp51
1 Replies

7. UNIX for Dummies Questions & Answers

understanding logical partition, physical partition

hi, 1) is logical partition the same as physical partition except that one is physical and the other is logical? 2) then it must a one to one ratio? (3 Replies)
Discussion started by: yls177
3 Replies

8. UNIX for Dummies Questions & Answers

Partition

Hi, wondering if anyone can help a self teaching lad out....I want to partition my windows 2000 box and install Redhat 7.2 on it...I know how to get to the disk management section...but how do I make a new partition..I clicked on the icons and read the help but it is very hard to read for some... (4 Replies)
Discussion started by: jonas27
4 Replies

9. UNIX for Dummies Questions & Answers

partition

i have a second HD installed on my computer, which came partioned in 3 fragments..how do i get rid of the partitions... i dont know how to use F disk thanks:D (5 Replies)
Discussion started by: pops
5 Replies

10. UNIX for Dummies Questions & Answers

partition

How do I create a new Disk partition in HP-unix machine??? (1 Reply)
Discussion started by: chakradharm
1 Replies
Login or Register to Ask a Question