Sponsored Content
Operating Systems Solaris The Fastest for copy huge data Post 302917412 by gandolf989 on Tuesday 16th of September 2014 01:14:51 PM
Old 09-16-2014
Have you tried using tar to create a tarball, then copying the tarball to the other server and untaring it? Newer OS's have a --compress option, which might help.

However you copy it, you may want to copy over the directories in different threads, hence you get some level of parallel. You can do a find to get the folder names, then split up the list into separate files and have a different process migrate each folder.
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

fastest copy command

wich is the fastest command in HP-UX to copy an entire disk to dat tapes, or even disk to disk? thanks (0 Replies)
Discussion started by: vascobrito
0 Replies

2. Shell Programming and Scripting

How to extract data from a huge file?

Hi, I have a huge file of bibliographic records in some standard format.I need a script to do some repeatable task as follows: 1. Needs to create folders as the strings starts with "item_*" from the input file 2. Create a file "contents" in each folders having "license.txt(tab... (5 Replies)
Discussion started by: srsahu75
5 Replies

3. UNIX for Advanced & Expert Users

A variable and sum of its value in a huge data.

Hi Experts, I got a question.. In the following output of `ps -elf | grep DataFlow` I get:- 242001 A mqsiadm 2076676 1691742 0 60 20 26ad4f400 130164 * May 09 - 3:02 DataFlowEngine EAIDVBR1_BROKER 5e453de8-2001-0000-0080-fd142b9ce8cb VIPS_INQ1 0 242001 A mqsiadm... (5 Replies)
Discussion started by: varungupta
5 Replies

4. UNIX for Dummies Questions & Answers

copy and paste certain many lines of huge file in linux

Dear All, I am working with windoes OS but remote a linux machine. I wonder the way to copy an paste some part of a huge file in linux machine. the contain of file like as follow: ... dump annealling all custom 10 anneal_*.dat id type x y z q timestep 0.02 run 200000 Memory... (2 Replies)
Discussion started by: ariesto
2 Replies

5. Solaris

The FASTEST copy method?

Hi Experts, I've been asked if there is a fast way to duplicate a file(10GB) and zip it at the same time. The zipped file would be FTP'd.....management is asking this. Maybe there is a better method all together? any ideas? CP will not cut it. Thanks in advance Harley (1 Reply)
Discussion started by: Harleyrci
1 Replies

6. AIX

Copy huge files system

Dear Guy’s By using dd command or any strong command, I’d like to copy huge data from file system to another file system Sours File system: /sfsapp File system has 250 GB of data Target File system: /tgtapp I’d like to copy all these files and directories from /sfsapp to /tgtapp as... (28 Replies)
Discussion started by: Mr.AIX
28 Replies

7. UNIX for Dummies Questions & Answers

Copy huge data into vi editor

Hi All, HP-UX dev4 B.11.11 U 9000/800 3251073457 I need to copy huge data from windows text file to vi editor. when I tried copy huge data, the format of data is not preserverd and appered to scatterd through the vi, something like give below. Please let me know, how can I correct this? ... (18 Replies)
Discussion started by: alok.behria
18 Replies

8. Red Hat

Disk is Full but really does not contain huge data

Hi All, My disk usage show 100 % . When I check “df –kh” it shows my root partition is full. But when I run the “du –skh /” shows only 7 GB is used. Filesystem Size Used Avail Use% Mounted on /dev/sda1 30G 28G 260MB 100% / How I can identify who is using the 20 GB of memory. Os: Centos... (10 Replies)
Discussion started by: kalpeer
10 Replies

9. Shell Programming and Scripting

Aggregation of huge data

Hi Friends, I have a file with sample amount data as follows: -89990.3456 8788798.990000128 55109787.20 -12455558989.90876 I need to exclude the '-' symbol in order to treat all values as an absolute one and then I need to sum up.The record count is around 1 million. How... (8 Replies)
Discussion started by: Ravichander
8 Replies

10. Shell Programming and Scripting

Phrase XML with Huge Data

HI Guys, I have Big XML file with Below Format :- Input :- <pokl>MKL=1,FN=1,GBNo=B10C</pokl> <d>192</d> <d>315</d> <d>35</d> <d>0,7,8</d> <pokl>MKL=1,dFN=1,GBNo=B11C</pokl> <d>162</d> <d>315</d> <d>35</d> <d>0,5,6</d> <pokl>MKL=1,dFN=1,GBNo=B12C</pokl> <d>188</d> (4 Replies)
Discussion started by: pareshkp
4 Replies
PRISTINE-TAR(1) 						   pristine-tar 						   PRISTINE-TAR(1)

NAME
pristine-tar - regenerate pristine tarballs SYNOPSIS
pristine-tar [-vdk] gendelta tarball delta pristine-tar [-vdk] gentar delta tarball pristine-tar [-vdk] [-m message] commit tarball [upstream] pristine-tar [-vdk] checkout tarball pristine-tar [-vdk] list DESCRIPTION
pristine-tar can regenerate an exact copy of a pristine upstream tarball using only a small binary delta file and the contents of the tarball, which are typically kept in an upstream branch in version control. The delta file is designed to be checked into version control along-side the upstream branch, thus allowing Debian packages to be built entirely using sources in version control, without the need to keep copies of upstream tarballs. pristine-tar supports compressed tarballs, calling out to pristine-gz(1), pristine-bz2(1), and pristine-xz(1) to produce the pristine gzip, bzip2, and xz files. COMMANDS
pristine-tar gendelta tarball delta This takes the specified upstream tarball, and generates a small binary delta file that can later be used by pristine-tar gentar to recreate the tarball. If the delta filename is "-", it is written to standard output. pristine-tar gentar delta tarball This takes the specified delta file, and the files in the current directory, which must have identical content to those in the upstream tarball, and uses these to regenerate the pristine upstream tarball. If the delta filename is "-", it is read from standard input. pristine-tar commit tarball [upstream] pristine-tar commit generates a pristine-tar delta file for the specified tarball, and commits it to version control. The pristine-tar checkout command can later be used to recreate the original tarball based only on the information stored in version control. The upstream parameter specifies the tag or branch that contains the same content that is present in the tarball. This defaults to "refs/heads/upstream", or if there's no such branch, any branch matching "upstream". The name of the tree it points to will be recorded for later use by pristine-tar checkout. Note that the content does not need to be 100% identical to the content of the tarball, but if it is not, additional space will be used in the delta file. The delta files are stored in a branch named "pristine-tar", with filenames corresponding to the input tarball, with ".delta" appended. This branch is created or updated as needed to add each new delta. pristine-tar checkout tarball This regenerates a copy of the specified tarball using information previously saved in version control by pristine-tar commit. pristine-tar list This lists tarballs that pristine-tar is able to checkout from version control. OPTIONS
-v --verbose Verbose mode, show each command that is run. -d --debug Debug mode. -k --keep Don't clean up the temporary directory on exit. -m message --message=message Use this option to specify a custom commit message to pristine-tar commit. EXAMPLES
Suppose you maintain the hello package, in a git repository. You have just created a tarball of the release, hello-1.0.tar.gz, which you will upload to a "forge" site. You want to ensure that, if the "forge" loses the tarball, you can always recreate exactly that same tarball. And you'd prefer not to keep copies of tarballs for every release, as that could use a lot of disk space when hello gets the background mp3s and user-contributed levels you are planning for version 2.0. The solution is to use pristine-tar to commit a delta file that efficiently stores enough information to reproduce the tarball later. cd hello git tag -s 1.0 pristine-tar commit ../hello-1.0.tar.gz 1.0 Remember to tell git to push both the pristine-tar branch, and your tag: git push --all --tags Now it is a year later. The worst has come to pass; the "forge" lost all its data, you deleted the tarballs to make room for bug report emails, and you want to regenerate them. Happily, the git repository is still available. git clone git://github.com/joeyh/hello.git cd hello pristine-tar checkout ../hello-1.0.tar.gz LIMITATIONS
Only tarballs, gzipped tarballs, bzip2ed tarballs, and xzed tarballs are currently supported. Currently only the git revision control system is supported by the "checkout" and "commit" commands. It's ok if the working copy is not clean or has uncommitted changes, or has changes staged in the index; none of that will be touched by "checkout" or "commit". ENVIRONMENT
TMPDIR Specifies a location to place temporary files, other than the default. AUTHOR
Joey Hess <joeyh@debian.org> Licensed under the GPL, version 2 or above. perl v5.14.2 2013-06-01 PRISTINE-TAR(1)
All times are GMT -4. The time now is 04:16 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy