07-10-2010
Quote:
Originally Posted by
m_wassal
how many lines per file do u think it will be good to split the fatty file
As this is very machine-dependent, I'd start with a ten-percent split, process one (!) file, time this, then either process the remaining nine splits - or split, process and time the "guinea pig" again (thus going down to e.g. five-percent splits). You get the idea ...
9 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hi Friends,
Getting an error while processing a very large file using an sqlloader........
The file is larger than 2 GB. Now need to change the compiler to 64-bit so that the file can be processed.
Is there any command for the same.
Thanks in advance. (1 Reply)
Discussion started by: Rohini Vijay
1 Replies
2. Shell Programming and Scripting
how to remove all zero byte files in a particular directory and also files that are morew than 1GB. pLEASE let me know (3 Replies)
Discussion started by: dsravan
3 Replies
3. UNIX for Advanced & Expert Users
We are experiencing a problem on a lengthy data transfer by FTP through a firewall. Since there are two ports in use on a ftp transfer (data and control), one sits idle while the other's transfering data. The idle port (control) will get timed out and the data transfer won't know that it's... (3 Replies)
Discussion started by: rprajendran
3 Replies
4. UNIX for Dummies Questions & Answers
Hi all;
I'm having a problem when want to list a large number of files in current directory using find together with the prune option.
First i used this command but it list all the files including those in sub directories:
find . -name "*.dat" | xargs ls -ltr
Then i modified the command... (2 Replies)
Discussion started by: ashikin_8119
2 Replies
5. UNIX for Dummies Questions & Answers
I have a large file, around 570 gb that I want to copy to tape. However, my tape drive will load only up to 500 gb. I don't have enough space on disk to compress it before copying to tape. Can I compress and tar to tape in one command without writing a compressed disk file?
Any suggestions... (8 Replies)
Discussion started by: iancrozier
8 Replies
6. UNIX for Dummies Questions & Answers
Hi, I am a torrent-maniak and I use Transmission.
All things were good but Nautilus begun to show problem while I was runnning Transmission.Its situation was becoming worse and worse.
Now, when I boot I can hardly open a nautilus window and browse my files.It will "stack" in seconds for sure!
I... (2 Replies)
Discussion started by: hakermania
2 Replies
7. Shell Programming and Scripting
Hello everyone!
I have 2 types of files in the following format:
1) *.fa
>1234
...some text...
>2345
...some text...
>3456
...some text...
.
.
.
.
2) *.info
>1234 (7 Replies)
Discussion started by: ad23
7 Replies
8. Solaris
Hello everyone. Need some help copying a filesystem. The situation is this: I have an oracle DB mounted on /u01 and need to copy it to /u02. /u01 is 500 Gb and /u02 is 300 Gb. The size used on /u01 is 187 Gb. This is running on solaris 9 and both filesystems are UFS.
I have tried to do it using:... (14 Replies)
Discussion started by: dragonov7
14 Replies
9. Shell Programming and Scripting
Hello everyone,
I have two matrices at same sizes. I need to re-calculate the numbers in matrix A according to the percentages in martix B
it is like
matrix A is
10.00 20.00 30.00 40.00
60.00 70.00 80.00 90.00
20.00 30.00 80.00 50.00
martix B is
00.08 00.05 ... (2 Replies)
Discussion started by: miriammiriam
2 Replies
LEARN ABOUT CENTOS
shell-quote
SHELL-QUOTE(1) User Contributed Perl Documentation SHELL-QUOTE(1)
NAME
shell-quote - quote arguments for safe use, unmodified in a shell command
SYNOPSIS
shell-quote [switch]... arg...
DESCRIPTION
shell-quote lets you pass arbitrary strings through the shell so that they won't be changed by the shell. This lets you process commands
or files with embedded white space or shell globbing characters safely. Here are a few examples.
EXAMPLES
ssh preserving args
When running a remote command with ssh, ssh doesn't preserve the separate arguments it receives. It just joins them with spaces and
passes them to "$SHELL -c". This doesn't work as intended:
ssh host touch 'hi there' # fails
It creates 2 files, hi and there. Instead, do this:
cmd=`shell-quote touch 'hi there'`
ssh host "$cmd"
This gives you just 1 file, hi there.
process find output
It's not ordinarily possible to process an arbitrary list of files output by find with a shell script. Anything you put in $IFS to
split up the output could legitimately be in a file's name. Here's how you can do it using shell-quote:
eval set -- `find -type f -print0 | xargs -0 shell-quote --`
debug shell scripts
shell-quote is better than echo for debugging shell scripts.
debug() {
[ -z "$debug" ] || shell-quote "debug:" "$@"
}
With echo you can't tell the difference between "debug 'foo bar'" and "debug foo bar", but with shell-quote you can.
save a command for later
shell-quote can be used to build up a shell command to run later. Say you want the user to be able to give you switches for a command
you're going to run. If you don't want the switches to be re-evaluated by the shell (which is usually a good idea, else there are
things the user can't pass through), you can do something like this:
user_switches=
while [ $# != 0 ]
do
case x$1 in
x--pass-through)
[ $# -gt 1 ] || die "need an argument for $1"
user_switches="$user_switches "`shell-quote -- "$2"`
shift;;
# process other switches
esac
shift
done
# later
eval "shell-quote some-command $user_switches my args"
OPTIONS
--debug
Turn debugging on.
--help
Show the usage message and die.
--version
Show the version number and exit.
AVAILABILITY
The code is licensed under the GNU GPL. Check http://www.argon.org/~roderick/ or CPAN for updated versions.
AUTHOR
Roderick Schertler <roderick@argon.org>
perl v5.16.3 2010-06-11 SHELL-QUOTE(1)