07-10-2010
a problem with large files
hello all,
kindly i need your help, i made a script to print a specific lines from a huge file about 3 million line. the output of the script will be about 700,000 line...the problem is the script is too slow...it kept working for 5 days and the output was only 200,000 lines !!!
the script is so simple:
for i in `cat file` ------> file is the file that contains the line no. to be printed from a file.
sed '$i q;d' file1 > file2 ----> where file1 is the huge file 3 millions lines and file 2 is the output file which will be 700.000 lines
so plz could anyone tell me how can i decrease the processing time of that script and why is it taking all that time !!!?
thanks in advance
9 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hi Friends,
Getting an error while processing a very large file using an sqlloader........
The file is larger than 2 GB. Now need to change the compiler to 64-bit so that the file can be processed.
Is there any command for the same.
Thanks in advance. (1 Reply)
Discussion started by: Rohini Vijay
1 Replies
2. Shell Programming and Scripting
how to remove all zero byte files in a particular directory and also files that are morew than 1GB. pLEASE let me know (3 Replies)
Discussion started by: dsravan
3 Replies
3. UNIX for Advanced & Expert Users
We are experiencing a problem on a lengthy data transfer by FTP through a firewall. Since there are two ports in use on a ftp transfer (data and control), one sits idle while the other's transfering data. The idle port (control) will get timed out and the data transfer won't know that it's... (3 Replies)
Discussion started by: rprajendran
3 Replies
4. UNIX for Dummies Questions & Answers
Hi all;
I'm having a problem when want to list a large number of files in current directory using find together with the prune option.
First i used this command but it list all the files including those in sub directories:
find . -name "*.dat" | xargs ls -ltr
Then i modified the command... (2 Replies)
Discussion started by: ashikin_8119
2 Replies
5. UNIX for Dummies Questions & Answers
I have a large file, around 570 gb that I want to copy to tape. However, my tape drive will load only up to 500 gb. I don't have enough space on disk to compress it before copying to tape. Can I compress and tar to tape in one command without writing a compressed disk file?
Any suggestions... (8 Replies)
Discussion started by: iancrozier
8 Replies
6. UNIX for Dummies Questions & Answers
Hi, I am a torrent-maniak and I use Transmission.
All things were good but Nautilus begun to show problem while I was runnning Transmission.Its situation was becoming worse and worse.
Now, when I boot I can hardly open a nautilus window and browse my files.It will "stack" in seconds for sure!
I... (2 Replies)
Discussion started by: hakermania
2 Replies
7. Shell Programming and Scripting
Hello everyone!
I have 2 types of files in the following format:
1) *.fa
>1234
...some text...
>2345
...some text...
>3456
...some text...
.
.
.
.
2) *.info
>1234 (7 Replies)
Discussion started by: ad23
7 Replies
8. Solaris
Hello everyone. Need some help copying a filesystem. The situation is this: I have an oracle DB mounted on /u01 and need to copy it to /u02. /u01 is 500 Gb and /u02 is 300 Gb. The size used on /u01 is 187 Gb. This is running on solaris 9 and both filesystems are UFS.
I have tried to do it using:... (14 Replies)
Discussion started by: dragonov7
14 Replies
9. Shell Programming and Scripting
Hello everyone,
I have two matrices at same sizes. I need to re-calculate the numbers in matrix A according to the percentages in martix B
it is like
matrix A is
10.00 20.00 30.00 40.00
60.00 70.00 80.00 90.00
20.00 30.00 80.00 50.00
martix B is
00.08 00.05 ... (2 Replies)
Discussion started by: miriammiriam
2 Replies
DIFF(1) General Commands Manual DIFF(1)
NAME
diff - differential file comparator
SYNOPSIS
diff [ -efbh ] file1 file2
DESCRIPTION
Diff tells what lines must be changed in two files to bring them into agreement. If file1 (file2) is `-', the standard input is used. If
file1 (file2) is a directory, then a file in that directory whose file-name is the same as the file-name of file2 (file1) is used. The
normal output contains lines of these forms:
n1 a n3,n4
n1,n2 d n3
n1,n2 c n3,n4
These lines resemble ed commands to convert file1 into file2. The numbers after the letters pertain to file2. In fact, by exchanging `a'
for `d' and reading backward one may ascertain equally how to convert file2 into file1. As in ed, identical pairs where n1 = n2 or n3 = n4
are abbreviated as a single number.
Following each of these lines come all the lines that are affected in the first file flagged by `<', then all the lines that are affected
in the second file flagged by `>'.
The -b option causes trailing blanks (spaces and tabs) to be ignored and other strings of blanks to compare equal.
The -e option produces a script of a, c and d commands for the editor ed, which will recreate file2 from file1. The -f option produces a
similar script, not useful with ed, in the opposite order. In connection with -e, the following shell program may help maintain multiple
versions of a file. Only an ancestral file ($1) and a chain of version-to-version ed scripts ($2,$3,...) made by diff need be on hand. A
`latest version' appears on the standard output.
(shift; cat $*; echo '1,$p') | ed - $1
Except in rare circumstances, diff finds a smallest sufficient set of file differences.
Option -h does a fast, half-hearted job. It works only when changed stretches are short and well separated, but does work on files of
unlimited length. Options -e and -f are unavailable with -h.
FILES
/tmp/d?????
/usr/lib/diffh for -h
SEE ALSO
cmp(1), comm(1), ed(1)
DIAGNOSTICS
Exit status is 0 for no differences, 1 for some, 2 for trouble.
BUGS
Editing scripts produced under the -e or -f option are naive about creating lines consisting of a single `.'.
DIFF(1)