Do you have room to unzip all the files somewhere?
Unzip all files and append to one big file (your havelist), then use awk to check each line of your havelist against file1.txt:
Code:
( find . -name "*processed" -print | xargs gunzip -c ) > /scratch/havelist
awk ' NR == FNR { F[i++]=$0; next}
{ for(i in F)if(index($0, F[i])) delete F[i]; }
END { for(i in F) print "number "F[i]" not found." } ' file1.txt havelist > outputfile.txt
rm /scratch/havelist
awk may chew a big chunk of memory as it loads the 5,000,000 numbers into it's array, and you won't get any output till it's done, but it will be much quicker that your original attempt.
Some more efficency can be gained if you can say that each processed file line only matches to 1 number from file1.txt (ie a line in processed file dosn't contain 2 or more numbers you are looking for).
But, with record counts this large you should really be considering using a database rather than flat/zipped files.
---------- Post updated at 08:47 AM ---------- Previous update was at 08:15 AM ----------
If you just don't have room to extract to a scratchfile you can do the search through a pipe:
Code:
( find . -name "*processed" -print | xargs gunzip -c ) | awk ' NR == FNR { F[i++]=$0; next}
{ for(i in F)if(index($0, F[i])) delete F[i]; }
END { for(i in F) print "number "F[i]" not found." } ' file1.txt - > outputfile.txt
Last edited by Chubler_XL; 01-20-2011 at 06:55 PM..
Reason: Orginal solution listed processed lines not in file1.txt
I have one file:
123*100*abcd*10
123*101*abcd*-29*def
123*100*abcd*-10
123*102*abcd*-105*asd
I would like to parameterize the search patterns in the following way so that the user could dynamically change the search pattern.
*100* and *- (ie *minus)
*102* and *-
The output that is... (6 Replies)
Hi
I'm not very good with the serach patterns and I'd need a sample how to find a line that has multiple patterns.
Say I want to find a line that has "abd", "123" and "QWERTY" and there can be any characters or numbers between the serach patterns, I have a file that has thousands of lines and... (10 Replies)
Good day, great gurus,
I'm new to Perl, and programming in general. I'm trying to retrieve a column of data from my text file which spans a non-specific number of lines. So I did a regexp that will pick out the columns. However,my pattern would vary. I tried using a foreach loop unsuccessfully.... (2 Replies)
I have two lists in a file that look like
a b
b a
e f
c d
f e
d c
I would like a final list
a b
c d
e f
I've tried multiple grep and awk but can't get it to work (8 Replies)
Hi,
I tried to search multiple pattern using awk
trans=1234
reason=LN MISMATCH
rec=`awk '/$trans/ && /'"$reason"'/' file`
whenevr i tried to run on command promt it is executing but when i tried to implment same logic in shell script,it is failing i.e $rec is empty
... (6 Replies)
I have a list of files all over a file system e.g.
/home/1/foo/bar.x
/www/sites/moose/foo.txtI'm looking for strings in these files and want to replace each occurrence with a replacement string, e.g.
if I find: '#@!^\&@ in any of the files I want to replace it with: 655#@11, etc.
There... (2 Replies)
Hi,
I want to grep multiple patterns from multiple files and save to multiple outputs. As of now its outputting all to the same file when I use this command.
Input : 108 files to check for 390 patterns to check for. output I need to 108 files with the searched patterns.
Xargs -I {} grep... (3 Replies)
Hi All,
I am starting a service which will redirect its out put into 2 logs say A and B.
Now for succesful startup of the service i need to search pattern1 in log A and pattern2 in log B which are writen continuosly.
Now my requirement is to find the patterns in the increasing logs A and B... (19 Replies)
Hi,
I have scenario like below and need to search for multiple patterns
Eg:
Test
Time Started= secs
Time Ended = secc
Green test
Test
Time Started= secs
Time Ended = secc
Green test
Output:
I need to display the text starting with Test and starting with Time... (2 Replies)
Im having an issue when trying to replace the first column with a new set of values in multiple files. The results from the following code only replaces the files with the last set of values in val.txt. I want to replace all the files with all the values.
for date in {1..31}
do
for val in... (1 Reply)
Discussion started by: ncwxpanther
1 Replies
LEARN ABOUT ULTRIX
bdiff
bdiff(1) General Commands Manual bdiff(1)Name
bdiff - big file differential comparator
Syntax
bdiff file1 file2 [n] [-s]
Description
The command is used to find lines that must be changed in two files to bring them into agreement. Its purpose is to allow processing of
files that are too large for
The command ignores lines common to the beginning of both files, splits the remainder of each file into n-line segments, and invokes upon
corresponding segments. The value of n is 3500 by default. If the optional third argument is given and if it is numeric, it is used as
the value for n. This is useful in those cases in which 3500-line segments are too large for causing it to fail.
The output of the command is the same as the output of the command: line numbers are adjusted to account for the segmenting of the files to
make it look as if the files had been processed whole. Note that because of the segmenting of the files, does not necessarily find the
smallest sufficient set of file differences.
If either file1 or file2 is -, the standard input is read. The optional -s (silent) argument specifies that no diagnostics are to be
printed by However, this does not suppress possible exclamations by If both optional arguments are specified, they must appear in the order
indicated above.
Options-s Suppresses normal diagnostic messages.
Diagnostics
Use for explanations.
Files
/tmp/bd?????
See Alsodiff(1)bdiff(1)