Sponsored Content
Top Forums Shell Programming and Scripting Search multiple patterns in multiple files Post 302489550 by Chubler_XL on Thursday 20th of January 2011 05:47:58 PM
Old 01-20-2011
Do you have room to unzip all the files somewhere?

Unzip all files and append to one big file (your havelist), then use awk to check each line of your havelist against file1.txt:

Code:
( find . -name "*processed" -print | xargs gunzip -c ) > /scratch/havelist
awk ' NR == FNR { F[i++]=$0; next}
    { for(i in F)if(index($0, F[i])) delete F[i]; }
    END { for(i in F) print "number "F[i]" not found." } ' file1.txt havelist > outputfile.txt
rm /scratch/havelist

awk may chew a big chunk of memory as it loads the 5,000,000 numbers into it's array, and you won't get any output till it's done, but it will be much quicker that your original attempt.
Some more efficency can be gained if you can say that each processed file line only matches to 1 number from file1.txt (ie a line in processed file dosn't contain 2 or more numbers you are looking for).

But, with record counts this large you should really be considering using a database rather than flat/zipped files.

---------- Post updated at 08:47 AM ---------- Previous update was at 08:15 AM ----------

If you just don't have room to extract to a scratchfile you can do the search through a pipe:

Code:
( find . -name "*processed" -print | xargs gunzip -c ) | awk ' NR == FNR { F[i++]=$0; next}
    { for(i in F)if(index($0, F[i])) delete F[i]; }
    END { for(i in F) print "number "F[i]" not found." } ' file1.txt - > outputfile.txt


Last edited by Chubler_XL; 01-20-2011 at 06:55 PM.. Reason: Orginal solution listed processed lines not in file1.txt
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

How to parameterize multiple search patterns and generate a new file

I have one file: 123*100*abcd*10 123*101*abcd*-29*def 123*100*abcd*-10 123*102*abcd*-105*asd I would like to parameterize the search patterns in the following way so that the user could dynamically change the search pattern. *100* and *- (ie *minus) *102* and *- The output that is... (6 Replies)
Discussion started by: augustinep
6 Replies

2. Shell Programming and Scripting

Perl: Match a line with multiple search patterns

Hi I'm not very good with the serach patterns and I'd need a sample how to find a line that has multiple patterns. Say I want to find a line that has "abd", "123" and "QWERTY" and there can be any characters or numbers between the serach patterns, I have a file that has thousands of lines and... (10 Replies)
Discussion started by: Juha
10 Replies

3. Shell Programming and Scripting

Perl - How to search a text file with multiple patterns?

Good day, great gurus, I'm new to Perl, and programming in general. I'm trying to retrieve a column of data from my text file which spans a non-specific number of lines. So I did a regexp that will pick out the columns. However,my pattern would vary. I tried using a foreach loop unsuccessfully.... (2 Replies)
Discussion started by: Sp3ck
2 Replies

4. Shell Programming and Scripting

search multiple patterns

I have two lists in a file that look like a b b a e f c d f e d c I would like a final list a b c d e f I've tried multiple grep and awk but can't get it to work (8 Replies)
Discussion started by: godzilla07
8 Replies

5. Shell Programming and Scripting

How to search Multiple patterns in unix

Hi, I tried to search multiple pattern using awk trans=1234 reason=LN MISMATCH rec=`awk '/$trans/ && /'"$reason"'/' file` whenevr i tried to run on command promt it is executing but when i tried to implment same logic in shell script,it is failing i.e $rec is empty ... (6 Replies)
Discussion started by: ns64110
6 Replies

6. Shell Programming and Scripting

Search & Replace: Multiple Strings / Multiple Files

I have a list of files all over a file system e.g. /home/1/foo/bar.x /www/sites/moose/foo.txtI'm looking for strings in these files and want to replace each occurrence with a replacement string, e.g. if I find: '#@!^\&@ in any of the files I want to replace it with: 655#@11, etc. There... (2 Replies)
Discussion started by: spacegoose
2 Replies

7. Shell Programming and Scripting

Grep from multiple patterns multiple file multiple output

Hi, I want to grep multiple patterns from multiple files and save to multiple outputs. As of now its outputting all to the same file when I use this command. Input : 108 files to check for 390 patterns to check for. output I need to 108 files with the searched patterns. Xargs -I {} grep... (3 Replies)
Discussion started by: Diya123
3 Replies

8. Shell Programming and Scripting

Search patterns in multiple logs parallelly.

Hi All, I am starting a service which will redirect its out put into 2 logs say A and B. Now for succesful startup of the service i need to search pattern1 in log A and pattern2 in log B which are writen continuosly. Now my requirement is to find the patterns in the increasing logs A and B... (19 Replies)
Discussion started by: Girish19
19 Replies

9. Shell Programming and Scripting

Search Multiple patterns and display

Hi, I have scenario like below and need to search for multiple patterns Eg: Test Time Started= secs Time Ended = secc Green test Test Time Started= secs Time Ended = secc Green test Output: I need to display the text starting with Test and starting with Time... (2 Replies)
Discussion started by: weknowd
2 Replies

10. UNIX for Beginners Questions & Answers

Issue with search and replacing multiple items in multiple files

Im having an issue when trying to replace the first column with a new set of values in multiple files. The results from the following code only replaces the files with the last set of values in val.txt. I want to replace all the files with all the values. for date in {1..31} do for val in... (1 Reply)
Discussion started by: ncwxpanther
1 Replies
PHARDATA.EXTRACTTO(3)							 1						     PHARDATA.EXTRACTTO(3)

PharData::extractTo - Extract the contents of a tar/zip archive to a directory

SYNOPSIS
bool PharData::extractTo (string $pathto, [string|array $files], [bool $overwrite = false]) DESCRIPTION
Extract all files within a tar/zip archive to disk. Extracted files and directories preserve permissions as stored in the archive. The optional parameters allow optional control over which files are extracted, and whether existing files on disk can be overwritten. The sec- ond parameter files can be either the name of a file or directory to extract, or an array of names of files and directories to extract. By default, this method will not overwrite existing files, the third parameter can be set to true to enable overwriting of files. This method is similar to ZipArchive.extractTo(3). PARAMETERS
o $pathto - Path to extract the given files to o $files - The name of a file or directory to extract, or an array of files/directories to extract o $overwrite - Set to TRUE to enable overwriting existing files RETURN VALUES
returns TRUE on success, but it is better to check for thrown exception, and assume success if none is thrown. ERRORS
/EXCEPTIONS Throws PharException if errors occur while flushing changes to disk. EXAMPLES
Example #1 A PharData.extractTo(3) example <?php try { $phar = new PharData('myphar.tar'); $phar->extractTo('/full/path'); // extract all files $phar->extractTo('/another/path', 'file.txt'); // extract only file.txt $phar->extractTo('/this/path', array('file1.txt', 'file2.txt')); // extract 2 files only $phar->extractTo('/third/path', null, true); // extract all files, and overwrite } catch (Exception $e) { // handle errors } ?> SEE ALSO
Phar.extractTo(3). PHP Documentation Group PHARDATA.EXTRACTTO(3)
All times are GMT -4. The time now is 12:14 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy