How do you [e]grep for multiple values within multiple files? Post: 302608765

Sponsored Content

Special Forums UNIX Desktop Questions & Answers How do you [e]grep for multiple values within multiple files? Post 302608765 by drl on Sunday 18th of March 2012 01:37:35 PM

03-18-2012

Registered User

Hi.

Here is a shell script that encapsulates an awk script. The awk reads each line of each file once to ensure that each file contains at least one match to the multiple strings.

This example uses 2 strings and looks through 4 files, but can be easily extended for more strings. One advantage is that each file is read only once. Another is that when all strings have matched at least once, the operation skips to the next file. For a small number of files, the awk probably is no faster than multiple greps, but for the grep, you can imagine that for 3000 files, if the first keyword appears in 2000 files, then those 2000 files will be read at least twice, and so on. On the other hand, awk will use more CPU time than grep. For simple alpha-numeric strings, fgrep might be slightly faster than plain grep.

Both this approach and the suggestion from bartus11 will require some work to fit a specific problem. Here is the awk:

Code:

#!/usr/bin/env bash

# @(#) s1	Demonstrate single-pass to find multiple strings in files.

pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }
db() { ( printf " db, ";for _i;do printf "%s" "$_i";done;printf "\n" ) >&2 ; }
db() { : ; }
C=$HOME/bin/context && [ -f $C ] && $C awk

FILES="data*"
pl " Data files:"
for i in $FILES
do
  pe
  pe " Data file $i:"
  cat $i
done

pl " Results, files must contain \"a\" AND \"b\":"
# "c" is not used for testing, and can be eliminated.
awk '
BEGIN	{ a = b = c = 0 }
FNR == 1	{ a = b = c = 0 }
/a/	{ a++ }
/b/	{ b++ }
/c/	{ c++ }
a>0 && b>0	{ print FILENAME ; nextfile }
' $FILES

exit 0

producing:

Code:

% ./s1

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 2.6.26-2-amd64, x86_64
Distribution        : Debian GNU/Linux 5.0.8 (lenny) 
bash GNU bash 3.2.39
awk GNU Awk 3.1.5

-----
 Data files:

 Data file data1:
a
b

 Data file data2:
b
a
c

 Data file data3:
c
a

 Data file data4:
a b
c
d

-----
 Results, files must contain "a" AND "b":
data1
data2
data4

See man pages for details.

Best wishes ... cheers, drl

drl

View Public Profile for drl

Find all posts by drl

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

grep command to find multiple strings in multiple lines in a file.

I want to search files (basically .cc files) in /xx folder and subfolders. Those files (*.cc files) must contain #include "header.h" AND x() function. I am writing it another way to make it clear, I wanna list of *.cc files that have 'header.h' & 'x()'. They must have two strings, header.h...

2. UNIX for Dummies Questions & Answers

Grep multiple strings in multiple files using single command

Hi, I will use below command for grep single string ("osuser" is search string) ex: find . -type f | xarg grep -il osuser but i have one more string "v$session" here i want to grep in which file these two strings are present. any help is appreciated, Thanks in advance. Gagan

3. Shell Programming and Scripting

Parsing common values across multiple files

Hi All, I have multiple (5+) text files with single columns and I would like to grep the common values across all the text files and parse it to a new file. All the values are numerical. Please let me know how to do it using awk.

4. Shell Programming and Scripting

Grep multiple strings in multiple files

Hi, every one! I have a file with multiple strings. file1 ATQRGNE ASQGVKFTE ASSQYRDRGGLET SPEQGARSDE ASSRDFTDT ASSYSGGYE ASSYTRLWNTGE ASQGHNTD PSLGGGNQPQH SLDRDSYNEQF I want to grep each string in hundreds of files in the same directory, further, I want to find out the string...

5. Shell Programming and Scripting

Find values in multiple csv files

Hi, I'd like to find the values of certain fields in multiple csv files stored in 1 directory based upon an input search string. An fgrep returns the complete record, I only want certain fields. Thanks in advance for your help. Perry

6. UNIX for Dummies Questions & Answers

Grep multiple values

This for i in /dev/disco/*;do lvdisplay $i|grep -i size;done Return me every size of lvm in vg "disco" I want to return me,the size and the name of lvm,how to do this? Thanks

7. Shell Programming and Scripting

Grep from multiple patterns multiple file multiple output

Hi, I want to grep multiple patterns from multiple files and save to multiple outputs. As of now its outputting all to the same file when I use this command. Input : 108 files to check for 390 patterns to check for. output I need to 108 files with the searched patterns. Xargs -I {} grep...

8. Shell Programming and Scripting

Get both common and missing values from multiple files

Hi, I have 5 files with two columns. I need to merge all the 5 files based on column 1. If any of them are missing then corresponding 2nd column should be substituted by missing value. I know hoe to do this for 2 files. but how can I implement for 5 files. I tried this based on 5 files but it...

9. Shell Programming and Scripting

Reading multiple values from multiple lines and columns and setting them to unique variables.

Hello, I would like to ask for help with csh script. An example of an input in .txt file is below, the number of lines varies from file to file and I have 2 or 3 columns with values. I would like to read all the values (probably one by one) and set them to independent unique variables that...

10. Shell Programming and Scripting

Grep strings on multiple files and output to multiple files

Hi All, I want to use egrep on multiple files and the results should be output to multiple files. I am using the below code in my shell script(working in Ksh shell). However with this code I am not attaining the desired results. #!/bin/ksh ( a="/path/file1" b="path/file2" for file in...

LEARN ABOUT DEBIAN

apsfilter-bug

APSFILTER-BUG(1)					    BSD General Commands Manual 					  APSFILTER-BUG(1)

NAME

     apsfilter-bug -- create a half-automatic bug report for apsfilter

SYNOPSIS

     apsfilter-bug

OPTIONS

     none

DESCRIPTION

     With the apsfilter-bug script you can automatically create a bug report for apsfilter(1).	A template report file is created which you can
     edit with your favourite editor ( emacs(1) by default, or whatever is entered in the environment variable EDITOR ).

     You will see a couple of lines starting with '#'; these lines will be removed before the report is sent. So if you want to use the '#' char-
     acter at the beginning of a line, just indent it with a space.

     A very important piece of the bug report is the debugging log created by aps2file(1), probably with a command like
	   aps2file -D -o /dev/null [-P...] [-Z...] [input] 2> log.txt
     Unless you don't even have a chance to create a debugging log, it is crucial for serious bug hunting.

     After you have filled in as much (useful) information as possible, you must save the file under its original name, then exit your editor.

     If you invoked apsfilter-bug by accident, or if you need some additional system information to include into the bug report, just exit your
     editor without saving the file.

NOTES

     In addition to the stuff you've edited, some more information is appended to the bug report automatically:

     apsfilter version	  7.2.6-stable
     ghostscript version  (from 'gs --version')
     system id		  (from 'uname -a')
     configure options	  --prefix=/usr --sysconfdir=/etc --mandir=/usr/share/man --with-awk=/usr/bin/awk --with-sendmail=/usr/sbin/sendmail
     shell executable	  /bin/bash
     awk executable	  /usr/bin/awk
     sendmail executable  /usr/sbin/sendmail

FILES

     /usr/bin/aps2file	script to create debugging output

SEE ALSO

     apsfilter(1), aps2file(1)

BUGS

     See apsfilter software center - http://www.apsfilter.org/ - for new versions, bugfixes and known bugs.

     Please use the new tool apsfilter-bug(1) to release bug- or problem reports. It automatically presents you a form in an editor window which
     asks you some standard questions.	If you save and quit the editor session, then this report is sent automatically via e-mail to the proper
     apsfilter mailinglist.

     If apsfilter fails to print something or prints it in a way you wouldn't expect and you want to report an apsfilter error then please save
     the debugging output of one print session using the new aps2file(1) utility by typing aps2file -D -Z options file > /dev/null 2> file.debug
     and including the debugging output in the file file.debug into the edit session of the apsfilter-bug utility, so that it is included into the
     mail to the apsfilter mailinglist.

     Please note that you need to run /bin/sh (Bourne Shell), bash or a compatible shell, so that the above mentioned output redirection works.
     Under C-shell (/bin/csh) or tcsh it would't work. If you don't know, then simply make sure you use the Bournce shell by typing /bin/sh or
     bash, then you should have no problems with redirection of stdout and stderr (> /dev/null 2> file.debug).

DOCUMENTATION

     See official apsfilter homepage
     http://www.apsfilter.org/handbook.html   Apsfilter Handbook including the Frequently Asked Questions (FAQ)

USER FORUM

     Please send questions to the official apsfilter help channel apsfilter-help@apsfilter.org.  The above section BUGS and the file
     HOWTO-BUGREPORTS tells you how to report bugs.  If you want to know how to troubleshoot your apsfilter installation, please read the manpage
     aps2file(1) and apsfilter-bug(1) as well as the Apsfilter Handbook carefully.

HISTORY

     The apsfilter-bug manpage has been written by Michael LoBin <phallobst@web.de> and first appeared in apsfilter V 7.1.0.

BSD
								   Dec 26, 2001 							       BSD