Unix/Linux Go Back    


Shell Programming and Scripting BSD, Linux, and UNIX shell scripting — Post awk, bash, csh, ksh, perl, php, python, sed, sh, shell scripts, and other shell scripting languages questions here.

Grep multiple keywords from a file

Shell Programming and Scripting


Tags
grep

Closed    
 
Thread Tools Search this Thread Display Modes
    #1  
Old Unix and Linux 05-12-2017   -   Original Discussion by Loc
Loc Loc is offline
Registered User
 
Join Date: Feb 2017
Last Activity: 17 May 2017, 1:06 PM EDT
Posts: 12
Thanks: 0
Thanked 0 Times in 0 Posts
Grep multiple keywords from a file

I have a script that will search for a keyword in all the log files. It work just fine.


Code:
 
 LOG_FILES={ "/Sandbox/logs/*" }
 for file in ${LOG_FILES[@]}; do
   grep $1 $file
done

This only works for 1 keyword. What if I want to search for more then 1 keywords, say 4 or maybe even 10 keywords.
So I would enter:

Code:
./myfile.sh 120.2.1.1 Chrome 400 POST

It would check for the IP address, chrome, 400 and POST keywords in all the log files. I see on the internet, they have something like this:

Code:
egrep 'A1|A2|A3' filename

This specify that it will grep the output that is either A1 or A2 or A3 but I would like to grep the output if it is A1 AND A2 AND A3. I'm not sure how to go about doing this? Any help would be greatly appreciated.

tks
Sponsored Links
    #2  
Old Unix and Linux 05-12-2017   -   Original Discussion by Loc
wbport wbport is offline
Registered User
 
Join Date: Sep 2013
Last Activity: 15 November 2017, 2:47 PM EST
Location: Mississippi
Posts: 121
Thanks: 34
Thanked 18 Times in 18 Posts
The way to do that is to create a list of files containing A1 and passing them to a grep that will look at only those files. To search for A3 the second grep would have to produce a list as well which you could save or pipe through.

The program xargs is very helpful for this in that it reads an input file or piped input, runs your command, and places every line of your input file as the last argument(s) of your command.

Code:
grep -l A1 {2nd parm for star name files if necessary} |tee my.list1
xargs <my.list1 grep -l A2 |xargs grep A3

If you will be doing a number of searches on files containing your first search term, creating a file with a list should be useful.

HTH
Sponsored Links
    #3  
Old Unix and Linux 05-12-2017   -   Original Discussion by Loc
Don Cragun's Unix or Linux Image
Don Cragun Don Cragun is offline Forum Staff  
Administrator
 
Join Date: Jul 2012
Last Activity: 20 November 2017, 12:28 AM EST
Location: San Jose, CA, USA
Posts: 10,661
Thanks: 572
Thanked 3,734 Times in 3,186 Posts
In addition to what wbport suggested, there are some other alternatives in cases where the strings you are looking for all occur on a single line in your log files...

If the strings you are searching for always appear in the same order on a line (i.e., A1 followed by A2 followed by A3), you can use:

Code:
grep 'A1.*A2.*A3' file

If they can appear in any order, try one of the following:

Code:
egrep 'A1.*A2.*A3|A1.*A3.*A2|A2.*A1.*A3|A2.*A3.*A1|A3.*A1.*A2|A3.*A2.*A1' file
grep -e 'A1.*A2.*A3' -e 'A1.*A3.*A2' -e 'A2.*A1.*A3' -e 'A2.*A3.*A1' -e 'A3.*A1.*A2' -e 'A3.*A2.*A1' file

or create a file containing your REs (named REfile in this example) containing:

Code:
A1.*A2.*A3
A1.*A3.*A2
A2.*A1.*A3
A2.*A3.*A1
A3.*A1.*A2
A3.*A2.*A1

and then use:

Code:
grep -f REfile file

Note that if the strings you're searching for do not all appear on a single line, the code suggested by wbport will only print lines containing A3 (not lines containing just A1 or A2). If you want to print all lines containing A1, A2, or A3 but only print those lines if the file contains all three strings, you need an extra level of grep to print the final results (i.e. read some of your files four times when looking for 3 strings; three times to find the names of files that contain each string and a final time to print all three strings (using OR instead of AND). Or, you can use awk to read the file once, gather lines that match any of your strings and keep track of which strings have been found, and then print all matching lines at the end if all of your strings have been found. If this is what you need, we can help you figure out a way to do that, but I'm not going to try to do it here if you don't need to do that. Your description of your problem isn't clear as to the extent of the problem you're trying to solve.
    #4  
Old Unix and Linux 05-12-2017   -   Original Discussion by Loc
RudiC RudiC is offline Forum Staff  
Moderator
 
Join Date: Jul 2012
Last Activity: 19 November 2017, 9:29 AM EST
Location: Aachen, Germany
Posts: 11,625
Thanks: 320
Thanked 3,617 Times in 3,322 Posts
Not sure what you're after. If it's the names of the files containig ALL of the keywords, try

Code:
grep -E 'A1|B2|C3' file[1-3] | cut -d: -f1 | uniq -c | grep '^ *3' | cut -d" " -f8
file1

You will need to adapt the keyword count (3) of the last grep, and the field number (8) of the last cut.


Or

Code:
grep -E 'A1|B2|C3' file[1-3] | awk -F: '{CNT[$1]++} END {for (c in CNT) if (CNT[c] == 3) print c}'
file1

Or

Code:
SRCH='A1|B2|C3'; grep -E $SRCH file[1-3] | awk -F: -vSRCH=$SRCH '{CNT[$1]++} END {for (c in CNT) if (CNT[c] == split (SRCH,T,"|")) print c}'


Last edited by RudiC; 05-12-2017 at 05:18 PM..
Sponsored Links
    #5  
Old Unix and Linux 05-12-2017   -   Original Discussion by Loc
AbelLuis AbelLuis is offline
Registered User
 
Join Date: Jan 2014
Last Activity: 10 August 2017, 11:24 PM EDT
Posts: 28
Thanks: 9
Thanked 10 Times in 10 Posts
Quote:
Originally Posted by Loc View Post
It would check for the IP address, chrome, 400 and POST keywords in all the log files. I see on the internet, they have something like this:

Code:
egrep 'A1|A2|A3' filename

This specify that it will grep the output that is either A1 or A2 or A3 but I would like to grep the output if it is A1 AND A2 AND A3. I'm not sure how to go about doing this? Any help would be greatly appreciated.

tks
Hi, you may try this:


Code:
RE="/$A1/ && /$A2/ && /$A3/"
eval awk "'" $RE "'" filename

The first line is for the expansion of parameters; the second, eval, parses and execute the built command.

Regards.
Sponsored Links
    #6  
Old Unix and Linux 05-12-2017   -   Original Discussion by Loc
wbport wbport is offline
Registered User
 
Join Date: Sep 2013
Last Activity: 15 November 2017, 2:47 PM EST
Location: Mississippi
Posts: 121
Thanks: 34
Thanked 18 Times in 18 Posts
If all three need to be on the same line but can be in any order, just pipe the output from the first grep (without using -l) through the 2nd and 3rd. It won't be necessary to search for all permutations of what you need.
Sponsored Links
    #7  
Old Unix and Linux 05-12-2017   -   Original Discussion by Loc
drl's Unix or Linux Image
drl drl is offline Forum Advisor  
Registered Voter
 
Join Date: Apr 2007
Last Activity: 19 November 2017, 4:02 PM EST
Location: Saint Paul, MN USA / BSD, CentOS, Debian, OS X, Solaris
Posts: 2,206
Thanks: 241
Thanked 413 Times in 357 Posts
Hi.

Here are a few alternatives; One is the use of shell expressions, the other is a relative of grep, agrep:

Code:
#!/usr/bin/env ksh
#!/usr/bin/env bash
#!/usr/bin/env zsh

# @(#) s1       Demonstrate match multiple strings AND and OR, agrep, shell.

# Utility functions: print-as-echo, print-line-with-visual-space.
LC_ALL=C ; LANG=C ; export LC_ALL LANG
pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }
em() { pe "$*" >&2 ; }
db() { ( printf " db, ";for _i;do printf "%s" "$_i";done;printf "\n" ) >&2 ; }
db() { : ; }
C=$HOME/bin/context && [ -f $C ] && . $C dixf agrep

FILE=${1-data1}
pl " Data file $FILE:"
cat $FILE

pl " With agrep, \"and\":a b c :"
agrep "a;b;c" $FILE

pl " With agrep, \"or\":a b c :"
agrep "a,b,c" $FILE

# A pattern-list is a list of one or more patterns
# separated from each other with a & or |. A & signifies
# that all patterns must be matched whereas | requires
# that only one pattern be matched. 

# shopt -s extglob
# shopt extglob

pl " With shell expressions, \"and\", a b c ksh (bash, zsh fail):"
while read line
do
  if [[ $line == @(*a*&*b*&*c*) ]]
  then
    pe "Matched line: $line"
  fi
done < $FILE

pl " With shell expressions, \"or\", a b c ksh (zsh fail):"
while read line
do
  if [[ $line == @(*a*|*b*|*c*) ]]
  then
    pe "Matched line: $line"
  fi
done < $FILE

pl " Some detail about agrep:"
dixf agrep

exit $?

producing:

Code:
$ ./s1

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 3.16.0-4-amd64, x86_64
Distribution        : Debian 8.7 (jessie) 
ksh 93u+
dixf (local) 1.42
agrep - ( /usr/bin/agrep, 2012-04-12 )

-----
 Data file data1:
also
abracadabra
chock full of beans
hello, world.
silent

-----
 With agrep, "and":a b c :
abracadabra
chock full of beans

-----
 With agrep, "or":a b c :
also
abracadabra
chock full of beans

-----
 With shell expressions, "and", a b c ksh (bash, zsh fail):
Matched line: abracadabra
Matched line: chock full of beans

-----
 With shell expressions, "or", a b c ksh (zsh fail):
Matched line: also
Matched line: abracadabra
Matched line: chock full of beans

-----
 Some detail about agrep:
agrep   search a file for a string or regular expression, with... (man)
Path    : /usr/bin/agrep
Version : - ( /usr/bin/agrep, 2012-04-12 )
Type    : ELF 64-bit LSB shared object, x86-64, version 1 (S ...)
Help    : probably available with -h
Repo    : Debian 8.7 (jessie)

The agrep will be much faster for files of more than trivial size.

Best wishes ... cheers, drl
Sponsored Links
Closed

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Find keywords in multiple log files dellanicholson Shell Programming and Scripting 13 06-07-2016 10:52 PM
How to grep keywords? khchong Shell Programming and Scripting 3 04-18-2014 07:56 AM
Grep from multiple patterns multiple file multiple output Diya123 Shell Programming and Scripting 3 11-02-2013 01:38 PM
Grep Keywords one by one dashing201 Shell Programming and Scripting 3 07-18-2013 08:11 AM
finding keywords in many files using grep raghulrajan UNIX for Dummies Questions & Answers 5 03-14-2012 12:05 PM



All times are GMT -4. The time now is 05:34 AM.