Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Finding matching patterns in two files

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Finding matching patterns in two files
# 1  
Old 06-23-2015
Finding matching patterns in two files
Hi,

I have requirement to find the matching patterns of two files in Unix. One file is the log file and the other is the error list file. If any pattern in the log file matches the list of errors in the error list file, then I would need to find the counts of the match.

For example,

Log file has the below information:

Code:
sftp> cd /home/development/project
sftp> MPUT ABCD_*.zip
Uploading ABCD_20150220_20150220012345.zip to /home/development/project/ABCD_20150220_20150220012345.zip
Couldn't get handle: Permission denied

error list file has the below information:

Code:
12,Permission denied,FTP error
14,No connection,Conection error

As "Permission denied" matches in both the files, the output count should be 1.

I tried to use grep -Ff option but I am getting the below error:

grep: illegal option -- F
grep: illegal option -- f
Usage: grep -hblcnsviw pattern file . . .

Please help
# 2  
Old 06-23-2015
You cannot use grep because you are matching only a part (the 2nd field) of the lines in the error file.
While awk can easily get the 2nd field.
Code:
awk -F, 'NR==FNR {s=s sep $2; sep="|"; next} $0~s' errorfile logfile

Counting the matches (like grep -c):
Code:
awk -F, 'NR==FNR {s=s sep $2; sep="|"; next} $0~s {c++} END {print c+0}' errorfile logfile

# 3  
Old 06-23-2015
Above is fine if you need just one lump sum for all errors that have occurred. If you need/want it by error text, try
Code:
awk -F, 'NR==FNR {E[$2]; next} {for (e in E) if ($0 ~ e) R[e]++} END {for (r in R) print r, ":", R[r]}'errorfile logfile
No connection : 1
Permission denied : 1

# 4  
Old 06-24-2015
Hi Rudi,

I tried according to your code but I am getting error:

Code:
 $ awk -F, 'NR==FNR {E[$2]; next} {for (e in E) if ($0 ~ e) R[e]++} END {for (r in R) print r, ":", R[r]}' errorfile logfile
awk: syntax error near line 1
awk: illegal statement near line

Do I need to declare any thing prior to executing this command ?
# 5  
Old 06-24-2015
Hello Bobby_2000,

As you haven't mentioned your OS details, on a Solaris/SunOS system, change awk to /usr/xpg4/bin/awk , /usr/xpg6/bin/awk , or nawk.

Thanks,
R. Singh
# 6  
Old 06-24-2015
Thank you Ravinder.I used nawk and it works.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Delete patterns matching

Delete patterns matching OS version: RHEL 7.3 Shell : Bash I have a file like below (pattern.txt). I need to delete all lines starting with the following words (words separated by comma below) and ) character. LOGGING, NOCOMPRESS, TABLESPACE , PCTFREE, INITRANS, MAXTRANS, STORAGE,... (3 Replies)
Discussion started by: John K
3 Replies

2. Shell Programming and Scripting

Bash - Find files excluding file patterns and subfolder patterns

Hello. For a given folder, I want to select any files find $PATH1 -f \( -name "*" but omit any files like pattern name ! -iname "*.jpg" ! -iname "*.xsession*" ..... \) and also omit any subfolder like pattern name -type d \( -name "/etc/gconf/gconf.*" -o -name "*cache*" -o -name "*Cache*" -o... (2 Replies)
Discussion started by: jcdole
2 Replies

3. Shell Programming and Scripting

Find files not matching multiple patterns and then delete anything older than 10 days

Hi, I have multiple files in my log folder. e.g: a_m1.log b_1.log c_1.log d_1.log b_2.log c_2.log d_2.log e_m1.log a_m2.log e_m2.log I need to keep latest 10 instances of each file. I can write multiple find commands but looking if it is possible in one line. m file are monthly... (4 Replies)
Discussion started by: wahi80
4 Replies

4. Shell Programming and Scripting

Finding files which contains anyone from the given patterns

Hi All, I need help as i am not able to create shell script for a scenario. i have 3000 numbers and want to search all the files which contain anyone of the above pattern. the files are in folder structure. Thanks and Regards Rishi Dhawan (14 Replies)
Discussion started by: Rishi26
14 Replies

5. Shell Programming and Scripting

Matching patterns

I have a file name in $f. If $f has "-" at the beginning, or "=", or does not have extension ".ry" or ".xt" or ".dat" then cerr would not be empty. Tried the following but having some problems. set cerr = `echo $f | awk '/^-|=|!.ry|!.xt|!.dat/'` (4 Replies)
Discussion started by: kristinu
4 Replies

6. Shell Programming and Scripting

Finding several patterns and outputting 4 lines after

I an trying to parse a file looking for pattern1, or pattern2, or pattern3 and when found print that line and the next 4 lines after it. I am using korn shell script on AIX and grep -A isn't available. (1 Reply)
Discussion started by: daveisme
1 Replies

7. Shell Programming and Scripting

AWK: matching patterns in 2 different files

In a directory, there are two different file extensions (*.txt and *.xyz) having similar names of numerical strings (*). The (*.txt) contains 5000 multiple files and the (*.xyz) also contains 5000 multiple files. Each of the files has around 4000 rows and 8 columns, with several unique string... (5 Replies)
Discussion started by: asanjuan
5 Replies

8. Shell Programming and Scripting

Finding patterns in a file

Hi, I have a file with 3 columns and I want to find when the average number of rows on column 3 is a certain value. The output will be put into another file indicating the range. Here is what I mean (file is tab separated): hhm1 2 0 hhm1 4 0.5 hhm1 6 0.3 hhm1 8 -1.4... (2 Replies)
Discussion started by: kylle345
2 Replies

9. Shell Programming and Scripting

finding duplicate files by size and finding pattern matching and its count

Hi, I have a challenging task,in which i have to find the duplicate files by its name and size,then i need to take anyone of the file.Then i need to open the file and find for more than one pattern and count of that pattern. Note:These are the samples of two files,but i can have more... (2 Replies)
Discussion started by: jerome Sukumar
2 Replies

10. Shell Programming and Scripting

Finding patterns through out all subdir

Hi all experts, here is a problem which i would appreciate ur expertise. I need to do this: Eg. Find a number: 1234567 which i dunno which file and which folder I do know which main folder it is in but it is hidden deep within a lot of subdir. Is it possible to find the file? + output... (4 Replies)
Discussion started by: unnerdy
4 Replies
Login or Register to Ask a Question