Shell script to get duplicate string


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Shell script to get duplicate string
# 1  
Old 02-12-2016
Error Shell script to get duplicate string

Hi All,

I have a requirement where I have to get the duplicate string count and uniq error message. Below is my file:
Code:
 Rejected - Error on table TableA, column ColA.
Error String 1.
 Rejected - Error on table TableA, column ColB.
Error String 2.
 Rejected - Error on table TableA, column ColB.
Error String 2.
 Rejected - Error on table TableA, column ColA.
Error String 1.
 Rejected - Error on table TableA, column ColA.
Error String 1.

Here I need to get the 1st line with table and column detail and second line for the error String. There are hundreds of the same error like this in my file ....so I need the count of the error for a particular error on table.column and the unique {error message} with table.column details.

It should come something like:
Code:
 Rejected - Error on table TableA, column ColA.
Error String 1.                                                   3 Rows
 Rejected - Error on table TableA, column ColB.
Error String 2.                                                   2 Rows

Thanks in advance

Last edited by Don Cragun; 02-12-2016 at 04:00 AM.. Reason: Add CODE tags.
# 2  
Old 02-12-2016
Code:
awk '{x=$0; getline; x=x"\n"$0; a[x]++}END{for (i in a){print i, a[i], "rows"}}' file

This User Gave Thanks to balajesuri For This Post:
# 3  
Old 02-12-2016
Working perfect

Thanks Balajesuri. The code is working perfect.
If you don't mind please explain the code as I am new in AWK.Smilie
# 4  
Old 02-12-2016
Quote:
Originally Posted by Deekhari
Thanks Balajesuri. The code is working perfect.
If you don't mind please explain the code as I am new in AWK.Smilie
Hello Deekhari,

Following may help you in same.
Code:
awk '{x=$0;            ###### Taking complete line into a variable named x
getline;               ###### getline is an awk keyword to move the cursor to next line.
x=x"\n"$0;             ###### taking value of variable named x with it's previous value and then new line and then complete line($0).
a[x]++}                ###### Here creating an array named a whose index is x and incrementing it's each occurances too.
END{                   ###### starting END section here.
for (i in a){          ###### starting a for loop in array named a
print i, a[i], "rows"} ###### here once we get into array a then printing the value of vafriable i, value of array a as a[i](which is nothing but count of rows) and string rows then. 
}' file                ###### calling Input_file here.

Let me know if you have any queries on same.


Thanks,
R. Singh
This User Gave Thanks to RavinderSingh13 For This Post:
# 5  
Old 02-12-2016
Thanks a lot Ravinder. One more question.
My file is a sqlldr log in which many other msg and content is there.
Actually I was getting the exact msg which contain only error by below command(that msg content is in my first question)

Code:
grep -A 1 "Rejected - Error" $FILE_NAME | cut -d':' -f2

From this command I am getting my desired error messages which i was initially redirecting to a file and then the awk function will use it to give me error count with error msg.

Now I don't want to create a separate file and want to directly pass the strings to the awk function. How I can do that.

Thanks again for the help.

Last edited by Don Cragun; 02-12-2016 at 04:02 AM.. Reason: Add CODE and ICODE tags; s/AWK/awk/g.
# 6  
Old 02-12-2016
Try a small adaption to balajesuri's proposal:
Code:
awk '/^ Rejected/ {x=$0; getline; x=x"\n"$0; a[x]++}END{for (i in a){print i, a[i], "rows"}}' file
 Rejected - Error on table TableA, column ColB.
Error String 2. 2 rows
 Rejected - Error on table TableA, column ColA.
Error String 1. 3 rows

This User Gave Thanks to RudiC For This Post:
# 7  
Old 02-12-2016
Here's a version that does not require AWK

Code:
perl -nle 'if($id){$save{"$id\n$_"}++}; ($id)=/^( Rejected.*$)/; END{for(keys %save){print "$_ $save{$_} rows"}}' deekhari.log

Code:
 Rejected - Error on table TableA, column ColB.
Error String 2. 2 rows
 Rejected - Error on table TableA, column ColA.
Error String 1. 3 rows

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Shell script to compare two files for duplicate..??

Hi , I had a requirement to compare two files whether the two files are same or different .... like(files contaisn of two columns each) file1.txt 121343432213 1234 64564564646 2345 343423424234 2456 file2.txt 121343432213 1234 64564564646 2345 31231313123 3455 how to... (2 Replies)
Discussion started by: hemanthsaikumar
2 Replies

2. Shell Programming and Scripting

Highlighting duplicate string on a line

Hi all I have a grep written to pull out values; below (in the code snip-it) is an example of the output. What I'm struggling to do, and looking for assistance on, is identifying the lines that have duplicate strings. For example 74859915K74859915K in the below is 74859915K repeated twice but... (8 Replies)
Discussion started by: brighty
8 Replies

3. Shell Programming and Scripting

Remove not only the duplicate string but also the keyword of the string in Perl

Hi Perl users, I have another problem with text processing in Perl. I have a file below: Linux Unix Linux Windows SUN MACOS SUN SUN HP-AUX I want the result below: Unix Windows SUN MACOS HP-AUX so the duplicate string will be removed and also the keyword of the string on... (2 Replies)
Discussion started by: askari
2 Replies

4. Shell Programming and Scripting

find duplicate string in many different files

I have more than 100 files like this: SVEAVLTGPYGYT 2 SVEGNFEETQY 10 SVELGQGYEQY 28 SVERTGTGYT 6 SVGLADYNEQF 21 SVGQGYEQY 32 SVKTVLGYEQF 2 SVNNEQF 12 SVRDGLTNSPLH 3 SVRRDREGLEQF 11 SVRTSGSYEQY 17 SVSVSGSPLQETQY 78 SVVHSTSPEAF 59 SVVPGNGYT 75 (4 Replies)
Discussion started by: xshang
4 Replies

5. Shell Programming and Scripting

How to get the last value from a string in shell script?

I have to get the last value from a string, below is the example string String -> Here is the test string 12233 O/P -> 12233 String -> Hello world 500 O/P -> 500 String -> 300 O/P -> 300 Please help (2 Replies)
Discussion started by: vel4ever
2 Replies

6. UNIX for Dummies Questions & Answers

Comparing a String variable with a string literal in a Debian shell script

Hi All, I am trying to to compare a string variable with a string literal inside a loop but keep getting the ./testifstructure.sh: line 6: #!/bin/sh BOOK_LIST="BOOK1 BOOK2" for BOOK in ${BOOK_LIST} do if then echo '1' else echo '2' fi done Please use next... (1 Reply)
Discussion started by: daveu7
1 Replies

7. Shell Programming and Scripting

how to duplicate an output in shell script

how to duplicate an output from a shell command? for example: `date` will give the current date to the console. I want this to be displayed in console and also parallely store it in file or variable. user1@solaris4:~> date Tue Feb 28 17:48:31 EST 2012 user1@solaris4:~> date > file ... (3 Replies)
Discussion started by: Arun_Linux
3 Replies

8. Shell Programming and Scripting

Delete duplicate in certain number of string

Hi, do you have awk or sed sommand taht will delete duplicate lines like. sample: server1-log1-14 server1-log2-14 superserver-time-2 superserver-log-2 output: server-log1-14 superserver-time-2 thansk (2 Replies)
Discussion started by: kenshinhimura
2 Replies

9. Shell Programming and Scripting

filtering out duplicate substrings, regex string from a string

My input contains a single word lines. From each line data.txt prjtestBlaBlatestBlaBla prjthisBlaBlathisBlaBla prjthatBlaBladpthatBlaBla prjgoodBlaBladpgoodBlaBla prjgood1BlaBla123dpgood1BlaBla123 Desired output --> data_out.txt prjtestBlaBla prjthisBlaBla... (8 Replies)
Discussion started by: kchinnam
8 Replies

10. Shell Programming and Scripting

Shell Script to Create non-duplicate lists from two lists

File_A contains Strings: a b c d File_B contains Strings: a c z Need to have script written in either sh or ksh. Derive resultant files (File_New_A and File_New_B) from lists File_A and File_B where string elements in File_New_A and File_New_B are listed below. Resultant... (7 Replies)
Discussion started by: mlv_99
7 Replies
Login or Register to Ask a Question