Login or Register to Ask a Question and Join Our Community


UNIX for Beginners Questions & Answers

Code to get unique values

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums UNIX for Beginners Questions & Answers Code to get unique values
# 1  
Old 05-10-2018
Code to get unique values
Hello All,

I am trying to write a script which returns me clientID,programId,userID indicated in bold from the below log files.Log file is having many such data , iam just presenting sample .



Sample Log file.

Code:
[2018-05-09 08:01:24,251] hostname 1525867288264 UA:MP:EP491418 http-nio-8080-exec-11 ERROR Get Price Failed! EP491418 , UA , MP (AbstractLocal
PricingServiceV1)
[2018-05-09 08:01:24,251] hostname 1525867288264 UA:MP:EP491418 http-nio-8080-exec-11 ERROR Exception while search for products (CartService)
com.xx.xx.pricing.pricingapi.exception.InvalidCartRequestException: Error: Failed to calculate US Tax. Error Number: '2' Error Description: 'A valid city/state or zip code required.' | Params: Address={
  "line1" : "6 allee de Mauregard",
  "line2" : "",
  "city" : "Gif sur Yvette",
  "stateCode" : "Essonne",
  "postalCode" : "91190",
  "countryCode" : "US"
} taxType=Sales

Expected Output is
Code:
UA:MP:EP491418 
UA:MP:HR722540
UA:MP:EG359458

zGrep Pattern

Code:
zgrep -B1 "Failed to calculate US Tax" log-gr_base.log.2018-05-08.gz  | zgrep "CartService" |sort --unique


Output from the above zgrep pattern

Code:
[2018-05-08 09:43:32,135] hostname 1525787013878 UA:MP:HR722540 http-nio-8080-exec-14 ERROR Exception while search for products (CartService)
[2018-05-08 09:43:41,636] hostname 1525787029223 UA:MP:HR722540 http-nio-8080-exec-45 ERROR Exception while search for products (CartService)
[2018-05-08 11:24:07,182] hostname 1525793054329 UA:MP:EG359458 http-nio-8080-exec-32 ERROR Exception while search for products (CartService)
[2018-05-08 11:24:26,796] hostname 1525793068993 UA:MP:EG359458 http-nio-8080-exec-45 ERROR Exception while search for products (CartService)
[2018-05-08 11:24:30,903] hostname 1525793073833 UA:MP:EG359458 http-nio-8080-exec-7 ERROR Exception while search for products (CartService)
[2018-05-08 11:24:54,760] hostname 1525793099189 UA:MP:EG359458 http-nio-8080-exec-5 ERROR Exception while search for products (CartService)
[2018-05-08 11:25:16,744] hostname 1525793124418 UA:MP:EG359458 http-nio-8080-exec-18 ERROR Exception while search for products (CartService)
[2018-05-08 11:25:21,622] hostname 1525793122821 UA:MP:EG359458 http-nio-8080-exec-2 ERROR Exception while search for products (CartService)


Please help in getting
1. Only the unique clientID,programId,userID. Am not familiar with sed for getting this result.


Any help is appreciated.


Moderator's Comments:
Mod Comment Please use CODE tags correctly as required by forum rules! Tagging your entire text is NOT the way to go!
Last edited by RudiC; 05-10-2018 at 06:53 AM.. Reason: Corrected wrong tagging... AGAIN!
# 2  
Old 05-10-2018
Quote:
Originally Posted by nextStep
.
.
.


Moderator's Comments:
Mod Comment Please use CODE tags correctly as required by forum rules! Tagging your entire text is NOT the way to go!


---------- Post updated at 07:34 AM ---------- Previous update was at 07:24 AM ----------

I will abide this rule.It was mistakenly put.
Hmmm - are you sure?
# 3  
Old 05-10-2018
Hello nextStep,

Could you please try following and let me know if this helps.
Code:
awk 'match($0,/UA:MP:[a-zA-Z]+[0-9]+/) && !a[substr($0,RSTART,RLENGTH)]++{print substr($0,RSTART,RLENGTH)}'  Input_file

Thanks,
R. Singh
This User Gave Thanks to RavinderSingh13 For This Post:
nextStep
# 4  
Old 05-10-2018
I am sorry for that.Hit the send button quickly

---------- Post updated at 05:11 AM ---------- Previous update was at 05:07 AM ----------

Hello Ravinder,

That code worked, much thanks. Pasting the output below.

Code:
 zgrep -B1 "Failed to calculate US Tax" logFile  | zgrep "CartService" |awk 'match($0,/UA:MP:[a-zA-Z]+[0-9]+/) && !a[substr($0,RSTART,RLENGTH)]++{print substr($0,RSTART,RLENGTH)}'

UA:MP:HR722540
UA:MP:EG359458

Could you please explain the logic you have used above.
# 5  
Old 05-10-2018
Here are some alternatives you could try:
Code:
zgrep -B1 "Failed to calculate US Tax" log-gr_base.log.2018-05-08.gz | grep "CartService" | grep -Eo "[[:upper:]]{2}:[[:upper:]]{2}:[[:upper:]]{2}[[:digit:]]{6}" | sort -u

or
Code:
zgrep -B1 "Failed to calculate US Tax" log-gr_base.log.2018-05-08.gz | sed -n '/CartService/s/.*\(..:..:........\).*/\1/p' | sort -u

Code:
gzcat log-gr_base.log.2018-05-08.gz | awk '/Failed to calculate US Tax/{print p}{p=$5}' | sort -u

or
Code:
gzcat log-gr_base.log.2018-05-08.gz | awk '/Failed to calculate US Tax/ && !A[p]++{print p}{p=$5}'

Last edited by Scrutinizer; 05-10-2018 at 07:24 AM..
# 6  
Old 05-10-2018
Quote:
Originally Posted by nextStep
Hello Ravinder,
That code worked, much thanks. Pasting the output below.
Code:
 zgrep -B1 "Failed to calculate US Tax" logFile  | zgrep "CartService" |awk 'match($0,/UA:MP:[a-zA-Z]+[0-9]+/) && !a[substr($0,RSTART,RLENGTH)]++{print substr($0,RSTART,RLENGTH)}'
UA:MP:HR722540
UA:MP:EG359458

Could you please explain the logic you have used above.
Hello nextStep,

Following is the explanation may help you on same.

Code:
awk '
match($0,/UA:MP:[a-zA-Z]+[0-9]+/) && !a[substr($0,RSTART,RLENGTH)]++{  ##Using match utility of awk to match REGEX UA:MP: alphabets till all occurrences then till all digits all occurrences AND(condition) to check if array a should NOT have index of substring of RSTART and RLENGTH variables then print substring of that line whose starting point is value of RSTART and final points RLENGTH.
   print substr($0,RSTART,RLENGTH)                                     ##Printing the substring whose value starts from RSTART to RLENGTH values.
}'  Input_file                                                                      ##Mentioning Input_file name here.

Thanks,
R. Singh
This User Gave Thanks to RavinderSingh13 For This Post:
nextStep
# 7  
Old 05-10-2018
Thanks.gzcat was not recogonized .Could you please help me to understand

Code:
sed -n '/CartService/s/.*\(..:..:........\).*/\1/p' | sort -u
grep -Eo "[[:upper:]]{2}:[[:upper:]]{2}:[[:upper:]]{2}[[:digit:]]{6}"
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Print count of unique values

Hello experts, I am converting a number into its binary output as : read n echo "obase=2;$n" | bc I wish to count the maximum continuous occurrences of the digit 1. Example : 1. The binary equivalent of 5 = 101. Hence the output must be 1. 2. The binary... (3 Replies)
Discussion started by: H squared
3 Replies

2. Shell Programming and Scripting

unique entry add values

Hi, I have a file with 3 columns ABC 3 1 ABC 5 1 XYZ 4 2 DEF 3 2 DEF 4 1 DEF 6 1 MNO 5 5 JKL 3 2 JKL 4 2 PQR 12 1 For each unique entry in column 1 I want to add values in column 2 and column3 o/p ABC 8 2 XYZ 4 2 (1 Reply)
Discussion started by: Diya123
1 Replies

3. Shell Programming and Scripting

calculating unique strings values

Hi, Im looking for a script which will calculate the unique strings column 2 & 3 values in a log as mentioned in example eg:- bag 12 12 bag 18 15 bags 15 13 bags 15 14 blazer 24 24 blazer 33 32 boots 19 15 Result should be:- bag 30 27 bags 30 27... (9 Replies)
Discussion started by: Paulwintech
9 Replies

4. Shell Programming and Scripting

How to count Unique Values from a file.

Hi I have the following info in a file - <Cell id="25D"/> <Cell id="26A"/> <Cell id="26B"/> <Cell id="26C"/> <Cell id="27A"/> <Cell id="27B"/> <Cell id="27C"/> <Cell id="28A"/> I would like to know how would you go about counting all... (4 Replies)
Discussion started by: Prega
4 Replies

5. Shell Programming and Scripting

print unique values of a column and sum up the corresponding values in next column

Hi All, I have a file which is having 3 columns as (string string integer) a b 1 x y 2 p k 5 y y 4 ..... ..... Question: I want get the unique value of column 2 in a sorted way(on column 2) and the sum of the 3rd column of the corresponding rows. e.g the above file should return the... (6 Replies)
Discussion started by: amigarus
6 Replies

6. UNIX for Dummies Questions & Answers

Extract Unique Values from file

Hello all, I have a file with following sample data 2009-08-26 05:32:01.65 spid5 Process ID 86:214 owns resources that are blocking processes on Scheduler 0. 2009-08-26 05:32:01.65 spid5 Process ID 86:214 owns resources that are blocking processes on Scheduler 0. 2009-08-26... (5 Replies)
Discussion started by: simonsimon
5 Replies

7. UNIX Desktop Questions & Answers

Fetching unique values from file

After giving grep -A4 "feature 1," <file name> I have extracted the following text feature 1, subfeat 2, type 1, subtype 5, dump '30352f30312f323030392031313a33303a3337'H -- "05/01/2009 11:30:37" -- -- ... (1 Reply)
Discussion started by: shivi707
1 Replies

8. Shell Programming and Scripting

Unique values from a Terabyte File

Hi, I have been dealing with a files only a few gigs until now and was able to get out by using the sort utility. But now, I have a terabyte file which I want to filter out unique values from. I have a server having 8 processor and 16GB RAM with a 5 TB hdd. Is it worthwhile trying to use... (6 Replies)
Discussion started by: Legend986
6 Replies

9. Shell Programming and Scripting

Getting Unique values in a file

Hi, I have a file like this: Some_String_Here 123 123 123 321 321 321 3432 3221 557 886 321 321 I would like to find only the unique values in the files and get the following output: Some_String_Here 123 321 3432 3221 557 886 I am trying to get this done using awk. Can someone please... (5 Replies)
Discussion started by: Legend986
5 Replies

10. Shell Programming and Scripting

to retrieve unique values

Hi all, I have a 10.txt file. In this file 2 words are present by name Active and Inactive and these words are repeated 7000 times. I want to take the unique 2 words from this 7000 lines. Thanks Mahalakshmi.A (3 Replies)
Discussion started by: mahalakshmi
3 Replies
Login or Register to Ask a Question