Code to get unique values


 
Thread Tools Search this Thread
Top Forums UNIX for Beginners Questions & Answers Code to get unique values
# 8  
Old 05-10-2018
Sometimes you need to use zcat instead of gzcat

The sed statement:
Code:
sed -n                         # -n: Do not automatically print records
/CartService/                  # Look for lines that contain "CartService"
s/.*\(..:..:........\).*/\1/p  # Discard everything but the last pattern on the line that looks like:
                               # 2 chars - colon - 2 chars - colon - 6 chars 
                               # (the .* gobbles up the first pattern on the line)
                               # \1 is a so-called back reference, it refers to the pattern within the escaped brackets
                               # So what it all does - in short - is look for "CartService" and then
                               # replace the entire line with the pattern and if it can do that, print the line

The grep statement:
-E means use Extended Regular Expression.
-o means print only the matching part of the line (this option is non-standard and is not available in every grep utility).
[[:upper:]]{2}:[[:upper:]]{2}:[[:upper:]]{2}[[:digit:]]{6} means 2 uppercase letters - colon - 2 uppercase letters - colon - 2 uppercase letters + 6 digits.

Last edited by Scrutinizer; 05-10-2018 at 08:00 AM..
# 9  
Old 05-10-2018
Thanks a lot. Of the two i found [[:upper:]]more convenient and easy.

---------- Post updated at 06:49 AM ---------- Previous update was at 06:12 AM ----------

Hi Scrutinizer

Of the two methods, is it possible to get the date as well.There is a new requirement requested after seeing the output. Sorry for the late update.


2018-05-09 08:01:24,251
# 10  
Old 05-10-2018
Quote:
Originally Posted by nextStep
Of the two methods, is it possible to get the date as well.There is a new requirement requested after seeing the output. Sorry for the late update.
2018-05-09 08:01:24,251
Hello nextStep,

Following awk may help you on same too.
Code:
awk 'match($0,/UA:MP:[a-zA-Z]+[0-9]+/) && !a[substr($0,RSTART,RLENGTH)]++{sub(/\[/,"",$1);sub(/\]/,"",$2);print $1,$2,substr($0,RSTART,RLENGTH-2)}'   Input_file

Adding a non-one liner form of solution too now.
Code:
awk '
match($0,/UA:MP:[a-zA-Z]+[0-9]+/) && !a[substr($0,RSTART,RLENGTH)]++{
   sub(/\[/,"",$1);
   sub(/\]/,"",$2);
   print $1,$2,substr($0,RSTART,RLENGTH-2)
}'   Input_file

Thanks,
R. Singh
This User Gave Thanks to RavinderSingh13 For This Post:
# 11  
Old 05-10-2018
That also worked. Thanks a lot Ravinder.
# 12  
Old 05-10-2018
Quote:
Originally Posted by nextStep
Thanks a lot. Of the two i found [[:upper:]]more convenient and easy.

---------- Post updated at 06:49 AM ---------- Previous update was at 06:12 AM ----------

Hi Scrutinizer

Of the two methods, is it possible to get the date as well.There is a new requirement requested after seeing the output. Sorry for the late update.


2018-05-09 08:01:24,251
You could try:
Code:
zgrep -B1 "Failed to calculate US Tax" log-gr_base.log.2018-05-08.gz | grep "CartService" | grep -Eo '.{4}-.{2}-[^]]*|[[:upper:]]{2}:[[:upper:]]{2}:[[:upper:]]{2}[[:digit:]]{6}'

or
Code:
zgrep -B1 "Failed to calculate US Tax" log-gr_base.log.2018-05-08.gz | grep "CartService" | grep -Eo '.{4}-.{2}-[^]]*|.{2}:.{2}:.{8}'

The reason that a simple dot suffices in this case is because the first match will get matched by the other expression. The vertical bar (|) means "OR" (alternation).
[^]] means NOT "]"

--
To get both results on one line:
Code:
zgrep -B1 "Failed to calculate US Tax" log-gr_base.log.2018-05-08.gz | grep "CartService" | grep -Eo '.{4}-.{2}-[^]]*|.{2}:.{2}:.{8}' | paste - -


Last edited by Scrutinizer; 05-10-2018 at 09:27 AM..
This User Gave Thanks to Scrutinizer For This Post:
# 13  
Old 05-10-2018
Thanks Scrutinizer
# 14  
Old 05-10-2018
If you want to also sort -u, some sorts can do this (but also that is not standard):
Code:
sort -u -k3,3

But your best bet might be:
Code:
gunzip -c log-gr_base.log.2018-05-08.gz | awk '/Failed to calculate US Tax/ && !A[p]++{print q OFS p} {split ($0,F,/[][]/); p=$5; q=F[2]}'


Last edited by Scrutinizer; 05-10-2018 at 12:05 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Print count of unique values

Hello experts, I am converting a number into its binary output as : read n echo "obase=2;$n" | bc I wish to count the maximum continuous occurrences of the digit 1. Example : 1. The binary equivalent of 5 = 101. Hence the output must be 1. 2. The binary... (3 Replies)
Discussion started by: H squared
3 Replies

2. Shell Programming and Scripting

unique entry add values

Hi, I have a file with 3 columns ABC 3 1 ABC 5 1 XYZ 4 2 DEF 3 2 DEF 4 1 DEF 6 1 MNO 5 5 JKL 3 2 JKL 4 2 PQR 12 1 For each unique entry in column 1 I want to add values in column 2 and column3 o/p ABC 8 2 XYZ 4 2 (1 Reply)
Discussion started by: Diya123
1 Replies

3. Shell Programming and Scripting

calculating unique strings values

Hi, Im looking for a script which will calculate the unique strings column 2 & 3 values in a log as mentioned in example eg:- bag 12 12 bag 18 15 bags 15 13 bags 15 14 blazer 24 24 blazer 33 32 boots 19 15 Result should be:- bag 30 27 bags 30 27... (9 Replies)
Discussion started by: Paulwintech
9 Replies

4. Shell Programming and Scripting

How to count Unique Values from a file.

Hi I have the following info in a file - <Cell id="25D"/> <Cell id="26A"/> <Cell id="26B"/> <Cell id="26C"/> <Cell id="27A"/> <Cell id="27B"/> <Cell id="27C"/> <Cell id="28A"/> I would like to know how would you go about counting all... (4 Replies)
Discussion started by: Prega
4 Replies

5. Shell Programming and Scripting

print unique values of a column and sum up the corresponding values in next column

Hi All, I have a file which is having 3 columns as (string string integer) a b 1 x y 2 p k 5 y y 4 ..... ..... Question: I want get the unique value of column 2 in a sorted way(on column 2) and the sum of the 3rd column of the corresponding rows. e.g the above file should return the... (6 Replies)
Discussion started by: amigarus
6 Replies

6. UNIX for Dummies Questions & Answers

Extract Unique Values from file

Hello all, I have a file with following sample data 2009-08-26 05:32:01.65 spid5 Process ID 86:214 owns resources that are blocking processes on Scheduler 0. 2009-08-26 05:32:01.65 spid5 Process ID 86:214 owns resources that are blocking processes on Scheduler 0. 2009-08-26... (5 Replies)
Discussion started by: simonsimon
5 Replies

7. UNIX Desktop Questions & Answers

Fetching unique values from file

After giving grep -A4 "feature 1," <file name> I have extracted the following text feature 1, subfeat 2, type 1, subtype 5, dump '30352f30312f323030392031313a33303a3337'H -- "05/01/2009 11:30:37" -- -- ... (1 Reply)
Discussion started by: shivi707
1 Replies

8. Shell Programming and Scripting

Unique values from a Terabyte File

Hi, I have been dealing with a files only a few gigs until now and was able to get out by using the sort utility. But now, I have a terabyte file which I want to filter out unique values from. I have a server having 8 processor and 16GB RAM with a 5 TB hdd. Is it worthwhile trying to use... (6 Replies)
Discussion started by: Legend986
6 Replies

9. Shell Programming and Scripting

Getting Unique values in a file

Hi, I have a file like this: Some_String_Here 123 123 123 321 321 321 3432 3221 557 886 321 321 I would like to find only the unique values in the files and get the following output: Some_String_Here 123 321 3432 3221 557 886 I am trying to get this done using awk. Can someone please... (5 Replies)
Discussion started by: Legend986
5 Replies

10. Shell Programming and Scripting

to retrieve unique values

Hi all, I have a 10.txt file. In this file 2 words are present by name Active and Inactive and these words are repeated 7000 times. I want to take the unique 2 words from this 7000 lines. Thanks Mahalakshmi.A (3 Replies)
Discussion started by: mahalakshmi
3 Replies
Login or Register to Ask a Question