Using grep and a parameter file to return unique values

03-20-2014

Registered User

65, 23

Join Date: Feb 2014

Last Activity: 22 February 2017, 10:30 PM EST

Location: Shanghai, PRC

Posts: 65

Thanks Given: 15

Thanked 23 Times in 22 Posts

Sorry clippertm,
I made a mistake, delete the double plus.

Code:

grep -h '4=[745-755]' *.* | grep '3078=' | sort -u -t'|' -k2,2

?
If [745-755] is not available, use this:

Code:

grep -h '4=\(74[5-9]\|75[0-5]\)' a | grep '3078=' | sort -u -t'|' -k2,2

This User Gave Thanks to Lucas_0418 For This Post:

Lucas_0418

View Public Profile for Lucas_0418

Find all posts by Lucas_0418

03-20-2014

Registered User

19, 2

Join Date: Mar 2014

Last Activity: 19 December 2016, 12:44 AM EST

Posts: 19

Thanks Given: 10

Thanked 2 Times in 2 Posts

Hi Lucas,

No problem.

Thanks, the count loop seems to work, but the whole command does not seem to return unique occurences anymore, but all of them

clippertm

View Public Profile for clippertm

Find all posts by clippertm

03-20-2014

Registered User

474, 160

Join Date: Feb 2011

Last Activity: 22 May 2020, 9:47 AM EDT

Posts: 474

Thanks Given: 51

Thanked 160 Times in 135 Posts

Your first problem is that you need to match the patterns in your pattern file with the second field. You cannot just use grep -f list.txt. You really need to use awk or perl. You can then pattern-match on the two fields you are interested in without a stray 254=47587 in, say, the last field, matching your patterns file.

Something I was not sure about. Are you after the first instance of 4=475 and 3078= AND the first instance of 4=476 and 3078= etc etc rather than the first instance of ANY 4=xxx and 3078=? In which case you will have to loop through your patterns file anyway.

I am not going to offer any code because I don't really know awk and my perl is rusty. Good luck

Andrew

This User Gave Thanks to apmcd47 For This Post:

apmcd47

View Public Profile for apmcd47

Find all posts by apmcd47

03-20-2014

Registered User

19, 2

Join Date: Mar 2014

Last Activity: 19 December 2016, 12:44 AM EST

Posts: 19

Thanks Given: 10

Thanked 2 Times in 2 Posts

Hi Apm,

Well thank you for your reply, but I only know grep

I have never used awk or perl

---------- Post updated at 04:28 AM ---------- Previous update was at 04:24 AM ----------

I have come up with this:

Quote:

grep '4=[0-1][0-9][0-9][0-9]' *.* | grep '3078=' | sort -u -t'|' -k 2,2

However it still returns all the occurrences, not only the first one

---------- Post updated at 04:39 AM ---------- Previous update was at 04:28 AM ----------

Actually, it seems that the simple:

Quote:

grep -h -m 1 '4=[1-2000]' *.* | grep '3078=' > output.txt

does the trick!

Really simple.. am I missing anything?

clippertm

View Public Profile for clippertm

Find all posts by clippertm

03-20-2014

Registered User

65, 23

Join Date: Feb 2014

Last Activity: 22 February 2017, 10:30 PM EST

Location: Shanghai, PRC

Posts: 65

Thanks Given: 15

Thanked 23 Times in 22 Posts

Sorry clippertm,
I think the problem of output all occurences is that we used the wildcard *.*
so maybe we must use sort after grep
Or use perl or awk as apmcd47 say, u know, both could solve the problem.

We know the wildcard *.* make the grep consider every first match in every file both are first occurence, may cat *.*|grep could work, but I am not able to test it when I am in a bus.

And sorry for my not very good English, let me check what's your desired output again.
a. line has 4=745 and 3078=
b. line only has 4=475 not 3078=
c. line only has 3078= not 4=475

For your addtional question:
\([1-9]\|[1-9][0-9]\|[1-9][0-9][0-9]\|1[0-9][0-9][0-9]\|2000\)

Last edited by Lucas_0418; 03-20-2014 at 07:03 AM..

This User Gave Thanks to Lucas_0418 For This Post:

Lucas_0418

View Public Profile for Lucas_0418

Find all posts by Lucas_0418

03-20-2014

Registered User

19, 2

Join Date: Mar 2014

Last Activity: 19 December 2016, 12:44 AM EST

Posts: 19

Thanks Given: 10

Thanked 2 Times in 2 Posts

Nope it does not work

Quote:

grep -h -m 1 '4=[1-2000]' *.* | grep '3078=' > output.txt

returns less lines than it should...

---------- Post updated at 04:44 AM ---------- Previous update was at 04:43 AM ----------

Hi Lucas,

I do not how to use awk/perl at all

---------- Post updated at 04:54 AM ---------- Previous update was at 04:44 AM ----------

Does anybody know how to check a range from 1 to 2000? Not 0001, 0002, 0003 etc. but 1, 2, 3

clippertm

View Public Profile for clippertm

Find all posts by clippertm

03-20-2014

Registered User

5,091, 1,931

Join Date: May 2012

Last Activity: 15 July 2020, 4:46 AM EDT

Location: Simplicity

Posts: 5,091

Thanks Given: 565

Thanked 1,931 Times in 1,668 Posts

With awk:

Code:

awk -F "|" '
# hash the search list
NR==FNR {L[$1]=0; next}
# now procede with the data files
# print if the following is true
($4~/^3078=/ && ($2 in L) && L[$2]++==0)
' searchlist.txt datafile1.txt datafile2.txt

Search the 3078= everywhere:

Code:

awk -F "|" '
# hash the search list
NR==FNR {L[$1]=0; next}
# now procede with the data files
# print if the following is true
(/|3078=/ && ($2 in L) && L[$2]++==0)
' searchlist.txt datafile1.txt datafile2.txt

---------- Post updated at 06:51 AM ---------- Previous update was at 05:35 AM ----------

Quote:

Originally Posted by Lucas_0418

What's your environment, found that your code could work in my cygwin.

Code:

grep -h '4=[745-755]++' *.* | grep '3078=' | sort -u -t'|' -k2,2

May you could put all the pattern in a file, then use option -m of grep to get the first match line.

Code:

while read a
do
grep -h -m 1 "$a" *.*
done< yourpatterfile

grep -m1 exits at every 1st match per file.
awk is much more flexible:

Code:

awk -F "|" -v low=745 -v high=755 '
# build the Lookup hash
BEGIN {for (i=low; i<=high; i++) L["4="i]}
# main loop
# if in Lookup hash and if a field begins with 3078=
($2 in L) && /|3078=/ {
  print
  # delete from the Lookup hash
  delete L[$2]
}
' datafile*.txt

Last edited by MadeInGermany; 03-20-2014 at 08:10 AM..

This User Gave Thanks to MadeInGermany For This Post:

MadeInGermany

View Public Profile for MadeInGermany

Find all posts by MadeInGermany

Shell Programming and Scripting

Using grep and a parameter file to return unique values

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Extracting unique values of a column from a feed file

Discussion started by: punpun66

2. UNIX for Dummies Questions & Answers

Grep to find matching patern and return unique values

Discussion started by: Siva SQL

3. Shell Programming and Scripting

compare 2 files and return unique lines in each file (based on condition)

Discussion started by: anurupa777

4. Shell Programming and Scripting

How to count Unique Values from a file.

Discussion started by: Prega

5. Shell Programming and Scripting

return a list of unique values of a column from csv format file

Discussion started by: phoeberunner

6. UNIX for Dummies Questions & Answers

Extract Unique Values from file

Discussion started by: simonsimon

7. Shell Programming and Scripting

Comparing 2 files and return the unique lines in first file

Discussion started by: shekhar_v4

8. UNIX Desktop Questions & Answers

Fetching unique values from file

Discussion started by: shivi707

9. Shell Programming and Scripting

Unique values from a Terabyte File

Discussion started by: Legend986

10. Shell Programming and Scripting

Getting Unique values in a file

Discussion started by: Legend986