Using grep and a parameter file to return unique values Post: 302894069

Sponsored Content

Top Forums Shell Programming and Scripting Using grep and a parameter file to return unique values Post 302894069 by Lucas_0418 on Sunday 23rd of March 2014 11:52:46 PM

03-24-2014

Registered User

Hi clippertm,
Let's use awk instead of grep, try this, it works fine in my cygwin, hope it could work in your cygwin too.

Code:

awk '$2~/^4=[0-9]+$/{split($2,a,/=/);if(int(a[2])>=0&&int(a[2])<=2000&&$0~/|3078=/&&!b[$2]){b[$2]++;print $0}}' FS='|' *.txt

Code:

$ cat data1.txt
5021=0|4=748|12=ABC|3078=7484561|4102=748
5021=0|4=749|12=ABC|3214=748|3078=7486512
5021=0|4=748|12=DEF|3078=7481564151|855=748
5021=0|4=750|12=ABC|987=748|3078=7481231
5021=0|4=750|12=DEF|3078=41561|6321=748
5021=0|4=750|12=DEF|3078=7812|8412=748
5021=0|4=750|12=DEF|3078=121888|8855=748
5021=0|4=749|12=ABC|3078=12688|2222=748
5021=0|4=748|12=GHI|3078=812135|8745=748
5021=0|4=748|12=ABC|3078=812121|9647=748
5021=0|4=753|12=GHI|7444=748|3078=121888
$ cat data2.txt
5022=0|4=755|12=ABC|3078=7484561|4102=748
5022=0|4=743|12=ABC|3214=748|3078=7486512
5022=0|4=755|12=DEF|3078=7481564151|855=748
5022=0|4=755|12=ABC|987=748|3078=7481231
5022=0|4=749|12=DEF|3078=41561|6321=748
5022=0|4=748|12=DEF|3078=7812|8412=748
5022=0|4=752|12=DEF|3078=121888|8855=748
5022=0|4=740|12=ABC|3078=12688|2222=748
5022=0|4=740|12=GHI|3078=812135|8745=748
5022=0|4=743|12=ABC|3078=812121|9647=748
5022=0|4=752|12=GHI|7444=748|3078=121888
$ awk '$2~/^4=[0-9]+$/{split($2,a,/=/);if(int(a[2])>=0&&int(a[2])<=2000&&$0~/|3078=/&&!b[$2]){b[$2]++;print $0}}' FS='|' *.txt
5021=0|4=748|12=ABC|3078=7484561|4102=748
5021=0|4=749|12=ABC|3214=748|3078=7486512
5021=0|4=750|12=ABC|987=748|3078=7481231
5021=0|4=753|12=GHI|7444=748|3078=121888
5022=0|4=755|12=ABC|3078=7484561|4102=748
5022=0|4=743|12=ABC|3214=748|3078=7486512
5022=0|4=752|12=DEF|3078=121888|8855=748
5022=0|4=740|12=ABC|3078=12688|2222=748

Lucas_0418

View Public Profile for Lucas_0418

Find all posts by Lucas_0418

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Getting Unique values in a file

Hi, I have a file like this: Some_String_Here 123 123 123 321 321 321 3432 3221 557 886 321 321 I would like to find only the unique values in the files and get the following output: Some_String_Here 123 321 3432 3221 557 886 I am trying to get this done using awk. Can someone please...

2. Shell Programming and Scripting

Unique values from a Terabyte File

Hi, I have been dealing with a files only a few gigs until now and was able to get out by using the sort utility. But now, I have a terabyte file which I want to filter out unique values from. I have a server having 8 processor and 16GB RAM with a 5 TB hdd. Is it worthwhile trying to use...

3. UNIX Desktop Questions & Answers

Fetching unique values from file

After giving grep -A4 "feature 1," <file name> I have extracted the following text feature 1, subfeat 2, type 1, subtype 5, dump '30352f30312f323030392031313a33303a3337'H -- "05/01/2009 11:30:37" -- -- ...

4. Shell Programming and Scripting

Comparing 2 files and return the unique lines in first file

Hi, I have 2 files file1 ******** 01-05-09|java.xls| 02-05-08|c.txt| 08-01-09|perl.txt| 01-01-09|oracle.txt| ******** file2 ******** 01-02-09|windows.xls| 02-05-08|c.txt| 01-05-09|java.xls| 08-02-09|perl.txt| 01-01-09|oracle.txt| ********

5. UNIX for Dummies Questions & Answers

Extract Unique Values from file

Hello all, I have a file with following sample data 2009-08-26 05:32:01.65 spid5 Process ID 86:214 owns resources that are blocking processes on Scheduler 0. 2009-08-26 05:32:01.65 spid5 Process ID 86:214 owns resources that are blocking processes on Scheduler 0. 2009-08-26...

6. Shell Programming and Scripting

return a list of unique values of a column from csv format file

Hi all, I have a huge csv file with the following format of data, Num SNPs, 549997 Total SNPs,555352 Num Samples, 157 SNP, SampleID, Allele1, Allele2 A001,AB1,A,A A002,AB1,A,A A003,AB1,A,A ... ... ... I would like to write out a list of unique SNP (column 1). Could you...

7. Shell Programming and Scripting

How to count Unique Values from a file.

Hi I have the following info in a file - <Cell id="25D"/> <Cell id="26A"/> <Cell id="26B"/> <Cell id="26C"/> <Cell id="27A"/> <Cell id="27B"/> <Cell id="27C"/> <Cell id="28A"/> I would like to know how would you go about counting all...

8. Shell Programming and Scripting

compare 2 files and return unique lines in each file (based on condition)

hi my problem is little complicated one. i have 2 files which appear like this file 1 abbsss:aa:22:34:as akl abc 1234 mkilll:as:ss:23:qs asc abc 0987 mlopii:cd:wq:24:as asd abc 7866 file2 lkoaa:as:24:32:sa alk abc 3245 lkmo:as:34:43:qs qsa abc 0987 kloia:ds:45:56:sa acq abc 7805 i...

9. UNIX for Dummies Questions & Answers

Grep to find matching patern and return unique values

Request: grep to find given matching patern and return unique values, eliminate the duplicate values I have to retrieve the unique folder on the below file contents like; /app/oracle/build_lib/pkg320.0_20120927 /app/oracle/build_lib/pkg320.0_20121004_prof...

10. Shell Programming and Scripting

Extracting unique values of a column from a feed file

Hi Folks, I have the below feed file named abc1.txt in which you can see there is a title and below is the respective values in the rows and it is completely pipe delimited file ,. ...

LEARN ABOUT PLAN9

split

SPLIT(1)						      General Commands Manual							  SPLIT(1)

NAME

       split - split a file into pieces

SYNOPSIS

       split [ option ...  ] [ file ]

DESCRIPTION

       Split reads file (standard input by default) and writes it in pieces of 1000 lines per output file.  The names of the output files are xaa,
       xab, and so on to xzz.  The options are

       -n     Split into n-line pieces.

       -e expression
	      File divisions occur at each line that matches a regular expression; see regexp(6).  Multiple -e options may appear.   If  a  subex-
	      pression	of  expression is contained in parentheses (...), the output file name is the portion of the line which matches the subex-
	      pression.

       -f stem
	      Use stem instead of x in output file names.

       -s suffix
	      Append suffix to names identified under -e.

       -x     Exclude the matched input line from the output file.

       -i     Ignore case in option -e; force output file names (excluding the suffix) to lower case.

SOURCE

       /sys/src/cmd/split.c

SEE ALSO

       sed(1), awk(1) grep(1), regexp(6)

																	  SPLIT(1)