Login or Register to Ask a Question and Join Our Community


UNIX for Dummies Questions & Answers

Grep to find matching patern and return unique values

 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Grep to find matching patern and return unique values
# 1  
Old 05-03-2013
HP Grep to find matching patern and return unique values
Request: grep to find given matching patern and return unique values, eliminate the duplicate values

I have to retrieve the unique folder on the below file contents like;

/app/oracle/build_lib/pkg320.0_20120927
/app/oracle/build_lib/pkg320.0_20121004_prof
/app/oracle/build_lib/pkg320.0_20121004
/app/oracle/build_lib/pkg320.0_20121011
/app/oracle/build_lib/pkg330.0_20121018

Here is the sample file: saw_batch_drive_cmd
Code:
# == Regular Build == Pre-processing
/app/oracle/build_lib/pkg320.0_20120927/ddl/scj_build_pre_rt.sh SID TEST_NUM 20120927 320.0 N N
# == Regular Build == Processing
/app/oracle/build_lib/pkg320.0_20120927/ddl/scj_build.sh SID TEST_NUM 20120927 320.0 N Y
# == Regular Build == Pre-processing - PROFIT!!!! - change goes into DEC MR pkg313
#/app/oracle/build_lib/pkg320.0_20121004_prof/ddl/scj_build_pre_rt.sh SID TEST_NUM 20121004_prof 320.0 N N
# == Regular Build == Pre-processing
/app/oracle/build_lib/pkg320.0_20121004/ddl/scj_build_pre_rt.sh SID TEST_NUM 20121004 320.0 N N
# == Regular Build == Processing
/app/oracle/build_lib/pkg320.0_20121004/ddl/scj_build.sh SID TEST_NUM 20121004 320.0 N Y
# == Regular Build == Pre-processing
/app/oracle/build_lib/pkg320.0_20121011/ddl/scj_build_pre_rt.sh SID TEST_NUM 20121011 320.0 N N
# == Regular Build == Processing
/app/oracle/build_lib/pkg330.0_20121011/ddl/scj_build.sh SID TEST_NUM 20121011 320.0 N Y
# == Regular Build == Pre-processing
/app/oracle/build_lib/pkg330.0_20121018/ddl/scj_build_pre_rt.sh SID TEST_NUM 20121018 320.0 N N
# == Regular Build == Processing
/app/oracle/build_lib/pkg330.0_20121018/ddl/scj_build.sh SID TEST_NUM 20121018 320.0 N Y

# 2  
Old 05-03-2013
You have a file "saw_batch_drive_cmd", from which you want to get unique directories. Will the file contents be always similar to the sample one you have posted?

Try the following commands on the file -

Code:
cat saw_batch_drive_cmd | grep "/app/oracle" | cut -d"/" -f 1-5 | uniq | sed "s/^\#//"
/app/oracle/build_lib/pkg320.0_20120927
/app/oracle/build_lib/pkg320.0_20121004_prof
/app/oracle/build_lib/pkg320.0_20121004
/app/oracle/build_lib/pkg320.0_20121011
/app/oracle/build_lib/pkg330.0_20121011
/app/oracle/build_lib/pkg330.0_20121018

Let me know if this is what you were looking for.

Thanks
Bhushan Pathak
Last edited by BhushanPathak; 05-03-2013 at 03:17 AM.. Reason: updated the command, missed a pipe last time
This User Gave Thanks to BhushanPathak For This Post:
Siva SQL
# 3  
Old 05-03-2013
No need to cat a file for grep thats useless use of cat Smilie

try below
Code:
 
awk -F"/" '/oracle/{A[$5]=$1"/"$2"/"$3"/"$4"/"$5}END{for(i in A){sub("#","",A[i]);print A[i]}}' filename

This User Gave Thanks to vidyadhar85 For This Post:
Siva SQL
# 4  
Old 05-03-2013
Quote:
Originally Posted by BhushanPathak
You have a file "saw_batch_drive_cmd", from which you want to get unique directories. Will the file contents be always similar to the sample one you have posted?

Try the following commands on the file -

Code:
cat saw_batch_drive_cmd | grep "/app/oracle" | cut -d"/" -f 1-5 | uniq | sed "s/^\#//"
/app/oracle/build_lib/pkg320.0_20120927
/app/oracle/build_lib/pkg320.0_20121004_prof
/app/oracle/build_lib/pkg320.0_20121004
/app/oracle/build_lib/pkg320.0_20121011
/app/oracle/build_lib/pkg330.0_20121011
/app/oracle/build_lib/pkg330.0_20121018

Let me know if this is what you were looking for.

Thanks
Bhushan Pathak
Thanks Bhushan Pathak Smilie

Please explain me what is that sed part of the command do.

---------- Post updated at 06:36 PM ---------- Previous update was at 06:23 PM ----------

Quote:
Originally Posted by vidyadhar85
No need to cat a file for grep thats useless use of cat Smilie

try below
Code:
 
awk -F"/" '/oracle/{A[$5]=$1"/"$2"/"$3"/"$4"/"$5}END{for(i in A){sub("#","",A[i]);print A[i]}}' filename


The command is very useful.

I would really appreciate, if you can expain this command as how it works.
# 5  
Old 05-03-2013
Another approach:
Code:
awk '{gsub(/\/ddl.*|#/,x)}!A[$0]++&&/oracle/' file

Explanation:
Code:
awk '
        {
                gsub( /\/ddl.*|#/, x)   # Remove every chars followed by "/ddl" or hash #
        }
        !A[$0]++ && /oracle/            # Print if non duplicate associative array and pattern: oracle matches
' file

# 6  
Old 05-03-2013
The sed part removes the leading # character from the list of files, if present
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Find unique values but only in column 1

Hi All, Does anyone have any suggestions/examples of how i could show only lines where the first field is not duplicated. If the first field is listed more than once it shouldnt be shown even if the other columns make it unique. Example file : 876,RIBDA,EC2 876,RIBDH,EX7 877,RIBDF,E28... (4 Replies)
Discussion started by: mutley2202
4 Replies

2. Shell Programming and Scripting

How to merge two files with unique values matching.?

I have one script as below: #!/bin/ksh Outputfile1="/home/OutputFile1.xls" Outputfile2="/home/OutputFile2.xls" InputFile1="/home/InputFile1.sql" InputFile2="/home/InputFile2.sql" echo "Select hobby, class, subject, sports, rollNumber from Student_Table" >> InputFile1 echo "Select rollNumber... (3 Replies)
Discussion started by: Sharma331
3 Replies

3. Shell Programming and Scripting

Using grep and a parameter file to return unique values

Hello Everyone! I have updated the first post so that my intentions are easier to understand, and also attached sample files (post #18). I have over 500 text files in a directory. Over 1 GB of data. The data in those files is organised in lines: My intention is to return one line per... (23 Replies)
Discussion started by: clippertm
23 Replies

4. Shell Programming and Scripting

How to find sum of any 'n' number of values from file matching target value?

I have a simple text file having payment amount value on each line. At the end of day 'n' number of payments created difference in amount that I need to match from this file. I have information about how many payments created difference and difference amount. Please help me to build shell... (3 Replies)
Discussion started by: swats007
3 Replies

5. UNIX for Dummies Questions & Answers

Grep to find lines matching given patern in a file

When I try with patern matching in a file with below code works cat scj_drive_commands | egrep '/app/oracle/build_lib/pkg32|/app/oracle/build_lib/pkg33' But when I have code with patern searching using of below does not work ! cat scj_drive_commands | egrep... (3 Replies)
Discussion started by: Siva SQL
3 Replies

6. Shell Programming and Scripting

Find and count unique date values in a file based on position

Hello, I need some sort of way to extract every date contained in a file, and count how many of those dates there are. Here are the specifics: The date format I'm looking for is mm/dd/yyyy I only need to look after line 45 in the file (that's where the data begins) The columns of... (2 Replies)
Discussion started by: ronan1219
2 Replies

7. Shell Programming and Scripting

Grep with regulare expression to find carrige return

Gurus, I have a files from where lines are like following <ns0:ccid>123456789</ns0:ccid> <ns0:ccid>1234 56789</ns0:ccid> I would like to grep any number which will be as below (with carrige return): As 123456789 any number so I have to use the regular expression <ns0:ccid>1234... (3 Replies)
Discussion started by: thepurple
3 Replies

8. UNIX and Linux Applications

grep file to find unique instances of username

hello - A SystemOut.log file has recurring entries that follow this format: Principal: auth9.nick.al.gov:389/USERNAME Over the course of a day thousands of lines similar to this are produced, with each username represented hundreds of times. I need to create a new file that shows... (4 Replies)
Discussion started by: 1075FJ40
4 Replies

9. Shell Programming and Scripting

return a list of unique values of a column from csv format file

Hi all, I have a huge csv file with the following format of data, Num SNPs, 549997 Total SNPs,555352 Num Samples, 157 SNP, SampleID, Allele1, Allele2 A001,AB1,A,A A002,AB1,A,A A003,AB1,A,A ... ... ... I would like to write out a list of unique SNP (column 1). Could you... (3 Replies)
Discussion started by: phoeberunner
3 Replies

10. UNIX for Dummies Questions & Answers

Need to find only unique values for a given tag across the files

Need to find only unique values for a given tag across the files: For eg: Test1: <Tag1>aaa</Tag1> <Tag2>bbb</Tag2> <Tag3>ccc</Tag3> Test2: <Tag1>aaa</Tag1> <Tag2>ddd</Tag2> <Tag3>eee</Tag3> Test3: <Tag1>aaa</Tag1> <Tag2>ddd</Tag2> <Tag3>eee</Tag3> Test4: (8 Replies)
Discussion started by: sudheshnaiyer
8 Replies
Login or Register to Ask a Question