Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Remove dupe values from file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Remove dupe values from file
# 1  
Old 10-31-2011
Remove dupe values from file
Hi All,
I have a file something like
Code:
||MY|VAL1|VAL2|VAL3|VAL4||||SOME|NOT||..
||MY|VAL1|VAL2|VAL3|VAL4||||SOME|NOT||..
||MY|VAL1|VAL2|VAL3|VAL4||||SOME|NOT||..
||MY|VAL1|VAL2|VAL3|VAL4||||SOME|NOT||..
||MY|VAL1|VAL2|VAL3|VAL4||||SOME|NOT||..
||MY|VAL12|VAL22|VAL32|VAL42||||SOME|NOT||..
||MY|VAL12|VAL22|VAL32|VAL42||||SOME|NOT||..
||MY|VAL12|VAL22|VAL32|VAL42||||SOME|NOT||..
||MY|VAL12|VAL22|VAL32|VAL42||||SOME|NOT||..

I need only distinct/uniq values based on columns 4-7
So my new file having output like
Code:
VAL1|VAL2|VAL3|VAL4
VAL12|VAL22|VAL32|VAL42

Any help on this ...

Thanks!

Moderator's Comments:
Mod Comment Video tutorial on how to use code tags in The UNIX and Linux Forums.
Last edited by vbe; 11-01-2011 at 12:26 PM..
# 2  
Old 10-31-2011
Code:
uniq <path-to-yourfile>|cut -d"|" -f 4,5,6,7

# 3  
Old 10-31-2011
Quote:
Originally Posted by lokaish23
...I need only distinct/uniq values based on columns 4-7
Based on requirement:
Code:
cut -d'|' -f4-7 Inp_File | sort -u

This User Gave Thanks to Shell_Life For This Post:
lokaish23
# 4  
Old 11-01-2011
Code:
$ nawk -F\| '!x[$4$5$6$7]++' infile

# 5  
Old 11-01-2011
Sorry to bother you, but cant stop myself from asking you guys.
Could you please tell me why did you take 4,5,6,7?

Thanks
# 6  
Old 11-01-2011
As per the requirement which states "I need only distinct/uniq values based on columns 4-7" (post #1), we have chosen coulumns from 4 to 7.

HTH
--ahamed
This User Gave Thanks to ahamed101 For This Post:
bobby1015
Login or Register to Ask a Question

Previous Thread | Next Thread

8 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove duplicate values in a column(not in the file)

Hi Gurus, I have a file(weblog) as below abc|xyz|123|agentcode=sample code abcdeeess,agentcode=sample code abcdeeess,agentcode=sample code abcdeeess|agentadd=abcd stereet 23343,agentadd=abcd stereet 23343 sss|wwq|999|agentcode=sample1 code wqwdeeess,gentcode=sample1 code... (4 Replies)
Discussion started by: ratheeshjulk
4 Replies

2. Shell Programming and Scripting

Filter file to remove duplicate values in first column

Hello, I have a script that is generating a tab delimited output file. num Name PCA_A1 PCA_A2 PCA_A3 0 compound_00 -3.5054 -1.1207 -2.4372 1 compound_01 -2.2641 0.4287 -1.6120 3 compound_03 -1.3053 1.8495 ... (3 Replies)
Discussion started by: LMHmedchem
3 Replies

3. Shell Programming and Scripting

Remove the values from a certain column without deleting the Column name in a .CSV file

(14 Replies)
Discussion started by: dhruuv369
14 Replies

4. Shell Programming and Scripting

To remove decimal values from the field

Hi Friends, Hi Friends, I have a file in the following format file.txt 1|jHDJ|1345.0000000|384837843|39084938 2|jkaehjk|5784.00000|jhejhwej|3398934 i want to remove the 3rd field data decimal points from all lines output.txt 1|jHDJ|1345|384837843|39084938... (5 Replies)
Discussion started by: i150371485
5 Replies

5. Shell Programming and Scripting

remove values of a file one by one from 2nd file and then print the remaining values of 2nd file

Hi all, I have 2 files. One contains only 1 column and other one contains 2 columns, let say 1_col.txt and 2_col.txt respectively. Here, I will try to explain with an example. Input files : 1_col.txt 2_col.txt a a b x a c p ... (5 Replies)
Discussion started by: AshwaniSharma09
5 Replies

6. Shell Programming and Scripting

remove 1st and last last values of a key

For every specific keys i.e. 1st and 4th columns (a1 and ABC_001144992) remove 1st and last value (bold ones - 87942437 and 87952030 ) and print remaining input a1 87942437 87943147 1E ABC_001144992 a1 87945162 87945276 2E ABC_001144992 a1 87949524 87952030 3E ABC_001144992 a1... (3 Replies)
Discussion started by: repinementer
3 Replies

7. Shell Programming and Scripting

remove unmatched values

Below is my requirement : unmatched values should get deleted from file1 file1 A-1 B-1 C-1 D-2 E-3 F-4 file2 D C F output C-1 D-2 F-4 (2 Replies)
Discussion started by: lavnayas
2 Replies

8. Shell Programming and Scripting

Remove end of values in file

leviathan:/lcl/apps/Tivoli/netcool/omnibus/bin>more pmonfile.dat entp_stdby ot1p_stdby lawp_stdby I am wonder how to remove the _stdby from all the values in this dat file using #!/bin/ksh Thanks! (4 Replies)
Discussion started by: LRoberts
4 Replies
Login or Register to Ask a Question