Login or Register to Ask a Question and Join Our Community


UNIX for Dummies Questions & Answers

Deleting all rows that contain redundant information

 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Deleting all rows that contain redundant information
# 1  
Old 06-16-2011
Deleting all rows that contain redundant information
My input file looks like this:
Code:
1	rs4040617	0.08356
1	rs4040617	0.06799
1	rs2977612	0.07948
1	rs2977612	0.07882
1	rs2977612	0.07783
1	rs2977612	0.08142
1	rs2977612	0.07716
1	rs2977612	0.08356
1	rs2977612	0.06799
1	rs2980300	0.08356
1	rs2980300	0.08142

I want to delete all rows that have the same string for column 2, except the first row that contains that specific string such that my output would look like:
Code:
1	rs4040617	0.08356
1	rs2977612	0.07948
1	rs2980300	0.08356

How do I go about doing that? Thanks!
Last edited by Scott; 06-16-2011 at 08:54 PM.. Reason: Code tags, please...
# 2  
Old 06-16-2011
Code:
$ awk '!A[$2]++' file
1	rs4040617	0.08356
1	rs2977612	0.07948
1	rs2980300	0.08356

This looks much like your previous thread!

And, please start using code tags.
This User Gave Thanks to Scott For This Post:
evelibertine
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

deleting rows under a certain condition

there are 20 variables and I would like to delete the rows if 13th-20th columns are all NA. Thank you! FID IID aspirpre statihos fibrahos ocholhos arbhos betabhos alphbhos cacbhos diurehos numbcig.x toast1 toast2 toast3 toast4 ischoth1 ischoth2 ischoth3 ischoth4 101 101 1 1 1 1 1 2 1 2... (2 Replies)
Discussion started by: johnkim0806
2 Replies

2. UNIX for Dummies Questions & Answers

Deleting all rows with empty columns

I have a text file that looks like this: 1 rs523634 8.22486 1 1 rs585160 8.22488 1 rs497228 8.2249 1 1 rs600933 8.225 1 rs480106 8.22531 1 rs600199 8.22533 1 rs529015 8.22534 1 rs598894 8.22534 I want to delete the rows with empty... (2 Replies)
Discussion started by: evelibertine
2 Replies

3. Shell Programming and Scripting

deleting rows that have certain characters

Hi, I want to delete rows whenever column one has the letters 'rpa'. The file is tab seperated. e.g. years 1 bears 1 cats 2 rpat 3 rpa99 4 rpa011 5 then removing 'rpa' containing rows based on the first column years 1 bears 1 cats 2 thanks (7 Replies)
Discussion started by: phil_heath
7 Replies

4. Shell Programming and Scripting

Deleting of Specific Rows.

Fruit : Price : Quantity apple : 20 : 40 chiku : 40 :30 Hey guys, i have written a code using sed to delete a specific char which is being typed in. But the problem i am having is , how can i expand my coding to actually allow it do delete the whole row. For example,... (21 Replies)
Discussion started by: gregarion
21 Replies

5. Shell Programming and Scripting

Deleting specific rows in large files having rows greater than 100000

Hi Guys, I need help in modifying a large text file containing more than 1-2 lakh rows of data using unix commands. I am quite new to the unix language the text file contains data in a pipe delimited format sdfsdfs sdfsdfsd START_ROW sdfsd|sdfsdfsd|sdfsdfasdf|sdfsadf|sdfasdf... (9 Replies)
Discussion started by: manish2009
9 Replies

6. Shell Programming and Scripting

deleting rows that dont have a certain # of columns

Hi, I want to delete rows that dont have a certain # of columns. In my case, rows that are less than 8 should be removed (those greater than 8 are ok). For instance: 1 2 3 4 5 6 7 8 2 3 2 4 3 2 1 5 1 2 3 4 5 6 8 2 2 4 3 1 1 1 1 1 1 1 1 1 after: 1... (8 Replies)
Discussion started by: gisele_l
8 Replies

7. Shell Programming and Scripting

deleting rows that dont have ....

Hi I posted earlier. This is sorta similar but I want to delete rows that dont have R, T, Y or U. Nam1 RTYU Nam2 RRTT Nam3 RYTU Nam4 IRTT So the output would look like this? Nam1 RTYU Nam2 RRTT Nam3 RYTU too many problems thanks (3 Replies)
Discussion started by: kylle345
3 Replies

8. Shell Programming and Scripting

Deleting rows from csv file

Hello, I am supposed to process about 100 csv files. But these files have some extra lines at the bottom of the file. these extra lines start with a header for each column and then some values below. These lines are actually a summary of the actual data and not supposed to be processed. These... (8 Replies)
Discussion started by: cobroraj
8 Replies

9. Shell Programming and Scripting

Deleting rows that begin with #

Hi, I have a file that has rows that start with # and ends with #. For example.. # hi text JK NM JK NM JK K JK NM # no # yes So I want to remove the #'s and put them into another file. so the output will be two files.. File 1: JK NM JK NM JK K JK NM (3 Replies)
Discussion started by: phil_heath
3 Replies

10. Shell Programming and Scripting

Deleting the emty rows in a file

I am getting some spaces between the two lines(rows) in file.i want delete that empty rows in the file example 1 abc xyz 2 def jkl like i am having lots of rows in a file i want to delete the spce between the two rows give any... (7 Replies)
Discussion started by: srivsn
7 Replies
Login or Register to Ask a Question