Login or Register to Ask a Question and Join Our Community


UNIX for Beginners Questions & Answers

How to delete identical lines while leaving one undeleted?

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums UNIX for Beginners Questions & Answers How to delete identical lines while leaving one undeleted?
# 1  
Old 08-18-2016
How to delete identical lines while leaving one undeleted?
Hi,

I have a file as follows.

file1

Code:
Hello
Hi
His
Hi
Hi
Hungry
hi

so I want to delete identical lines while leaving one of them undeleted.

So desired output will be

Code:
Hello
Hi
His
Hungry
hi

please note that 1 'Hi' is remaining. Any ideas?

Moderator's Comments:
Mod Comment Start using code tags, thanks.
Last edited by zaxxon; 08-18-2016 at 09:21 AM..
# 2  
Old 08-18-2016
Code:
$ awk '! a[$0]++' infile
Hello
Hi
His
Hungry
hi

# 3  
Old 08-18-2016
Hello beginner_99,

Could you please try following and let us know how it goes then.
Code:
awk '!A[$0]++'  Input_file

OR
Code:
uniq -u  Input_file

Output will be as follows in both of above commands.
Code:
Hello
Hi
His
Hungry
hi

Thanks,
R. Singh
This User Gave Thanks to RavinderSingh13 For This Post:
rbatte1
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Leaving last 30 lines

If we want to display lines from file leaving last 30 lines. i dont know the count of lines in file (14 Replies)
Discussion started by: mirwasim
14 Replies

2. Shell Programming and Scripting

sed print all lines between second and third identical lines

I am trying to extract a table of data (mysql query output) from a log file. I need to print everything below the header and not past the end of the table. I have spent many hours searching with little progress. I am matching the regexp +-\{99\} with no problem. I just can't figure out how to print... (5 Replies)
Discussion started by: godfreydanials
5 Replies

3. UNIX for Dummies Questions & Answers

more than 10 identical lines

I have a file that looks like this 10 user1s, 5 user2s and 10 users3. 10.10.1.1 user1 10.10.1.1 user1 10.10.1.1 user1 10.10.1.1 user1 10.10.1.1 user1 10.10.1.1 user1 10.10.1.1 user1 10.10.1.1 user1 10.10.1.1 user1 10.10.1.1 user1 10.10.1.2 user2 10.10.1.2 user2 10.10.1.2 user2... (7 Replies)
Discussion started by: lawsongeek
7 Replies

4. UNIX for Advanced & Expert Users

In a huge file, Delete duplicate lines leaving unique lines

Hi All, I have a very huge file (4GB) which has duplicate lines. I want to delete duplicate lines leaving unique lines. Sort, uniq, awk '!x++' are not working as its running out of buffer space. I dont know if this works : I want to read each line of the File in a For Loop, and want to... (16 Replies)
Discussion started by: krishnix
16 Replies

5. Shell Programming and Scripting

Using Bash/Sed to delete between identical strings

Hi. I'm hoping that someone can help me with a bash script to delete a block of lines from a file. What I want to do is delete every line between two stings that are the same, including the line the first string is on but not the second. (Marked lines to match with !) For example if I... (2 Replies)
Discussion started by: Zykr
2 Replies

6. UNIX for Dummies Questions & Answers

Delete all rows but leaving first and last ones

Hello, Merry Christmas to all! I wish you the best for these holidays and the best for the next year 2011. I'd like your help please, I need to delete all the rows in the third column of my file, but without touching nor changing the first and last value position, this is an example of my... (2 Replies)
Discussion started by: Gery
2 Replies

7. Shell Programming and Scripting

sed/awk: Delete matching words leaving only the first instance

I have an input text that looks like this (comes already sorted): on Caturday 22 at 10:15, some event on Caturday 22 at 10:15, some other event on Caturday 22 at 21:30, even more events on Funday 23 at 11:00, yet another event I need to delete all the matching words between the lines, from... (2 Replies)
Discussion started by: GrinningArmor
2 Replies

8. Shell Programming and Scripting

Ignore identical lines

Hello Experts, I have two files called "old" and "new". My old file contains 10 lines and my new file contains 10 + "n" lines. The first field in both these files contain ID. I sort these two files on ID. I am interested in only the lines that are in the new file and not in old. I tried... (4 Replies)
Discussion started by: forumthreads
4 Replies

9. Shell Programming and Scripting

replace 2 identical strings on different lines

I am looking to replace two or more strings on different lines using sed, but not with the same variable. IE # cat xxx.file <abc> abc def ghi abc def ghi abc def ghi currently I can only change each line with the same pattern: # sed -e '/<abc>/!s/abc\(.*\)/jkl mno/' xxx.file abc jkl mno... (3 Replies)
Discussion started by: prkfriryce
3 Replies

10. UNIX for Dummies Questions & Answers

delete string in a text file leaving the first occurrence

Hi, How can i delete the second and subsequent occurrence of a particular string from a file ? eg) test.txt cattle bat battle mat matter cattle cattle my output file should be cattle bat battle mat matter (12 Replies)
Discussion started by: gopskrish
12 Replies
Login or Register to Ask a Question