Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Remove multiple lines in the file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Remove multiple lines in the file
# 1  
Old 06-26-2010
Remove multiple lines in the file
Hi,

How to remove duplicate lines in the file.

Code:
a 
------
b
c
24
23
a
----
c
b

# 2  
Old 06-26-2010
Code:
nawk '!a[$0]++' myFile

# 3  
Old 06-26-2010
Maybe
Code:
sort -u file

# 4  
Old 06-28-2010
awk '!a[$0]++' myFile

This will work for me. But how to redirect the output to same file "myFile"
# 5  
Old 06-28-2010
Code:
awk '!a[$0]++' myFile > myFile.tmp
mv myFile.tmp myFile

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Remove newline character from column spread over multiple lines in a file

Hi, I came across one issue recently where output from one of the columns of the table from where i am creating input file has newline characters hence, record in the file is spread over multiple lines. Fields in the file are separated by pipe (|) delimiter. As header will never have newline... (4 Replies)
Discussion started by: Prathmesh
4 Replies

2. Shell Programming and Scripting

How to remove html tag which has multiple lines in SHELL?

I want to clean a html file. I try to remove the script part in the html and remove the rest of tags and empty lines. The code I try to use is the following: sed '/<script/,/<\/script>/d' webpage.html | sed -e 's/<*>//g' | sed '/^\s*$/d' > output.txt However, in this method, I can not... (10 Replies)
Discussion started by: YuhuiFeng
10 Replies

3. Shell Programming and Scripting

Removing multiple lines from input file, if multiple lines match a pattern.

GM, I have an issue at work, which requires a simple solution. But, after multiple attempts, I have not been able to hit on the code needed. I am assuming that sed, awk or even perl could do what I need. I have an application that adds extra blank page feeds, for multiple reports, when... (7 Replies)
Discussion started by: jxfish2
7 Replies

4. Shell Programming and Scripting

Remove multiple lines that match pattern

Not sure how I can accomplish this. I would like to remove all interfaces that have the commands I would like to see: switchport port-security, spanning-tree portfast. One line is no problem. interface FastEthernet0/8 spanning-tree portfast interface FastEthernet0/9 spanning-tree... (4 Replies)
Discussion started by: mrlayance
4 Replies

5. Shell Programming and Scripting

Remove multiple lines from a text file

Hi I have a text file named main.txt with 10,000 lines. I have another file with a list of line numbers (around 1000) of the lines to be deleted from main.txt file. I tried with sed but it removes only a range of line numbers. Thanks for any help!! (1 Reply)
Discussion started by: prvnrk
1 Replies

6. UNIX for Dummies Questions & Answers

Want to remove all lines but not latest 50 lines from a file

Hi, I have a huge file which has Lacs of lines. File system got full. I want your guys help to suggest me a solution so that I can remove all lines from that file but not last 50,000 lines. I want solution which can remove lines from existing file so that I can have some space left with. (28 Replies)
Discussion started by: prashant2507198
28 Replies

7. Shell Programming and Scripting

How to search multiple patterns and remove lines from a file?

Hi, I have a file content as below. Table : PAYR Displayed fields: 15 of 15 Fixed columns: 4 List width 0999... (4 Replies)
Discussion started by: shirdi
4 Replies

8. UNIX for Advanced & Expert Users

To Remove multiple new lines

Hi , I have file which consists of multiple new lines. I need to identify the new line in every row of the file and remove it. It is having record delimiter as "|" and final delimiter of the line is "end" Can anyone from the forum suggest me how to remove the news lines from entire file? example... (11 Replies)
Discussion started by: etldeveloper
11 Replies

9. Shell Programming and Scripting

sed remove multiple set of lines in one command

is there a way with sed to removed more than one set of lines in one line? so i mean sed ${firstElem},${lastIndex}d web.xml > web1.xml this will delete lines between ${firstElem},${lastIndex} i want in the same line to do somethinkg like this (doesn't work so far) sed... (3 Replies)
Discussion started by: Poki
3 Replies

10. Shell Programming and Scripting

How to remove certain lines in multiple txt files?

Hi , I have this type of files:- BGH.28OCT2008.00000001.433155.001 BGH.28OCT2008.00000002.1552361.001 BGH.28OCT2008.00000003.1438355.001 BGH.28OCT2008.00000004.1562602.001 Inside them contains the below: 5Discounts 6P150 - Max Total Usage RM150|-221.00 P150 EPP - Talktime RM150... (5 Replies)
Discussion started by: olloong
5 Replies
Login or Register to Ask a Question