Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

grep and delete ONE only

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting grep and delete ONE only
# 1  
Old 01-15-2007
grep and delete ONE only
Hi,

I have a txt file will a lot of data...however, some of them a duplicated and I would like to remove it. Normally, I need to edit txt and delete it. But it take a lot of time to do....

filename.txt:
orange
apple
cake
pink
apple

if using grep -v 'apple' filename.txt > new_filename.txt, it will remove all 'apple' and I just want to delete 2nd 'apple'.

anyone can help?
# 2  
Old 01-15-2007
Use 'sort -u'. Check the man page for details.
# 3  
Old 01-15-2007
Quote:
Originally Posted by blowtorch
Use 'sort -u'. Check the man page for details.
sorry..the text file cannot sort..because of it must follow the order..so that I ONLY delete 2nd duplicated name in the file...

any other idea?
# 4  
Old 01-16-2007
Try...
Code:
awk 'a[$0]!=1;{a[$0]=1}' filename.txt

# 5  
Old 01-16-2007
Quote:
Originally Posted by Ygor
Try...
Code:
awk 'a[$0]!=1;{a[$0]=1}' filename.txt

Thanks...can you explain above for me? and can i special name to delete 2nd duplicated (such as apple, orange, etc)?
# 6  
Old 01-16-2007
Code:
awk 'a[$0]!=1;{a[$0]=1}' filename.txt

The pattern a[$0]!=1 is true when a string occurs first time because for example a["apple"] does not contain any value.

After this {a[$0]=1} is executed which assigns 1 to a["apple"]. So when text ( "apple" ) occurs next time a[$0]!=1 is not true and the string is not printed.
# 7  
Old 01-16-2007
try this

sort -u <filename>

or

sort <filename>|uniq
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Grep date pattern folder and zip -->delete

Hi All, OS: Redhat Linux 7.5 Shell: Bash I have some folder like this 2018-09-16 2018-09-17 2018-09-18 2018-09-19 and so on... Everyday one script create a folder with pattern YYYY-MM-DD (it will have so many sub directories files in it) Now what I would like to achieve is a... (1 Reply)
Discussion started by: onenessboy
1 Replies

2. Shell Programming and Scripting

Script needed to delete to the list of files in a directory based on last created & delete them

Hi My directory structure is as below. dir1, dir2, dir3 I have the list of files to be deleted in the below path as below. /staging/retain_for_2years/Cleanup/log $ ls -lrt total 0 drwxr-xr-x 2 nobody nobody 256 Mar 01 16:15 01-MAR-2015_SPDBS2 drwxr-xr-x 2 root ... (2 Replies)
Discussion started by: prasadn
2 Replies

3. Shell Programming and Scripting

Inconsistent `ps -eaf -o args | grep -i sfs_pcard_load_file.ksh | grep -v grep | wc -l`

i have this line of code that looks for the same file if it is currently running and returns the count. `ps -eaf -o args | grep -i sfs_pcard_load_file.ksh | grep -v grep | wc -l` basically it is assigned to a variable ISRUNNING=`ps -eaf -o args | grep -i sfs_pcard_load_file.ksh |... (6 Replies)
Discussion started by: wtolentino
6 Replies

4. Shell Programming and Scripting

sed command to grep multiple pattern present in single line and delete that line

here is what i want to achieve.. i have a file with below contents cat fileName blah blah blah . .DROP this REJECT that . --sport 7800 -j REJECT --reject-with icmp-port-unreachable --dport 7800 -j REJECT --reject-with icmp-port-unreachable . . . more blah blah blah --dport 3306... (14 Replies)
Discussion started by: vivek d r
14 Replies

5. Shell Programming and Scripting

Emergency grep/delete trouble!!!

Hi everyone, thank you so much for reading. I built a user reference page from /etc/password for a class project. Now I need two shell scripts: 1) Add names to the reference page 2) Delete names from the reference page. I know grep -v is involved somehow and some piping but I am super stuck. ... (1 Reply)
Discussion started by: jsmpdx
1 Replies

6. Shell Programming and Scripting

Grep a string from input file and delete next three lines including the line contains string in xml

Hi, 1_strings file contains $ cat 1_strings /home/$USER/Src /home/Valid /home/Review$ cat myxml <projected value="some string" path="/home/$USER/Src"> <input 1/> <estimate value/> <somestring/> </projected> <few more lines > <projected value="some string" path="/home/$USER/check">... (4 Replies)
Discussion started by: greet_sed
4 Replies

7. Shell Programming and Scripting

Sed or Grep to delete line containing patter plus extra line

I'm new to using sed and grep commands, but have found them extremely useful. However I am having a hard time figuring this one out: Delete every line containing the word CEN and the next line as well. ie. test.txt blue 324 CEN green red blue 324 CEN green red blue to produce:... (2 Replies)
Discussion started by: rocketman88
2 Replies

8. Shell Programming and Scripting

Grep a string in a range and delete the line

Hi, i need to delete a lines after searching a particular string but this searching should only happen after the 4th line.. basically imagine a file like this From: abcd.yahoo.com To: cdeb.yahoo.com Subject: hi all sdfsd sadasd asdasd dfsdf From: abcd.yahoo.com To:... (3 Replies)
Discussion started by: depakjan
3 Replies

9. Shell Programming and Scripting

Grep and delete lines except the lines with strings

Hi I am writing a script which should read a file and search for certain strings 'approved' or 'removed' and retain only those lines that contain the above strings. Ex: file name 'test' test: approved package waiting for approval package disapproved package removed package approved... (14 Replies)
Discussion started by: vj8436
14 Replies

10. Shell Programming and Scripting

grep and delete 2nd duplicated of txt... -part2

Hi, I find out one problem is...the main point is we must delete 2nd duplicated of word in txt file. For example apple orange pink green orange yellow orange red output should be: apple orange pink green yellow orange (16 Replies)
Discussion started by: happyv
16 Replies
Login or Register to Ask a Question