Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Delete duplicate strings in a line

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Delete duplicate strings in a line
# 1  
Old 12-12-2013
Delete duplicate strings in a line
Hi,

i need help to remove duplicates in my file. The problem is i need to delete one duplicate for each line only. the input file as follows and it is not tab delimited:-

Quote:
&aff2g0440 aff2g0440 aspl2221 nos:scad1 blablablabla
&aff2g0740 aff2g0740 aspl5221 nos:scad1 blablablabla
&aff4g0160 aff4g01600 aspl2251 nos:scad1 blablablabla
&aff9g0020 aff9g0020 aspl3391 nos:scad2 blablablabla
The output need to remove 2nd word (in red) that duplicate with 1st word (in blue). Other duplicates should remained unchanged. my output should be like this:-
Quote:
&aff2g0440 aspl2221 nos:scad1 blablablabla
&aff2g0740 aspl5221 nos:scad1 blablablabla
&aff4g0160 aspl2251 nos:scad1 blablablabla
&aff9g0020 aspl3391 nos:scad2 blablablabla
i don't know how to do this. i did try but it deleted all the other duplicates as well in that lines. tried to google too and it seems that most of the issue is the duplicate lines. Please kindly help. Thanks
# 2  
Old 12-12-2013
Here is an awk solution. Note that the 3rd record in your output file does not match your input and requirement as the fields do not match. Assuming that the '&' in the beginning of field 1 is not included when matching field 2 even though you highlighted in blue.

Code:
awk '{if (substr($1,2,length($1)-1)==$2) print $1,$3,$4,$5; else print $1,$2,$3,$4,$5 }' file.txt
&aff2g0440 aspl2221 nos:scad1 blablablabla
&aff2g0740 aspl5221 nos:scad1 blablablabla
&aff4g0160 aff4g01600 aspl2251 nos:scad1 blablablabla
&aff9g0020 aspl3391 nos:scad2 blablablabla

Last edited by mjf; 12-12-2013 at 07:23 PM.. Reason: added assumption
This User Gave Thanks to mjf For This Post:
redse171
# 3  
Old 12-12-2013
Code:
awk '$1~$2{$2=x}1' file

This User Gave Thanks to Yoda For This Post:
redse171
# 4  
Old 12-12-2013
Code:
$ sed 's/&\([^ ]* \)\1/\&\1/' file
&aff2g0440 aspl2221 nos:scad1 blablablabla
&aff2g0740 aspl5221 nos:scad1 blablablabla 
&aff4g0160 aff4g01600 aspl2251 nos:scad1 blablablabla
&aff9g0020 aspl3391 nos:scad2 blablablabla

Last edited by Scrutinizer; 12-12-2013 at 08:42 PM..
This User Gave Thanks to Scrutinizer For This Post:
redse171
# 5  
Old 12-12-2013
Hi guys,

Thanks so much for your fast responses. I tried all of your codes, and Yoda codes perfectly solved my problem. mjf, your codes worked too but it deleted some of the strings that i have in my file. I have a huge files that has many weird things, and i tried changing your codes to see it how it goes. There are still strings missing though i managed to get some. and Scrutinizer, i have a problem with your codes too. But, i really appreciate your ideas on this. Thanks a lot guys!

---------- Post updated at 09:10 PM ---------- Previous update was at 09:09 PM ----------

hi Yoda,

if possible, can u explain your code here? thanks
# 6  
Old 12-12-2013
OR
Code:
$ awk 'substr($1,2) == $2{$2=x;$0=$0;$1=$1}1' file

&aff2g0440 aspl2221 nos:scad1 blablablabla
&aff2g0740 aspl5221 nos:scad1 blablablabla
&aff4g0160 aff4g01600 aspl2251 nos:scad1 blablablabla
&aff9g0020 aspl3391 nos:scad2 blablablabla

This User Gave Thanks to Akshay Hegde For This Post:
redse171
# 7  
Old 12-12-2013
Hi Akshay,

your codes work perfectly..thanks..can you pls explain it? thanks Smilie
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Find duplicate values in specific column and delete all the duplicate values

Dear folks I have a map file of around 54K lines and some of the values in the second column have the same value and I want to find them and delete all of the same values. I looked over duplicate commands but my case is not to keep one of the duplicate values. I want to remove all of the same... (4 Replies)
Discussion started by: sajmar
4 Replies

2. UNIX for Dummies Questions & Answers

Log file - Delete duplicate line & keep last date

Hello All ! I need your help on this case, I have a csv file with this: ITEM105;ARI FSR;2016-02-01 08:02;243 ITEM101;ARI FSR;2016-02-01 06:02;240 ITEM032;RNO TLE;2016-02-01 11:03;320 ITEM032;RNO TLE;2016-02-02 05:43;320 ITEM032;RNO TLE;2016-02-01 02:03;320 ITEM032;RNO... (2 Replies)
Discussion started by: vadim-bzh
2 Replies

3. Shell Programming and Scripting

Delete 2 strings from 1 line with sed?

Hi guys, I wonder if it's possible to search for a line containing 2 strings and delete that line and perhaps replace the source file with already deleted line(s). What I mean is something like this: sourcefile.txt line1: something 122344 somethin2 24334 45554676 line2: another something... (6 Replies)
Discussion started by: netrom
6 Replies

4. Shell Programming and Scripting

Delete Duplicate line (not really) from the file

I need help in figuring out hoe to delete lines in a data file. The data file is huge. I am currently using "vi" to search and delete the lines - which is cumbersome since it takes lots of time to save that file (due to its huge size). Here is the issue. I have a data file with the following... (4 Replies)
Discussion started by: GosarJunk
4 Replies

5. Shell Programming and Scripting

Delete lines in file containing duplicate strings, keeping longer strings

The question is not as simple as the title... I have a file, it looks like this <string name="string1">RZ-LED</string> <string name="string2">2.0</string> <string name="string2">Version 2.0</string> <string name="string3">BP</string> I would like to check for duplicate entries of... (11 Replies)
Discussion started by: raidzero
11 Replies

6. UNIX for Dummies Questions & Answers

Delete duplicate second line

Hi ALL I need a help I need to retain only the first line of 035 if I have two line before =040 , if only one then need to take that Eg: Input =035 (ABC)12324141241 =035 (XYZPQR)704124 =040 AB$QS$WEWR =035 (ABC)08080880809 =035 (XYZPQR)9809314 =040 ... (4 Replies)
Discussion started by: umapearl
4 Replies

7. Shell Programming and Scripting

How to delete a duplicate line and original with sed.

I am completely new to shell scripting but have been assigned the task of creating several batch files to manipulate data. My final task requires me to find lines that have duplicates present then delete not only the duplicate but the original as well. The script will be used in a windows... (9 Replies)
Discussion started by: chino_1
9 Replies

8. UNIX for Dummies Questions & Answers

Delete strings in file1 based on the list of strings in file2

Hello guys, should be a very easy questn for you: I need to delete strings in file1 based on the list of strings in file2. like file2: word1_word2_ word3_word5_ word3_word4_ word6_word7_ file1: word1_word2_otherwords..,word3_word5_others... (7 Replies)
Discussion started by: roussine
7 Replies

9. Shell Programming and Scripting

delete repeated strings (tags) in a line and concatenate corresponding words

Hello friends! Each line of my input file has this format: word<TAB>tag1<blankspace>lemma<TAB>tag2<blankspace>lemma ... <TAB>tag3<blankspace>lemma Of this file I need to eliminate all the repeated tags (of the same word) in a line, as in the example here below, but conserving both (all) the... (2 Replies)
Discussion started by: mjomba
2 Replies

10. UNIX for Dummies Questions & Answers

Delete lines with duplicate strings based on date

Hey all, a relative bash/script newbie trying solve a problem. I've got a text file with lots of lines that I've been able to clean up and format with awk/sed/cut, but now I'd like to remove the lines with duplicate usernames based on time stamp. Here's what the data looks like 2007-11-03... (3 Replies)
Discussion started by: mattv
3 Replies
Login or Register to Ask a Question