Remove duplicate line on condition


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Remove duplicate line on condition
# 15  
Old 03-30-2012
Hi
for
Code:
83 2 
86 3 
90 2
94  2 
107 2
110  2

I get
Code:
86 3 
90 2

instead of
Code:
83 2
86 3

I think it is skipping the first line, if I understand right?

---------- Post updated at 04:40 PM ---------- Previous update was at 04:28 PM ----------

Something like this perhaps,

if NR=1 print and store to lastline
else compare with last line, NR+1 and compare with next line, if any one of it is equal then don't print. else print and restore NR.

I have no clue how to write this down as code though Smilie
# 16  
Old 03-30-2012
Somewhat inelegant, but it appears to work:
Code:
[/tmp] cat x.awk
{
   lines[NR]=$0;
   matches[NR]=$2$3$4;
}
END {
   i=1;
   while (i <= NR) {
      found=0;
      check=i;
      while (matches[check]==matches[++i]) {
         found=1;
      }
      if (! found) {
         print lines[check];
      }
   }
}
[/tmp] cat x.txt
5 bla bla  1
5 bla bla  1
9 bla bla 3
78 bla bla 1
83 bla bla 2
83 bla bla 2
130 bla bla 1
83 bla bla 2
187 bla bla 0
188 bla bla 0
189 bla bla 3
190 bla bla 3
[/tmp] awk -f x.awk x.txt
9 bla bla 3
78 bla bla 1
130 bla bla 1
83 bla bla 2


Last edited by CarloM; 03-30-2012 at 11:50 AM.. Reason: formatting
# 17  
Old 03-30-2012
My mistake again. I am checking it. Sorryyyyyy...

Thanks a lot Smilie

Last edited by jamie_123; 03-30-2012 at 12:20 PM.. Reason: mistake
# 18  
Old 03-30-2012
awk

Hi,
Little bit change,
Code:
awk 'BEGIN{c=1;}{if(!a[$2$3$4]){a[$2$3$4]=$0;b[c]=$0;c++  ;}}END{for(i=1;i<c;i+  ){print b[i];}}' file

Sorry for the delay response.
Increment the c and i variable. In my mobile stucks.
Cheers,
Ranga:-)
# 19  
Old 04-02-2012
Hi Ranga,
Thanks so much for the reply, I am just seeing it, didnt receive any mails for your post Smilie. I am checking it out now.

---------- Post updated at 10:20 AM ---------- Previous update was at 09:48 AM ----------

Hi,

Since I have been giving the most simplified example input and output I have been having a lot of rough edges in many cases. Below is an actual input,
Code:
1 bla bla 0
6 bla bla 2
10 bla bla 3
15 bla bla 3
71 bla bla 3
75 bla bla 3
79 bla bla 1
83 bla bla 2
86 bla bla 3
90 bla bla 2
95 bla bla 2
107 bla bla 2
110 bla bla 2
135 bla bla 1
187 bla bla 0

I am running this for different values of column 4. So when I run it for value 0 in column 4, I expect something like
Code:
1 bla bla 0
6 bla bla 2
187 bla bla 0

. In the case of it being a last line, I have scripts to add another line, so its ok to have one value without a pair.
When running it for value 1, something like-
Code:
79 bla bla 1
83 bla bla 1
135 bla bla 1
187 bla bla 0

.

Thought the previous codes ran with specific examples, for the overall case I am having to cheat in some ways.

is it possible to make it run automatically somehow Smilie

Thanks a lot.
# 20  
Old 04-02-2012
What does 'running it for value x' mean? You don't seem to just be removing duplicates anymore...
# 21  
Old 04-02-2012
Code:
awk '{$1=$1} m==$2 FS $3 FS $4{p=x;next}p{print p};{p=$0;m=$2 FS $3 FS $4}END{if(p)print p}' infile

produces
Code:
1 bla bla 0
6 bla bla 2
79 bla bla 1
83 bla bla 2
86 bla bla 3
135 bla bla 1
187 bla bla 0

But it seems your requirements have changed?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove sections based on duplicate first line

Hi, I have a file with many sections in it. Each section is separated by a blank line. The first line of each section would determine if the section is duplicate or not. if the section is duplicate then remove the entire section from the file. below is the example of input and output.... (5 Replies)
Discussion started by: ahmedwaseem2000
5 Replies

2. Shell Programming and Scripting

Remove duplicate values with condition

Hi Gents, Please can you help me to get the desired output . In the first column I have some duplicate records, The condition is that all need to reject the duplicate record keeping the last occurrence. But the condition is. If the last occurrence is equal to value 14 or 98 in column 3 and... (2 Replies)
Discussion started by: jiam912
2 Replies

3. Shell Programming and Scripting

Remove duplicate line starting with a pattern

HI, I have the below input file /* ----------------- cmdsDlyStartFWJ -----------------*/ UNIX_JOB CMDS065J RUN ANY CMDNAME sleep 5 AGENT CMDSHP USER proddata RUN MON,TUE,WED,THU,FRI DELAYSUB 02:00 /* "Triggers daily file watcher jobs" */ ENVAR... (5 Replies)
Discussion started by: varun22486
5 Replies

4. Shell Programming and Scripting

Remove duplicate entries from the same line

Hello, I have a file which have several duplicate entries on the same line: File ID source 1 GM GF GM 2 GM GF GM GF GM GF GM GF GM GF 3 GM GF GM SF GM GF GM SF 4 FF FF FF FF 5 FF GM FF ... (2 Replies)
Discussion started by: nans
2 Replies

5. UNIX for Dummies Questions & Answers

Remove Duplicate Two Line Pairs?

So I have a bunch of files that look like this >gi|33332323 MMKCRGVIMVVEKVMKRDGRIVPFDESRIRWAVQ--- >gi|45235353 MMKCR----VEKMRDVFFDESIRWAVQ They go on...sequences are much longer but all in two line (fasta) format. I want to remove duplicate pairs of ID(GI) number and sequence. I tried... (12 Replies)
Discussion started by: bakere19
12 Replies

6. Shell Programming and Scripting

remove duplicate lines with condition

hi to all Does anyone know if there's a way to remove duplicate lines which we consider the same only if they have the first and the second column the same? For example I have : us2333 bbb 5 us2333 bbb 3 us2333 bbb 2 and I want to get us2333 bbb 10 The thing is I cannot... (2 Replies)
Discussion started by: vlm
2 Replies

7. Shell Programming and Scripting

remove of duplicate line from a file

I have a file a.txt having content like deepak ram sham deepram sita kumar I Want to delete the first line containing "deep" ... I tried using... grep -i 'deep' a.txt It gives me 2 rows...I want to delete the first one.. + need to know the command to delete the line from... (5 Replies)
Discussion started by: saluja.deepak
5 Replies

8. Shell Programming and Scripting

remove duplicate words in a line

Hi, Please help! I have a file having duplicate words in some line and I want to remove the duplicate words. The order of the words in the output file doesn't matter. INPUT_FILE pink_kite red_pen ball pink_kite ball yellow_flower white no white no cloud nine_pen pink cloud pink nine_pen... (6 Replies)
Discussion started by: sam_2921
6 Replies

9. UNIX for Dummies Questions & Answers

Remove duplicate entry in one line

Can anyone help me how can i print only the unique entry in a line? MI_AP MI_AP MI_CM MI_MF RC_NAP MBS_AP SF_RAN MBS_AP NT_CAR so that it will on output the one unique entry per line. MI_AP MI_CM MI_MF RC_NAP MBS_AP SF_RAN NT_CAR I can't find the same situation on the knowledge... (5 Replies)
Discussion started by: kharen11
5 Replies

10. UNIX for Dummies Questions & Answers

Remove Duplicate line

Hi, I have a scenario here where I have created a flatfile with the below mentioned information. File as you can see is dispalyed in three columns 1st column is FileNameString 2nd column is Report_Name (this has spaces) 3rd column is Flag Result file needed is, removal of duplicate... (1 Reply)
Discussion started by: Student37
1 Replies
Login or Register to Ask a Question