Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

awk question keep dups, delete if only one row

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting awk question keep dups, delete if only one row
# 1  
Old 05-07-2013
Linux awk question keep dups, delete if only one row
Hello,
I have the following data file which has about 500,000 rows
in $4, I want to keep all the consecutive row matches,
I want to delete the single row entries


Code:
1000
1000
1000
2000      # I want to delete because it does not have a match row
1000
1000
3000
3000
3000
1000     # I want to delete because it does not have a match row
5000
5000
5000
6000    # I want to delete because it does not have a match row
3000
3000
3000

---------------
any help is greatly appreciated.
Thanks,
Charlie
Last edited by Scrutinizer; 05-08-2013 at 12:56 AM.. Reason: code tags
# 2  
Old 05-07-2013
Assuming values are in 1st field:
Code:
awk '
        p == $1 {
                if (f)
                        A[++c] = $1
                A[++c] = $1
                f = 0
        }
        p != $1 {
                f = 1
        }
        {
                p = $1
        }
        END {
                for ( i = 1; i <= c; i++ )
                        print A[i]
        }
' file

# 3  
Old 05-07-2013
Hello Yoda,
Thanks for helping me out !, I will try this tomorrow at work,
I don't have awk here on my pc.

---------- Post updated at 09:36 PM ---------- Previous update was at 09:23 PM ----------

my datafile has text and not numbers, I hope the example works on this data
also:

Code:
apple_1
apple_1
orange_0   # I want to delete this row, because the row prev && getline != match
grape_1
grape_1
grape_1
apple_1    # I want to delete this row, because the row prev && getline != match
orange_0
orange_0

Last edited by Scrutinizer; 05-08-2013 at 12:57 AM..
# 4  
Old 05-07-2013
Yes, it does work. Here is what I get when running on above set of data:
Code:
$ ./charlieglen.awk
apple_1
apple_1
grape_1
grape_1
grape_1
orange_0
orange_0

I don't get orange_0 and apple_1 in the output.
# 5  
Old 05-07-2013
nice ! Thanks again Yoda.
I hope to understand how the arrays work.
This problem really through me for a loop, I will have about 200 las files that need to be formatted
to load into OpenWorks. This was just the first pass at solving one of the problems.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Delete duplicate row

Hi all, how can delete duplicate files in file form, e.g. $cat file1 aaa 123 234 345 456 bbb 345 345 657 568 ccc 345 768 897 456 aaa 123 234 345 456 ddd 786 784 234 263 ccc 345 768 897 456 aaa 123 234 345 456 ccc 345 768 897 456 then i need ouput file1 some, (4 Replies)
Discussion started by: aav1307
4 Replies

2. Shell Programming and Scripting

Finding missing records and Dups

I have a fixed width file. The records looks something similar to below: Type ID SSN NAME .....AND SOME MORE FIELDS A1 1234 ..... A1 1234 ..... B1 1234 ..... M2 4567 ..... M2 4567 ..... N2 4567 ..... N2 4567 ..... A1 9999 N2 9999 Now if A1 is present then B1 has to be present.... (2 Replies)
Discussion started by: Saanvi1
2 Replies

3. Shell Programming and Scripting

Delete a row if either of column value is zero

Hi, My input file is this way 1.1 0.0 2.4 3.5 7.9 1.8 22.3 4.7 8.9 0.9 1.3 0.0 3.4 5.6 0.0 1.1 2.2 0.0 0.0 1.1 0.0 0.0 3.4 5.6 I would like to delete the entire row, if either of 2nd and 3rd columns are 0.0. Please note that my values are all decimal values. So, my output would... (4 Replies)
Discussion started by: jacobs.smith
4 Replies

4. Shell Programming and Scripting

Subtracting each row from the first row in a single column file using awk

Hi Friends, I have a single column data like below. 1 2 3 4 5 I need the output like below. 0 1 2 3 4 where each row (including first row) subtracting from first row and the result should print below like the way shown in output file. Thanks Sid (11 Replies)
Discussion started by: ks_reddy
11 Replies

5. UNIX for Dummies Questions & Answers

awk question row into column

I have a csv file: test1.csv with 26 columns Sample: Data collected Comp1,,,,,,,,,,,,,,,,,,,,,,,,,Average Number of Arrivals with non Zero,0,0,0,0,0,0,0,0,0,...,0 %Utilization,0.1,0.23,0.14,...,0.36 Data collected Comp2,,,,,,,,,,,,,,,,,,,,,,,,,Average Number of Arrivals with non... (2 Replies)
Discussion started by: calitiggr
2 Replies

6. Shell Programming and Scripting

awk command : row by row merging of two files

I want to write a scrpit to merge files row wise (actually concatinating) main.txt X Y Z file 1 A B C file 2 1 2 3 now i want the script to check if the file1 is empty or not, if empty then make it like A B C 1 2 3 again to check if second file is empty if not do as done... (0 Replies)
Discussion started by: shashi792
0 Replies

7. UNIX for Dummies Questions & Answers

How to delete a row in a file

hi team, i have a file txt1 , in that file there r 10 lines . my aim is to cut 2nd and 6th line of the file . what command we can use for this scenario. (8 Replies)
Discussion started by: natraj005
8 Replies

8. Shell Programming and Scripting

delete last row in PERL

How to delete last row in the file in PERL. file1 has a.output b.output c.output d.output e.output expected output is a.output b.output c.output d.output (1 Reply)
Discussion started by: adaleru
1 Replies

9. Shell Programming and Scripting

Delete first row last column

Hi All, I am having following file and I want to delete 1 row last column. Current File Content: ================ procedure test421 put_line procedure test321 test421 procedure test521 test321 procedure test621 test521 Expected File Content: =========================== procedure... (3 Replies)
Discussion started by: susau_79
3 Replies

10. Shell Programming and Scripting

How to delete a row in a file?

Does anyone know a way to delete rows 6-9 from the below output? I have searched the forum but did not find any thing helpful. backups01laxint.liuc(s){jsandova}<0>$ nsrjb -v | awk -F' ' '$3>=01{print $0}' | cut -b 1-79 | cat -n setting verbosity level to `1' 1 slot volume ... (17 Replies)
Discussion started by: jsandova
17 Replies
Login or Register to Ask a Question