Sponsored Content
Top Forums UNIX for Dummies Questions & Answers Using awk to remove duplicate line if field is empty Post 302931842 by tugar on Thursday 15th of January 2015 10:31:38 PM
Old 01-15-2015
Using awk to remove duplicate line if field is empty

Hi all,

I've got a file that has 12 fields. I've merged 2 files and there will be some duplicates in the following:

FILE:

Code:
1. ABC, 12345, TEST1, BILLING, GV, 20/10/2012, C, 8, 100, AA, TT, 100
2. ABC, 12345, TEST1, BILLING, GV, 20/10/2012, C, 8, 100, AA, TT, (EMPTY)
3. CDC, 54321, TEST3, BILLING, VG, 10/11/2012, D, 9, 99, BB, AA, 99
4. CDC, 54321, TEST3, BILLING, VG, 10/11/2012, D, 9, 99, BB, AA, (EMPTY)
5. AAA, 34535, TEST1, BILLING, CC, 12/09/2011, C, 9, 99, BB, CC, (EMPTY)

OUTPUT:

Code:
1. ABC, 12345, TEST1, BILLING, GV, 20/10/2012, C, 8, 100, AA, TT, 100
3. CDC, 54321, TEST3, BILLING, VG, 10/11/2012, D, 9, 99, BB, AA, 99
5. AAA, 34535, TEST1, BILLING, CC, 12/09/2011, C, 9, 99, BB, CC, (EMPTY)

Need an awk script to remove lines 2 and 4 because there is no data in the 12th field. Line 5 would still remain as there is no other duplicate line.

I have no problem removing these in excel but i need the script to remove them so that the user doesn't need to remove them in excel.

Thanks!

Last edited by Scrutinizer; 01-17-2015 at 04:00 AM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Awk scrip to remove empty field

Hi I have a file which looks like this name: Sally group: Group4 name: Tim group: Group1 name: Dan group: Group2 name: Chris group: Group3 name: Peter group: name: Fred group: name: Mary group: Group2 Well I want to get rid of the... (4 Replies)
Discussion started by: bombcan
4 Replies

2. Shell Programming and Scripting

awk script to remove duplicate rows in line

i have the long file more than one ns and www and mx in the line like . i need the first ns record and first www and first mx from line . the records are seperated with tthe ; i am try ing in awk scripting not getiing the solution. ... (4 Replies)
Discussion started by: kiranmosarla
4 Replies

3. Shell Programming and Scripting

Using sed to remove lines where field is empty

I was just looking at this post: https://www.unix.com/shell-programming-scripting/22893-delete-multiple-empty-lines.html. and I am looking to achieve the same with sed. So the idea is to delete lines from a file where a certain field has no value. Inputfile: EMID MMDDYY HOURS JOB EMNAME 0241... (4 Replies)
Discussion started by: figaro
4 Replies

4. Shell Programming and Scripting

awk - if field is empty, move line to new file

I have a script with this statement: /usr/xpg4/bin/awk -F"" 'NR==FNR{s=$2;next}{printf "%s\"%s\"\n", $0, s}' LOOKUP.TXT finallistnew.txt >test.txt I want to include logic or an additional step that says if there is no data in field 3, move the whole line out of test.txt into an additional... (9 Replies)
Discussion started by: scriptr2be
9 Replies

5. Shell Programming and Scripting

Remove duplicate lines (the first matching line by field criteria)

Hello to all, I have this file 2002 1 23 0 0 2435.60 131.70 5.60 20.99 0.89 0.00 285.80 2303.90 2002 1 23 15 0 2436.60 132.90 6.45 21.19 1.03 0.00 285.80 2303.70 2002 1 23 ... (6 Replies)
Discussion started by: joggdial3000
6 Replies

6. Shell Programming and Scripting

awk - remove row if specific field is empty/blank

I have this text.filecharles darwin sam delight george washington johnson culper darwin sam delight micheal jackson penny lite and would like to remove the row, if the first field is blank. so the result would be: result.filecharles darwin sam ... (4 Replies)
Discussion started by: charles33
4 Replies

7. UNIX for Dummies Questions & Answers

remove empty field

Hi all ! I'm sure it is a basic question but I didn't find any threads that fit my need. How to remove empty fields with awk? Or in other words, how to shift all the fields after an empty field on the left? input: 1|2||3|4|5||6 wanted: 1|2|3|4|5|6 I tried: awk '{for(i=1; i<=NF;... (7 Replies)
Discussion started by: lucasvs
7 Replies

8. Shell Programming and Scripting

Remove duplicate value based on two field $4 and $5

Hi All, i have input file like below... CA009156;20091003;M;AWBKCA72;123;;CANADIAN WESTERN BANK;EDMONTON;;2300, 10303, JASPER AVENUE;;T5J 3X6;; CA009156;20091003;M;AWBKCA72;321;;CANADIAN WESTERN BANK;EDMONTON;;2300, 10303, JASPER AVENUE;;T5J 3X6;; CA009156;20091003;M;AWBKCA72;231;;CANADIAN... (2 Replies)
Discussion started by: mohan sharma
2 Replies

9. Shell Programming and Scripting

How to remove empty field in a text file?

Hi all, I want to remove empty field in a text file. I tried to used sed. But it failed. Input: LG10_PM_map_19_LEnd 1000560 G AG AG LG10_PM_map_19_LEnd 1005621 G AG LG10_PM_map_19_LEnd 1011214 A AG AG LG10_PM_map_19_LEnd 1011673 T CT CT ... (3 Replies)
Discussion started by: huiyee1
3 Replies

10. Shell Programming and Scripting

awk to remove line if field has symbols in it

Trying to use awk to remove a line only if $1 contains either ; or :. Thje awk below runs but no lines are removed. Thank you :). awk awk '$1 !~ /;/ || $1 !~ /:/ { print }' file file AARS2;TMEM151B 1 AASS 2 ABAT 3 ABCA1 3 ABCA10 1 ABCA12 2 ABCA13 1 ABCA13:AX746840 2 ABCA2 5 (5 Replies)
Discussion started by: cmccabe
5 Replies
cdc(1)							      General Commands Manual							    cdc(1)

Name
       cdc - change delta commentary of an SCCS file

Syntax
       cdc -rSID [-m[mrlist]]  [-y[comment]] files

Description
       The command changes the delta commentary of each named SCCS file, for the SID specified by the -r option.

       The  delta  commentary  is  defined to be the Modification Request (MR) and comment information usually specified by the command (-m and -y
       options).

       The commentary may consist of one or more lines, terminated by a dot in column one of a new line.

       If a directory is named, behaves as though each file in the directory were specified as a named file, except non-SCCS files (last component
       of the path name does not begin with s.)  and unreadable files, which are silently ignored.  If a name of - is given, the standard input is
       read (see RESTRICTIONS).  Each line of the standard input is taken to be the name of an SCCS file to be processed.

       Arguments to which may appear in any order, consist of option arguments, and file names.

       All the described option arguments apply independently to each named file.

Options
       -m[mrlist]     Adds or deletes modification numbers.  If the SCCS file has the v flag set then a list of MR  numbers  to  be  added  and/or
		      deleted  in the delta commentary of the SID specified by the -r option may be supplied.  For further information, see A null
		      MR list has no effect.

		      MR entries are added to the list of MRs in the same manner as that of In order to delete an MR, precede the MR  number  with
		      the  character  ! (see Examples).  If the MR to be deleted is currently in the list of MRs, it is removed and changed into a
		      comment line.  A list of all deleted MRs is placed in the comment section of the delta commentary and is preceded by a  com-
		      ment line stating that they were deleted.

		      If  -m  is not used and the standard input is a terminal, the prompt MRs?  is issued on the standard output before the stan-
		      dard input is read.  If the standard input is not a terminal, no prompt is issued.  The MRs? prompt always precedes the com-
		      ments? prompt (see -y option).
		      MRs in a list are separated by blanks and/or tab characters.  An unescaped new-line character terminates the MR list.

		      Note  that  if  the v flag has a value it is taken to be the name of a program (or shell procedure) which validates the cor-
		      rectness of the MR numbers.  For further information, see If a nonzero exit status is returned from the MR number validation
		      program, terminates and the delta commentary remains unchanged.

       -rSID	      Specifies the SCCS Identification string of a delta for which the delta commentary is to be changed.

       -y[comment]    Replaces	existing  commentary  for  the delta specified by the -r option.  The previous comments are kept and preceded by a
		      comment line stating that they were changed.  A null comment has no effect.

		      If -y is not specified and the standard input is a terminal, the prompt comments? is issued on the  standard  output  before
		      the standard input is read; if the standard input is not a terminal, no prompt is issued.  A dot in column one of a new line
		      terminates the comment text.

       Certain permissions are necessary to modify the SCCS file; generally, however, if you made the delta, you can change its delta  commentary,
       and if you own the file and directory you can modify the delta commentary.

Examples
       This example shows how to add bl78-12345 and bl79-00001 to the MR list, remove bl77-54321 from the MR list, and add the comment ``trouble''
       to delta 1.6 of s.file.
       sccscdc -r1.6 -m"bl78-12345 !bl77-54321 bl79-00001" -ytrouble .file

       This example does the same thing.
       sccscdc -r1.6 .file

       -MRs? !bl77-54321 bl78-12345 bl79-00001

       comments? trouble

Restrictions
       If SCCS file names are supplied to the command via the standard input (- on the command line), then the -m and  -y  options  must  also	be
       used.

Diagnostics
       See for explanations.

Files
       For more information, see

       For more information, see

See Also
       admin(1), delta(1), get(1), help(1), prs(1), sccs(1), sccsfile(5)
       Guide to the Source Code Control System

																	    cdc(1)
All times are GMT -4. The time now is 09:43 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy