Search and Remove No data Section


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Search and Remove No data Section
# 1  
Old 02-08-2010
Search and Remove No data Section

Hello,

I have written a script that removes duplicates within a file and places them in another report.

File:
Code:
 
ABC1 012345 header
ABC2 7890-000
ABC3 012345 Content Header
ABC5 593.0000 587.4800
ABC5 593.5000 587.6580
ABC5 593.5000 587.6580
ABC1 67890 header
ABC2 1234-0001
ABC3 67890 Content Header
ABC5 594.5000 588.5361
ABC5 602.0000 595.0642
ABC5 602.0000 595.0642
ABC1 234567 header            #   This section removed from 
ABC2 2345-0009                #   the script because there
ABC3 23456 Content Header     #   are no duplicates found 
ABC5 590.5000 588.5361        #   In this section
ABC5 600.5000 594.1603        #
ABC1 019283 header
ABC2 5748-0101
ABC3 019283 Content Header
ABC5 590.5000 588.5361
ABC5 590.5000 588.5361
ABC5 600.5000 594.1603

Here is the output from my script (which is how I need it outputted):

Code:
 
ABC1 012345 header
ABC3 012345 Content Header
ABC5 593.5000 587.6580
 
ABC1 67890 header
ABC3 67890 Content Header
ABC5 602.0000 595.0642
 
ABC1 019283 header
ABC3 019283 Content Header
ABC5 590.5000 588.5361
ABC5 600.5000 594.1603

Here is my code:
Code:
 
# This will remove the duplicate lines and place the unique lines in file.out
awk '{ if ($1=="ABC1") {delete arr}
      if ( !arr[$0]++ ) { print $0 } }' file.dat > file.out
 
# This will create the duplicate report keeping ABC1 as the section
# header and keeping the Content header as well
awk '/ABC[13]/&&h=$2;_[h,$2,$3]++==1' file.dat > file.tmp
 
#   Seperates the sections by adding an empty line
awk '/ABC1/{print ""}1' file.tmp > file.tmp2
 
# Removes any ABC1 section without any duplicates.  This is done by 
# searching for ABC3 and a blank line after it.  If found it will remove
# only that line, then it will go back and search for the ABC1 header with
# a blank line after it and remove that line as well as the blank line.
sed '/ABC3/{N;/\n *$/D;}' file.tmp2 > file.tmp3
sed '/ABC1/{N;/\n *$/d;}' file.tmp3 > file.dup

My questions - Is there a way to combine these two sed commands and perform the same task?
# 2  
Old 02-08-2010
Code:
sed -e '/ABC3/{N;/\n *$/D;}' -e '/ABC1/{N;/\n *$/d;}' file.tmp2 > file.tmp3

# 3  
Old 02-08-2010
Thanks,
but that didn't seem to work.

Here's the code:
Code:
sed -e '/ABC3/{N;/\n *$/D;}' -e '/ABC1/{N;/\n *$/d;}' file.tmp2 > file.dup

My output was:
Code:
ABC1 012345 header
ABC3 012345 Content Header
ABC5 593.5000 587.6580

ABC1 67890 header
ABC3 67890 Content Header
ABC5 602.0000 595.0642

ABC1 234567 header            -> These two lines need to be
ABC3 23456 Content Header     -> removed

ABC1 019283 header
ABC3 019283 Content Header
ABC5 590.5000 588.5361
ABC5 600.5000 594.1603

Any advise? Thanks for your help!
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Grep a section from an UNIX file obtaining only part of the data

Hello, I have a log file that has several sections "BEGIN JOB, End of job" like in the following example: 19/06/12 - 16:00:57 (27787398-449294): BEGIN JOB j1(27787398-449294) JOB1 19/06/12 - 16:00:57 (27787398-449294): DIGIT: 0 number of present logs : 1 19/06/12 - 16:00:57... (4 Replies)
Discussion started by: mvalonso
4 Replies

2. Shell Programming and Scripting

Search data file2 for entries in data file1

Hello, I have two data files: file1: 9780205646999 28.31 20 Cengage 9780205647040 51.94 20 Cengage 9780205660568 49.11 20 Cengage 9780205696758 51.75 20 Cengage 9780205727643 41.63 20 Cengage file2: 9780020080954 9780020080954 2 ... (7 Replies)
Discussion started by: palex
7 Replies

3. Shell Programming and Scripting

Copying section of file based on search criteria

Hi Guru's, I am new to unix scripting. I have a huge file with user details in it(file2) and I have another file with a list of users(file1). Script has to search a user from file1 and get all the associated lines from file2. Example: fiel1: cn=abc cn=DEF cn=xyx File 2: dn:... (10 Replies)
Discussion started by: Samingla
10 Replies

4. Shell Programming and Scripting

Getting last section of data from logfile

Hi, I have a log file from Munin like this:2012/12/04 13:45:31 : Munin-update finished (29.68 sec) 2012/12/04 13:50:01 Opened log file 2012/12/04 13:50:01 : Starting munin-update 2012/12/04 13:50:01 Error occured in under in the configuration. 2012/12/04 13:50:01 Could not parse datafile... (9 Replies)
Discussion started by: Jotne
9 Replies

5. Shell Programming and Scripting

take a section of a data with conditions

I have a data file like below: 2011 0701 2015 21.2 L 37.692 46.202 18.0 Teh 4 0.3 2.1 LTeh 1 GAP=233 E Iranian Seismological Center, Institute of Geophysics, University of Tehran 6 STAT SP IPHASW D HRMM SECON CODA AMPLIT PERI AZIMU VELO SNR AR TRES W DIS CAZ7 TBZ SN EPg 0 2015 31.19 -0.3... (3 Replies)
Discussion started by: saeed.soltani
3 Replies

6. Shell Programming and Scripting

Deleting a section based on search from other file

Hi Everyone, I need some help to accomplish the below. help is highly appriciated. I have a 45 mb file with ldap entries. Each user entry is separated by a string # entry-id: 1 and so on. Some of the entries has a string xyz: true. I want to delete the section if the user section has xyz: true... (6 Replies)
Discussion started by: Samingla
6 Replies

7. Shell Programming and Scripting

remove <br> not in section

I need to remove the <BR> from all sections of a page, except what is between a section of text: #!/bin/sh sed ' /Testing Considerations/,/<B>PT# - Description:/ ! { s/<BR>// } ' But this isn't working. I'm not using the ! operator correctly, can someone... (2 Replies)
Discussion started by: dba_frog
2 Replies

8. Shell Programming and Scripting

Extract section of file based on word in section

I have a list of Servers in no particular order as follows: virtualMachines="IIBSBS IIBVICDMS01 IIBVICMA01"And I am generating some output from a pre-existing script that gives me the following (this is a sample output selection). 9/17/2010 8:00:05 PM: Normal backup using VDRBACKUPS... (2 Replies)
Discussion started by: jelloir
2 Replies

9. Shell Programming and Scripting

Search and extract by section from configuration

Hi, I understand either AWK or SED can do this, but I not sure how to extract the following configuration in section. Meaning when I need to find code with " ip helper-address 192.168.11.2" , it would start from "interface Serial0/0" and "interface FastEthernet0/1". Only displaying both section... (2 Replies)
Discussion started by: haphazard
2 Replies

10. Shell Programming and Scripting

Remove certain section from the line

A typical line looks like this... ) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_bin AUTO_INCREMENT=129 COMMENT='Compiled E-Mails';I want to remove DEFAULT CHARSET= and COLLATE= after resetting AUTO_INCREMENT=0 I do not want to change the engine and comment. (7 Replies)
Discussion started by: shantanuo
7 Replies
Login or Register to Ask a Question