Deleting Repeating lines from a txt file via script Post: 302903985

Sponsored Content

Top Forums Shell Programming and Scripting Deleting Repeating lines from a txt file via script Post 302903985 by Scrutinizer on Saturday 31st of May 2014 01:55:53 PM

05-31-2014

Moderator

Quote:

Originally Posted by Don Cragun

Scrutinizer's code makes assumptions about what the 1st line of a header looks like (based on your sample input) that might not be true in your real data and only requires that the 3rd line of a header contain From= somewhere on the line. It will also delete a 3 line header if the next 3 line header is the same as the previous 3 line header even if those headers are not adjacent.
[..]

Good catch about the adjacency, Don. Quick fix:

Code:

awk -F, '                                                                       
  NF==4 && $1~/^[0-9]+$/ {                          # If there are 4 fields and the first field is numerical, then found a header                               
    is_header=1
    previous=current
    current=""                                                                 
  }                                                                             
  is_header {                                                                   
    current=current $0 ORS                          # Add lines to current header                                                                               
    if(/From=/){                                    # Last line of header       
      if(current!=previous) printf "%s", current                                
      is_header=0                                                               
    }                                                                           
    next                                                                        
  }
  {
    current=""
  }                                                               
  1 
' file

But further robustness would need to be added and "From=" would need to be always present...

Scrutinizer

View Public Profile for Scrutinizer

Find all posts by Scrutinizer

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

print all even lines of a txt file

In other news, I have a colors text file with hundreds of lines, and I want to print only the even numbered lines. for example I have this file looks something like this: ALLCOLORS.TXT red red green red blue red red red green red red blue green green green blue blue blue red blue blue blue...

2. Shell Programming and Scripting

Deleting lines that contain spaces in a txt file

I need some help deleting lines in a file that contain spaces. Im sure awk or sed will work but i dont know much about those commands. Any help is appreciated :D

3. UNIX for Dummies Questions & Answers

Deleting lines in .txt with nonspecific value

Hello, i am new to the forum and know nothing about programing, Linux or Unix :( hope somebody can help me out. I have a .txt file that i need to delete certain lines from. After searching the forum i noticed that using "sed" was the way to go, so i installed gnuwin32 (i use windows xp...

4. Shell Programming and Scripting

script to delete lines from a txt file if pattern matches

File 6 dbnawldb010-b office Memphis_Corp_SQL_Diff Memphis-Corp-SQL-Inc-Application-Backup 03/09/11 03:24:04 42 luigi-b IPNRemitDB Memphis_Corp_SQL_Diff Memphis-Corp-SQL-Inc-Application-Backup 03/10/11 00:41:36 6 ebs-sqldev1-b IPNTracking Memphis_Corp_SQL_Diff...

5. Shell Programming and Scripting

sed to cp lines x->y from 1.txt into lines a->b in file2.txt

I have one base file, and multiple target files-- each have uniform line structure so no need to use grep to find things-- can just define sections by line number. My question is quite simple-- can I use sed to copy a defined block of lines (say lines 5-10) from filename1.txt to overwrite an...

6. Shell Programming and Scripting

merging two .txt files by alternating x lines from file 1 and y lines from file2

Hi everyone, I have two files (A and B) and want to combine them to one by always taking 10 rows from file A and subsequently 6 lines from file B. This process shall be repeated 40 times (file A = 400 lines; file B = 240 lines). Does anybody have an idea how to do that using perl, awk or sed?...

7. UNIX for Dummies Questions & Answers

find lines in file1.txt not found in file2.txt memory problem

I have a diff command that does what I want but when comparing large text/log files, it uses up all the memory I have (sometimes over 8gig of memory) diff file1.txt file2.txt | grep '^<'| awk '{$1="";print $0}' | sed 's/^ *//' Is there a better more efficient way to find the lines in one file...

8. UNIX for Dummies Questions & Answers

redirecting arguments in a script to multiple lines in a .txt file

Ok hope my vocab is right here, i'm trying to write multiple sets of arguments to another file for example: I have a script that accepts four arguments and sends them to a another file $write.sh it then out in so the file receiver.txt would contain this: it then out in what...

9. Shell Programming and Scripting

Getting lines from .txt file

Hi I have a file with contents: NAMES John carrey williams How can I get all the names and store them in seperate variables(or arrays) please keep in mind that the no. of such names is not known.Three here is a bogus value ~thanks

10. Shell Programming and Scripting

Creating repeating record in between file through script

apprecieate your help to resove this. My source file looke like 1001 000 HEADER 1001 001 RAJESH 1001 002 100 1001 002 200 1001 002 500 1001 006 FOOTER 1002 000 HEADER 1002 001 RAMESH 1002 002 100 1002 002 200 1002 002 500 1002 006 FOOTER my...

LEARN ABOUT HPUX

uuencode

uuencode(4)						     Kernel Interfaces Manual						       uuencode(4)

NAME

       uuencode - format of an encoded uuencode file

DESCRIPTION

       Files  output  by consist of a header line followed by a number of body lines, and a trailer line.  The command ignores any lines preceding
       the header or following the trailer (see uuencode(1)).  Lines preceding a header must not look like a header.

       The header line consists of the word followed by a space, a mode (in octal), another space, and a string which specifies the  name  of  the
       remote file.

       The  body  consists  of	a number of lines, each containing 62 or fewer characters (including trailing new-line).  These lines consist of a
       character count, followed by encoded characters, followed by a newline.

       The character count is a single printing character, which represents an integer.  This integer is the number of bytes in the  rest  of  the
       line, and always ranges from 0 to 63.  The byte count can be determined by subtracting the equivalent octal value of an ASCII space charac-
       ter (octal 40) from the character.

       Groups of 3 bytes are stored in 4 characters, 6 bits per character.  All are offset by a space to make the characters printable.  The  last
       line may be shorter than the normal 45 bytes.  If the size is not a multiple of 3, this fact can be determined by the value of the count on
       the last line.  Extra meaningless data will be included, if necessary, to make the character count a multiple of 4.  The body is terminated
       by a line with a count of zero.	This line consists of one ASCII space.

       The trailer line consists of the word on a line by itself.

SEE ALSO

       mail(1), uuencode(1), uucp(1).

																       uuencode(4)

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

print all even lines of a txt file

Discussion started by: ajp7701

2. Shell Programming and Scripting

Deleting lines that contain spaces in a txt file

Discussion started by: r04dw4rri0r

3. UNIX for Dummies Questions & Answers

Deleting lines in .txt with nonspecific value

Discussion started by: luis3141

4. Shell Programming and Scripting

script to delete lines from a txt file if pattern matches

Discussion started by: ajiwww