Removing consecutive lines in a file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Removing consecutive lines in a file
# 1  
Old 10-05-2011
Question Removing consecutive lines in a file

We have very large transaction logs that have transactions which start with a line that starts with 'Begin :' and ends with a line that starts with 'End :'. For most transactions there is valid data between those two lines.

I am trying to get rid of lines that look like this:

Begin : Transaction Id 1245 04-Oct-2011 04:01:48 Username smith
End : Transaction Id 1245 04-Oct-2011 04:01:49

So, get rid of the first line, then get rid of the very next line if that next line starts with 'End :'. If there is data between, I want to preserve those lines throughout the entire file.

Thank you !
# 2  
Old 10-05-2011
Try:
Code:
perl -lp0e 's/^Begin.*\nEnd.*\n//mg' file

Didn't test it.
# 3  
Old 10-05-2011
Using awk
Code:
awk '/^End :/ && A { A=""; next }
  A { print A ; A="" }
  /^Begin :/ { A=$0 ; next }
  1' infile

# 4  
Old 10-06-2011
Both of these gave me syntax errors.

Perl: Substitution replacement not terminated at -e line 1.
awk: syntax error near line 1
awk: bailing out near line 1
# 5  
Old 10-06-2011
Quote:
Originally Posted by deneuve01
Both of these gave me syntax errors.

Perl: Substitution replacement not terminated at -e line 1.
awk: syntax error near line 1
awk: bailing out near line 1
use nawk (instead of awk) if on Solaris.
# 6  
Old 10-06-2011
nawk ran but did not change the file ...
# 7  
Old 10-06-2011
Is the format exactly as you posted in your exampe ie "Begin :" with a capital B and space between n and colon?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Delete all CONSECUTIVE text lines from file shell scripting

Hi I have a text file like below. THe content of the text will vary. Entire text file have four consecutive lines followed with blank line. I want to delete the occurrence of the two consicutive lines in the text file. I don't have pattern to match and delete. Just i need to delete all... (5 Replies)
Discussion started by: RJSKR28
5 Replies

2. Shell Programming and Scripting

Removing 3 consecutive lines from script

Hi I need to remove three consecutive lines of code which appear multiple times during a script. Two of the lines also appear in other parts of the scripts and need to stay so I can't strip out the code per se - It needs to be the exact three lines. Hope that makes sense ! Any help much... (5 Replies)
Discussion started by: Grueben
5 Replies

3. Shell Programming and Scripting

Grep 2 consecutive lines and replace the second line in a file

I have a file lake this cat ex1.txt </DISCOUNTS> <B2B_SPECIFICATION elem="0"> <B2B_SPECIFICATION elem="0"> <DESCR>Netti 2 </DESCR> <NUMBER>D02021507505</NUMBER> </B2B_SPECIFICATION> <B2B_SPECIFICATION elem="1"> <DESCR>Puhepaketti</DESCR>... (2 Replies)
Discussion started by: Dhoni
2 Replies

4. UNIX for Dummies Questions & Answers

Delete 26 consecutive lines in a file

I have a text file that is about 90,000 lines long. How would I delete lines 64-89, 152-177, 240-265, 328-353... etc? The sections I would like to delete are 26 lines long and the number of lines between the sections I would like to delete is 62 lines. Thanks very much in advance. (6 Replies)
Discussion started by: MDeBiasse
6 Replies

5. Shell Programming and Scripting

How to compare the values of a column in awk in a same file and consecutive lines..

I would like to compare the values of 2nd column of consecutive lines of same file in such a way so that if the difference between first value and second value is more than 100 it should print complete line else ignore line. Input File ========== PDB 2500 RTDB 123 RTDB-EAGLE 122 VSCCP 2565... (4 Replies)
Discussion started by: manuswami
4 Replies

6. Shell Programming and Scripting

Scan a file in realtime and execute certain commands on encountering 5 consecutive identical lines

Mysql log has something like below: I need a bash shell script that will do the following: 1) The script will scan the mysql.log file constantly in real time (something like tail -F mysql.log) 2) If it encounters 5 consecutive identical lines then it would invoke some commands (say... (4 Replies)
Discussion started by: proactiveaditya
4 Replies

7. Shell Programming and Scripting

how to delete two consecutive lines from the file

Hi guys I am deleting a unique line from the file and also need to remove the line above it which is NOT unique and servers as a record separator. Here is an example: # 101 803E 823F 8240 # 102 755f 4F2A 4F2B # 290 747D 0926 0927 # 999 8123 813E ... (5 Replies)
Discussion started by: aoussenko
5 Replies

8. Shell Programming and Scripting

Find time difference between two consecutive lines in same file.

Hello I have a file in following format: IV 08:09:07 NM 08:12:01 IC 08:12:00 MN 08:14:20 NM 08:14:15 I need a script to compare time on each line with previous line and show the inconsecutive line. Ex.: 08:12:00 08:14:15 A better way... (6 Replies)
Discussion started by: vilibit
6 Replies

9. Shell Programming and Scripting

merging of 2 consecutive lines in a file for a specific pattern

Hi , I'm looking for a way to merge two lines only for a given pattern / condition. Input : abcd/dad + -49.201 2.09 -49.5 34 ewrew rewtre * fdsgfds/dsgf/sdfdsfasdd + -4.30 0.62 -49.5 45 sdfdsf cvbbv * sdfds/retret/asdsaddsa + ... (1 Reply)
Discussion started by: novice_man
1 Replies

10. UNIX for Dummies Questions & Answers

Cutting n consecutive lines from a file...

Hi, I have this problem of separating 10 consecutive lines from a file, say starting from 21 to 30... I have used a filter like this.. head -n 30 myfile | tail -n 10 Is there a simpler way than this? (2 Replies)
Discussion started by: Vishnu
2 Replies
Login or Register to Ask a Question