How to extract block from a file?


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting How to extract block from a file?
# 1  
Old 12-17-2010
How to extract block from a file?

I have siebel log file as following
EventContext .......
123
.......
SELECT
...
..
EventConext <---- Question 1 , I should get this line
345
......
SELECT <----- Question 2 , print this line
Test..... <----- Question 2 , print this line
.... <----- Question 2 , print this line
ID 16318020: 111.678 seconds <----- Question 2 , print this line
SELECT
.......

1. Here all I know is string "ID 16318020: 111.678 seconds". I want to print EventContext line before this string(there may be more occurance of EventContext but I want to print only the first previous occurance from this line.


2. Also I have need to print block of SQL that start with last SELECT (one before string ID 16318020) and ending with ID 16318020.
e.g
SELECT
Test.....
....
ID 16318020: 111.678 seconds

I am sure some Guru will help me to solve this problem.

Thanks.
# 2  
Old 12-17-2010
This should work:
Code:
pattern="16318020: 111.678"
file=siebel.log
awk -v p="$pattern" '
/^EventContext/ { ec=NR }
/^SELECT/ { s=NR }
$0 ~ p {
        c=NR;
        cmd=sprintf("sed -n -e %dp -e %d,%dp %s",ec,s,c,FILENAME);
        system(cmd);
        exit(0);
}
' $file

As usual, replace awk by nawk when relevant.
# 3  
Old 12-17-2010
Thanks

Brilliant !! You saved a lot of work for me. Thank you so much.

Thanks
Ranjit
# 4  
Old 12-21-2010
Hi.

Here is an alternate solution. The nonstandard cgrep allows one to specify the edges of a window around a regular expression. The window edges may themselves be specified as regular expressions. So we can tell cgrep to look for the line matching the last line of interest, and then go backwards to find the preceding line of interest.

Then the sub-block needs to be deleted except for the first and last lines -- again a set of bounds or edges.

In the middle of this script is the call to cgrep, piped into a short perl script to do the sub-block work:
Code:
#!/usr/bin/env bash

# @(#) s2	Demonstrate block extraction, deletion with cgrep, perl.

# Section 1, setup, pre-solution.
# Infrastructure details, environment, commands for forum posts. 
# Uncomment export command to test script as external user.
# export PATH="/usr/local/bin:/usr/bin:/bin"
set +o nounset
pe() { for i;do printf "%s" "$i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }
C=$HOME/bin/context && [ -f $C ] && . $C specimen cgrep perl
set -o nounset
pe

FILE=${1-data1}

# Section 2, display input file.
# Display sample of data file, with head & tail as a last resort.
pe " || start [ first:middle:last ]"
specimen 8 $FILE \
|| { pe "(head/tail)"; head -n 5 $FILE; pe " ||"; tail -n 5 $FILE; }
pe " || end"

# Section 3, solution.
pl " Results (see file t1 for intermediate results):"
cgrep -D -w "EventContext" 'ID 16318020: 111.678 seconds' $FILE |
tee t1 |
perl -e '

while (<>) {
  # Check if we are in desired range.
  if ( $line = /EventContext/ .. /SELECT/ ) {
    if ( $line =~ /E0/ ) {
      print $_;
    }
    elsif ( $line == 1 ) {
      print $_;
    }    # Otherwise, delete by implication
    next;
  }
  # Print everything else.
  print;
}

'

exit 0

which, when run, produces:
Code:
% ./s2

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 2.6.26-2-amd64, x86_64
Distribution        : Debian GNU/Linux 5.0 (lenny) 
GNU bash 3.2.39
specimen (local) 1.17
cgrep - (local: ~/executable/cgrep May 29 2009 )
perl 5.10.0

 || start [ first:middle:last ]
Whole: 8:0:8 of 15 lines in file "data1"
EventContext (one) .......
123
.......
SELECT
...
..
EventContext (two) <---- Question 1 , I should get this line
345
......
SELECT <----- Question 2 , print this line
Test..... <----- Question 2 , print this line
.... <----- Question 2 , print this line
ID 16318020: 111.678 seconds <----- Question 2 , print this line
SELECT
.......
 || end

-----
 Results (see file t1 for intermediate results):
EventContext (two) <---- Question 1 , I should get this line
SELECT <----- Question 2 , print this line
Test..... <----- Question 2 , print this line
.... <----- Question 2 , print this line
ID 16318020: 111.678 seconds <----- Question 2 , print this line

As it turns out, extracting and deleting blocks is a common task, so one might create a script to do that based on parameters.

Changing the operative lines to do that in a script s3:
Code:
cgrep -D -w "EventContext" 'ID 16318020: 111.678 seconds' $FILE |
tee t1 |
debb "EventContext" "SELECT" d tt

produces:
Code:
% ./s3

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 2.6.26-2-amd64, x86_64
Distribution        : Debian GNU/Linux 5.0 (lenny) 
GNU bash 3.2.39
specimen (local) 1.17
cgrep - (local: ~/executable/cgrep May 29 2009 )
debb (local) 1.15

 || start [ first:middle:last ]
Whole: 8:0:8 of 15 lines in file "data1"
EventContext (one) .......
123
.......
SELECT
...
..
EventContext (two) <---- Question 1 , I should get this line
345
......
SELECT <----- Question 2 , print this line
Test..... <----- Question 2 , print this line
.... <----- Question 2 , print this line
ID 16318020: 111.678 seconds <----- Question 2 , print this line
SELECT
.......
 || end

-----
 Results (see file t1 for intermediate results):
EventContext (two) <---- Question 1 , I should get this line
SELECT <----- Question 2 , print this line
Test..... <----- Question 2 , print this line
.... <----- Question 2 , print this line
ID 16318020: 111.678 seconds <----- Question 2 , print this line

Best wishes ... cheers, drl

The source for cgrep can be found at: http://sourceforge.net/projects/cgrep/
# 5  
Old 05-04-2011
I want to know the use of system function in linux and also the difference between system() and backtrick term?
# 6  
Old 05-04-2011
Hi, Mac91.
Quote:
Originally Posted by Mac91
I want to know the use of system function in linux and also the difference between system() and backtrick term?
Welcome to the forum.

Your question will probably not receive much attention because you tacked it onto a thread that has already been answered and has nothing to do with your problem.

I suggest you start a new thread with your question.

You can do that by going to the head of this sub-forum (shell programming-scripting) and clicking the New Thread button ... cheers, drl
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Extract XML block when value is matched (Shell script)

Hi everyone, So i'm struggling with an xml (log file) where we get information about some devices, so the logfile is filled with multiple "blocks" like that. Based on the <devId> i want to extract this part of the xml file. If possible I want it to have an script for this, cause we'll use... (5 Replies)
Discussion started by: Pouky
5 Replies

2. Shell Programming and Scripting

How can I extract XML block around matching search string?

I want to extract XML block surrounding search string Ex: print XML block for string "myapp1-ear" surrounded by "<application> .. </application>" Input XML: <?xml version="1.0" encoding="UTF-8"?> <deployment-request> <requestor> <first-name>kchinnam</first-name> ... (16 Replies)
Discussion started by: kchinnam
16 Replies

3. Shell Programming and Scripting

Extract a block of text

Hello all, I am working on a script which should parse a large file called input.txt which contains table definitions, index definitions and comments like these ones: ------------------------------------------------ -- DDL Statements for table "CMWSYS"."CMWD_TEC_SUIVI_TRT"... (12 Replies)
Discussion started by: kiki_riki_miki
12 Replies

4. Shell Programming and Scripting

Printing a block of lines from a file, if that block does not contain two patterns using sed

I want to process a file block by block using sed, and if that block does not contain two patterns, then that complete block has to be printed. See below for the example data. ................................server 1............................... running process 1 running... (8 Replies)
Discussion started by: Kesavan
8 Replies

5. UNIX for Advanced & Expert Users

Move a block of lines to file if string found in the block.

I have a "main" file which has blocks of data for each user defined by tags BEGIN and END. BEGIN ID_NUM:24879 USER:abc123 HOW:47M CMD1:xyz1 CMD2:arp2 STATE:active PROCESS:id60 END BEGIN ID_NUM:24880 USER:def123 HOW:4M CMD1:xyz1 CMD2:xyz2 STATE:running PROCESS:id64 END (7 Replies)
Discussion started by: grep_me
7 Replies

6. Shell Programming and Scripting

[Awk] Extract block of with a particular pattern

Hi, I have some CVS log files, which are divided into blocks. Each block has many fields of information and I want to extract those blocks with a pattern. Here is the sample input. RCS file: /cvsroot/eclipse/org.eclipse.debug.core/core/org/eclipse/debug/core/DebugPlugin.java,v head: 1.174... (7 Replies)
Discussion started by: sandeepk1611
7 Replies

7. Shell Programming and Scripting

Extract a block of text??

Hello all, I have a large output file from which I would like to extract a single block of text. An example block of text is shown below: ***** EQUILIBRIUM GEOMETRY LOCATED ***** COORDINATES OF ALL ATOMS ARE (ANGS) ATOM CHARGE X Y Z ... (10 Replies)
Discussion started by: marcozd
10 Replies

8. Shell Programming and Scripting

Extract selective block from XML file

Hi, There's an xml file produced from a front-end tool as shown below: <INPUT DATABASE ="ORACLE" DBNAME ="UNIX" NAME ="FACT_TABLE" OWNERNAME ="DIPS"> <INPUTFIELD DATATYPE ="double" DEFAULTVALUE ="" DESCRIPTION ="" NAME ="STORE_KEY" PICTURETEXT ="" PORTTYPE ="INPUT" PRECISION ="15" SCALE... (6 Replies)
Discussion started by: dips_ag
6 Replies

9. Shell Programming and Scripting

Extract block of data and the error reason too. So so urgent

Hi , this is my first enty in our forum. Problem scenario: Using informatica tool am loding records from source DB to target DB. While loading some records getting rejected due to some reason. Informatica will capture those rejected records in session log file.now the session log ll be... (2 Replies)
Discussion started by: Gopal_Engg
2 Replies

10. UNIX for Dummies Questions & Answers

extract block in file

I need to extract a particular block from a file whose locations are not known but the only identity is a word. For example in a file I have ABC asdklf asdfk FGH dfdfg asdlfk asdfl ... JHK (5 Replies)
Discussion started by: sskb
5 Replies
Login or Register to Ask a Question