Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Last occurrence of code between two tags

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Last occurrence of code between two tags
# 15  
Old 01-20-2012
Perfect ! Both commands are working perfect for the example that I posted. \

But it is not working if I change the script to my actual.

Some useful points about my real logs..
1. Responses can have some names like ListABCRes, ABCRes, DDARes, etc..
where in example log I gave you response1,response2 etc ..

To get ListABCRes, I changed your command as below

Code:
ddddddd$ /usr/xpg4/bin/awk '$0~"<"ListABCRes{sect=1}
/admin/home/abc>      sect{p=p (p?RS:x) $0}
/admin/home/abc>      $0~"<user>"user && sect{found=1}
/admin/home/abc>      $0~"</"Res{if(found)n=p;found=0;p=x;sect=0}
/admin/home/abc>      END{print n}' resp=ListABCRes user=narayana12345 infile

I am getting the error like below

Code:
/usr/xpg4/bin/awk: line 3 (NR=8499138): Record too long (LIMIT: 19999 bytes)

So, I thought I should need to filter something and so I changed like below

Code:
ddddddd$ /usr/xpg4/bin/awk '$0~"<"ListABCRes{sect=1}
/admin/home/abc>      sect{p=p (p?RS:x) $0}
/admin/home/abc>      $0~"<user>narayana"user && sect{found=1}
/admin/home/abc>      $0~"</"Res{if(found)n=p;found=0;p=x;sect=0}
/admin/home/abc>      END{print n}' resp=ListABCRes user=12345 infile

it is still having the same error.

Responses are big and There are too many responses for one user and too many same responses for different users say for example many users can have ListABCRes response and one user can have ListABCRes in the logs many times.

So do you suggest anything on this ?
Last edited by nariwithu; 01-20-2012 at 05:17 PM..
# 16  
Old 01-23-2012
Mirni or Scrutinizer,

Thanks for your continuous effort in helping me.

Do you have anything for me to do to rewrite the command to work on my real logs ?
# 17  
Old 01-23-2012
You did not modify Scrutinizer's code well. His code was as follows:
Code:
/usr/xpg4/bin/awk '$0~"<"resp{sect=1} 
   sect{p=p (p?RS:x) $0} 
   $0~"<user>"user && sect{found=1} 
   $0~"</"resp{if(found)n=p;found=0;p=x;sect=0}
   END{print n}' resp=response2 user=user1 infile

This is setting a variable 'resp' to have the value 'response2', and variable 'user' to a string 'user1'. Do not change the code inside the single quotes, just change the "response2" to what is your response actually named. If you are hunting a section
Code:
<ListABCResp>
...
</ListABCResp>

, with user "user1", then you'd do:
Code:
/usr/xpg4/bin/awk '$0~"<"resp{sect=1} 
   sect{p=p (p?RS:x) $0} 
   $0~"<user>"user && sect{found=1} 
   $0~"</"resp{if(found)n=p;found=0;p=x;sect=0}
   END{print n}' resp=ListABCResp user=user1 infile

However, I don't see why you would be getting "record too long" error... the code is appending to an internal variable. Have you tried to preprocess the file with the sed command and then pipe it to awk as suggested before?
Last edited by mirni; 01-23-2012 at 03:18 PM..
# 18  
Old 01-23-2012
Mirni,

It is still getting me record too long exception with the following code as you suggested.

Code:
/usr/xpg4/bin/awk '$0~"<"resp{sect=1} sect{p=p (p?RS:x) $0} $0~"<user>"user && sect{found=1} $0~"</"resp{if(found)n=p;found=0;p=x;sect=0}END{print n}' resp=ListABCResp user=user1 infile

I tried to preprocess with sed , then also it is giving me the same exception
Code:
Record too long (LIMIT: 19999 bytes)

Code:
sed -e 's/.*<ListABCResp[0-9]*>.*/\n&/; s/<\/ListABCResp[0-9]*>/&\n/' infile | /usr/xpg4/bin/awk '$0~"<"resp{sect=1} sect{p=p (p?RS:x) $0} $0~"<user>"user && sect{found=1} $0~"</"resp{if(found)n=p;found=0;p=x;sect=0}END{print n}' resp=ListABCResp user=user1 infile

getting same error with the following code also

Code:
sed -e 's/.*<ListABCResp>.*/\n&/; s/<\/ListABCResp>/&\n/' infile | /usr/xpg4/bin/awk '$0~"<"resp{sect=1} sect{p=p (p?RS:x) $0} $0~"<user>"user && sect{found=1} $0~"</"resp{if(found)n=p;found=0;p=x;sect=0}END{print n}' resp=ListABCResp user=user1 infile

I am again noting that one user can have two or more ListABCResp in the logs, Logs can be big, Response can be big and Many users can have ListABCResp responses. Similarly other responses also.

I have no clue how to proceed. Hope you will help me again.
# 19  
Old 01-24-2012
I recommend getting GNU awk, it may prove useful in the future also. You can download it from here:
ftp://ftp.gnu.org/gnu/gawk/gawk-3.1.8.tar.gz
and then standard build:
Code:
tar xvzf gawk-3.1.8.tar.gz
cd gawk-3.1.8
./configure
make
make install

If you don't have the privilege to install it system wide, just install it in a non-standard directory:
Code:
./configure --prefix=/path/to/MyDir
make && make install

---------- Post updated at 06:54 PM ---------- Previous update was at 06:34 PM ----------

Do you have perl?
Code:
r=response1 #substitute with the real name
u=user1  #substitute with the real name
sed -e '/.*<'$r'*>.*/ {x;p;x}' -e '/<\/'$r'*>/ {G}' logFile | 
perl -ne '$/="\n\n"; if(/<'$r'/&&/<user>'$u'/){print;}'

# 20  
Old 01-24-2012
Hi Mirni,

Downloaded gawk-3.1.8.tar.gz
Extracted in windows machine as tar was not working for me.
uploaded the extracted gawk-3.1.8 folder to my directory in unix machine.
ran the command ./configure make make install

I am getting the following error while installing.

Code:
 ./configure make && make install
configure: WARNING: you should use --build, --host, --target
checking for a BSD-compatible install... ./install-sh -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... ./install-sh -c -d
checking for gawk... no
checking for mawk... no
checking for nawk... nawk
checking whether make sets $(MAKE)... yes
checking for style of include used by make... GNU
checking for make-gcc... no
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking dependency style of gcc... gcc3
checking how to run the C preprocessor... gcc -E
checking for grep that handles long lines and -e... /usr/xpg4/bin/grep
checking for egrep... /usr/xpg4/bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking minix/config.h usability... no
checking minix/config.h presence... no
checking for minix/config.h... no
checking whether it is safe to define __EXTENSIONS__... yes
checking for egrep... (cached) /usr/xpg4/bin/grep -E
checking for bison... no
checking for byacc... no
checking whether ln -s works... yes
checking for make-gcc... gcc
checking whether we are using the GNU C compiler... (cached) yes
checking whether gcc accepts -g... (cached) yes
checking for gcc option to accept ISO C89... (cached) none needed
checking dependency style of gcc... (cached) gcc3
checking how to run the C preprocessor... gcc -E
checking whether make sets $(MAKE)... (cached) yes
checking for special development options... no
checking for z/OS USS compilation... no
checking for strerror in -lcposix... no
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... 64
checking for AIX compilation hacks... no
checking for Linux/Alpha compilation hacks... no
checking for function prototypes... yes
checking for string.h... (cached) yes
checking whether NLS is requested... yes
checking for msgfmt... no
checking for gmsgfmt... :
checking for xgettext... no
checking for msgmerge... no
checking build system type... Invalid configuration `make': machine `make' not recognized
configure: error: /bin/bash ./config.sub make failed

Quote:
Do you have perl?
I have perl installed. When I ran your suggested perl code by modifying like below, I am getting the following error.

Code:
r=ListABCResp #substitute with the real name
u=user1 #substitute with the real name
sed -e '/.*<'$r'*>.*/ {x;p;x}' -e '/<\/'$r'*>/ {G}' abc.log  | 
perl -ne '$/="\n\n"; if(/<'$r'/&&/<user>'$u'/){print;}'

Code:
$ sed -e '/.*<'$r'*>.*/ {x;p;x}' -e '/<\/'$r'*>/ {G}' abc.log | 
perl -ne '$/="\n\n"; if(/<'$r'/&&/<user>'$u'/){print;}'
sed: command garbled: /.*<ListABCResp*>.*/ {x;p;x}

do you have anything to suggest ?
# 21  
Old 01-25-2012
Quote:
Code:
./configure make make install

This is not correct. These are three steps; you have to run them one at a time:
Code:
./configure

then
Code:
make

then
Code:
make install

As for the garbled command... hmm... interesting. It is working ok, with my version of sed. Please post the output of
Code:
sed --version | head -n 1

And try to take out the spaces and add a semicolon. Like this:
Code:
sed -e '/.*<'$r'>.*/{x;p;x}' -e '/<\/'$r'>/G'

If that still comes out as garbled, try to put the commands one per line:
Code:
sed -e '/.*<'$r'>.*/{x
p
x
}' -e '/<\/'$r'>/G' file

---------- Post updated at 03:26 AM ---------- Previous update was at 03:05 AM ----------

Although my perl skills are not very developed at all, I wrote a pure perl script; give this a try:
Code:
#!/usr/bin/perl -w

use strict; 

my  @out = ();
my  @buf = ();

my $start = 0;
my $r = "response1";
my $u = "user1";

while(<>) {
    if(/<$r>/) {
      $start = 1;
    }
    if($start) {
      push(@buf,$_);
    }
    if(/<\/$r>/) {
      $start = 0;
      foreach (@buf) {
        if(/<user>$u/) {
          @out = @buf;
          last;
        }
      }
    @buf = ();
    }
}
print @out;

This User Gave Thanks to mirni For This Post:
nariwithu
Login or Register to Ask a Question

Previous Thread | Next Thread

5 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

sed print from last occurrence match until the end of last occurrence match

Hi, i have file file.txt with data like: START 03:11:30 a 03:11:40 b END START 03:13:30 eee 03:13:35 fff END jjjjjjjjjjjjjjjjjjjjj START 03:14:30 eee 03:15:30 fff END ggggggggggg iiiiiiiiiiiiiiiiiiiiiiiii I want the below output START (13 Replies)
Discussion started by: Jyotshna
13 Replies

2. UNIX for Dummies Questions & Answers

Code for exact match to count occurrence

Hi all, I have an input file as below. I would like to count the occurrence of pattern matching 8th field for each line. Input: field_01 field_02 field_03 field_04 field_05 field_06 field_07 field_08 TA T TA T TA TA TA... (3 Replies)
Discussion started by: huiyee1
3 Replies

3. Shell Programming and Scripting

Substitute first occurrence of keyword if occurrence between two other keywords

Assume a string that contains one or multiple occurrences of three different keywords (abbreviated as "kw"). I would like to replace kw2 with some other string, say "qux". Specifically, I would like to replace that occurrence of kw2 that is the first one that is preceded by kw1 somewhere in the... (4 Replies)
Discussion started by: M Gruenstaeudl
4 Replies

4. Shell Programming and Scripting

number of occurrence

: i need a bash script to convert the displayed output 12 14 15 12 15 13 to 12 * 2 ,13 * 1,14*1,15*1 Thanks, nevil (2 Replies)
Discussion started by: nevil
2 Replies

5. Shell Programming and Scripting

Replace second occurrence only

HPUX /bin/sh (posix) I have a file as such cat dog mouse deer elk rabbit mouse rat pig I would like to replace the second occurrence of mouse in this file with mouse2. The rest of the file has to stay exactly as is. I'm not sure exactly where mouse might be (could be first,second,third... (5 Replies)
Discussion started by: lyoncc
5 Replies
Login or Register to Ask a Question