Shell Programming and Scripting

Sorry, the example was incorrect.

Basically, what I am calling the key is the field: <_05_1:MessageIdentifier>ERR:38736086_1215781057901</_05_1:MessageIdentifier> as the number within is always unique eg. 115781057901, 1215781057902, 1215781057903 and so on. In each file the data is placed after the key. Each file contains one type of data, so I am trying to report on the data by the key.

Originally, I have one file that contains all the data. So I egrep <_05_1:MessageIdentifier> and <Error:Exception> in one file, <_05_1:MessageIdentifier> and <06etail> in another and finally <_05_1:MessageIdentifier> and <DataPosted> in another. The reason I am doing this is because I am going to CUT the data to get what we want before I merge the files. If there is way of egreping all the fields and cutting each piece of data, that would sort my problem in one go.

<_05_1:MessageIdentifier>ERR:38736086_1215781057901</_05_1:MessageIdentifier>
<Error:Exception> Error was 121238123... </Error:Exeption>
<_05_1:MessageIdentifier>ERR:38736086_1215781057903</_05_1:MessageIdentifier>
<Error:Exception> Error was 4554641..... </Error:Exeption>
<_05_1:MessageIdentifier>ERR:38736086_1215781057905</_05_1:MessageIdentifier>
<Error:Exception> Error was 1277123.... </Error:Exeption>


<_05_1:MessageIdentifier>ERR:38736086_1215781057901</_05_1:MessageIdentifier>
<06etail> Code XYZ... </06etail>
<_05_1:MessageIdentifier>ERR:38736086_1215781057903</_05_1:MessageIdentifier>
<_05_1:MessageIdentifier>ERR:38736086_1215781057905</_05_1:MessageIdentifier>
<06etail> Code ABC... </06etail>
<06etail> Code AAA... </06etail>


<_05_1:MessageIdentifier>ERR:38736086_1215781057901</_05_1:MessageIdentifier>
<DataPosted> Data....... </DataPosted>
<DataPosted> Data....... </DataPosted>
<DataPosted> Data....... </DataPosted>
<_05_1:MessageIdentifier>ERR:38736086_1215781057903</_05_1:MessageIdentifier>
<DataPosted> Data....... </DataPosted>
<_05_1:MessageIdentifier>ERR:38736086_1215781057905</_05_1:MessageIdentifier>
<DataPosted> Data....... </DataPosted>
<DataPosted> Data....... </DataPosted>
<DataPosted> Data....... </DataPosted>
<DataPosted> Data....... </DataPosted>
<DataPosted> Data....... </DataPosted>
<DataPosted> Data....... </DataPosted>

Sorry, the example was incorrect.

Basically, what I am calling the key is the field: <_05_1:MessageIdentifier>ERR:38736086_1215781057901</_05_1:MessageIdentifier> as the number within is always unique eg. 115781057901, 1215781057902, 1215781057903 and so on. In each file the data is placed after the key. Each file contains one type of data, so I am trying to report on the data by the key.

Originally, I have one file that contains all the data. So I egrep <_05_1:MessageIdentifier> and <Error:Exception> in one file, <_05_1:MessageIdentifier> and <06etail> in another and finally <_05_1:MessageIdentifier> and <DataPosted> in another. The reason I am doing this is because I am going to CUT the data to get what we want before I merge the files. If there is way of egreping all the fields and cutting each piece of data, that would sort my problem in one go.

<_05_1:MessageIdentifier>ERR:38736086_1215781057901</_05_1:MessageIdentifier>
<Error:Exception> Error was 121238123... </Error:Exeption>
<_05_1:MessageIdentifier>ERR:38736086_1215781057903</_05_1:MessageIdentifier>
<Error:Exception> Error was 4554641..... </Error:Exeption>
<_05_1:MessageIdentifier>ERR:38736086_1215781057905</_05_1:MessageIdentifier>
<Error:Exception> Error was 1277123.... </Error:Exeption>


<_05_1:MessageIdentifier>ERR:38736086_1215781057901</_05_1:MessageIdentifier>
<06etail> Code XYZ... </06etail>
<_05_1:MessageIdentifier>ERR:38736086_1215781057903</_05_1:MessageIdentifier>
<_05_1:MessageIdentifier>ERR:38736086_1215781057905</_05_1:MessageIdentifier>
<06etail> Code ABC... </06etail>
<06etail> Code AAA... </06etail>


<_05_1:MessageIdentifier>ERR:38736086_1215781057901</_05_1:MessageIdentifier>
<DataPosted> Data....... </DataPosted>
<DataPosted> Data....... </DataPosted>
<DataPosted> Data....... </DataPosted>
<_05_1:MessageIdentifier>ERR:38736086_1215781057903</_05_1:MessageIdentifier>
<DataPosted> Data....... </DataPosted>
<_05_1:MessageIdentifier>ERR:38736086_1215781057905</_05_1:MessageIdentifier>
<DataPosted> Data....... </DataPosted>
<DataPosted> Data....... </DataPosted>
<DataPosted> Data....... </DataPosted>
<DataPosted> Data....... </DataPosted>
<DataPosted> Data....... </DataPosted>
<DataPosted> Data....... </DataPosted>

Something like this?

Code:

perl -ne'
  $key = $1 and next if /ERR:\d+_(\d+)/;
  $data{$key} = $data{$key} ?
    $data{$key} . "\n" . $1 :
      $1 if m|DataPosted>(.+?)</Data|;
  END {
    print map { $_ . "\n" . $data{$_} . "\n" } keys %data;
  }' logfile

Radoulov,

Thank you for persevering with my query, I really apprecaite. As I am new Shell Scripting could you please give me some idea of what each line is doing? I have some idea but I do not completely apprecaite the code. Where is the files to process specified?

Well,
try to execute it first passing your files as arguments:
(just copy/paste it on the command line)

Code:

perl -ne'
  $key = $1 and next if /ERR:\d+_(\d+)/;
  $data{$key} = $data{$key} ?
    $data{$key} . "\n" . $1 :
      $1 if m|DataPosted>(.+?)</Data|;
  END {
    print map { $_ . "\n" . $data{$_} . "\n" } keys %data;
  }' fileA fileB fileC ...

I'm trying to guess here, I'm not sure if you want only the lines inside the <DataPosted> tags and if they span over multiple lines.

Radoulov,

I need the data from all the files. For example, for the first key, I would want:

<_05_1:MessageIdentifier>ERR:38736086_1215781057901</_05_1:MessageIdentifier>
<Error:Exception> Error was 121238123... </Error:Exception>
<O6etail> Code XYZ... </O6etail>
<DataPosted> Data....... </DataPosted>
<DataPosted> Data....... </DataPosted>
<DataPosted> Data....... </DataPosted>

and so on

The code works and it does give me the data portion. Can it be expanded to give me the data from the other files too for each key?

1215781057901
Data.......
Data.......
Data.......
1215781057903
Data.......
1215781057905
Data.......
Data.......
Data.......
Data.......
Data.......
Data.......