Extracting part of data from files Post: 302978004

Sponsored Content

Top Forums Shell Programming and Scripting Extracting part of data from files Post 302978004 by ROCK_PLSQL on Monday 25th of July 2016 08:14:36 AM

07-25-2016

Banned

Extracting part of data from files

Hi All,

I have log files as below.

Code:

log1.txt

<table name="content_analyzer" primary-key="id">
  <type="global" />
</table>
<table name="content_analyzer2" primary-key="id">
  <type="global" />
</table>
Time taken: 1.008 seconds
ID = gd54321bbvbvbcvb

<table name="content_analyzer" unique-key="code">
  <type="global" />
</table>
<table name="content_analyzer2" primary-key="sid">
  <type="global" />
</table>
Table astcms.dfile1 stats: [numberoffiles=2, blocks=5, numberofrows=2000, totalSize=589, rawDataSize=6779]
OK
Time taken: 10.23 seconds
ID = fffgs5667uyt

<table name="content_analyzer" primary-key="did">
  <type="global" />
</table>
<table name="content_analyzer2" unique-key="scod">
  <type="global" />
</table>
Table astcms.dfile2 stats: [numberoffiles=5, numberofrows=100, totalSize=68, rawDataSize=89]
OK
Time taken: 15.12 seconds
FAILED:Exception [Error 56700]

log2.txt

Time taken: 2.005 seconds
ID = gd54321bbvbvbcvb

<table name="content_analyzer" primary-key="did">
  <type="global" />
</table>
<table name="content_analyzer2" unique-key="scod">
  <type="global" />
</table>
Table astcms.dfile4 stats: [numberoffiles=3, numberofrows=60, totalSize=10, rawDataSize=15]
OK
Time taken: 12.123 seconds

This is my code.

Code:

for filename in /home/usr/*.log
do
echo "$filename"
sed 's/[[]//g;s/[]]//g' < $filename | 
     awk '
     {  if (index($0, "stats" ) > 0)
        {
         { for(i=1;i<=NF; i++)                               
             {               
                if( $(i) ~ /numberoffiles/ || $(i) ~ /numberofrows/ || $(i) "Table") 
                {
                 printf("%s ", $(i) ) 
                }
             }
         }
       }
       if (index($0, "Time")> 0 )  
       {
         print $0
       } 
      if (index($0, "FAILED:")> 0 )  
       {
              print $0
	  }
     } ' 
   done

I am getting output as below.

Code:

/home/usr/log1.txt
Time taken: 1.008
Table astcms.dfile1 stats: numberoffiles=2, blocks=5, numberofrows=2000, totalSize=589, rawDataSize=6779 Time taken: 10.23 seconds
Table astcms.dfile2 stats: numberoffiles=5, numberofrows=100, totalSize=68, rawDataSize=89 Time taken: 15.12 seconds
FAILED:Exception [Error 56700]
/home/usr/log2.txt
Time taken: 2.005 seconds
Table astcms.dfile4 stats: numberoffiles=3, numberofrows=60, totalSize=10, rawDataSize=15 Time taken: 12.123 seconds

I want the output as below.

Code:

astcms,dfile1,2,2000,10.23,PASS,/home/usr/log1.txt
astcms,dfile2,5,100,15.12,PASS,/home/usr/log1.txt
NULL,NULL,NULL,NULL,NULL,FAILED:Exception [Error 56700],/home/usr/log1.txt
astcms,dfile4,3,60,12.123,PASS,/home/usr/log2.txt

Please help me.

Thanks in Advance.

Last edited by ROCK_PLSQL; 07-25-2016 at 09:25 AM..

ROCK_PLSQL

View Public Profile for ROCK_PLSQL

Find all posts by ROCK_PLSQL

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Perl - extracting data from .csv files

PROJECT: Extracting data from an employee timesheet. The timesheets are done in excel (for user ease) and then converted to .csv files that look like this (see color code key below): ,,,,,,,,,,,,,,,,,,, 9/14/2003,<-- Week Ending,,,,,,,,,,,,,,,,,, Craig Brennan,,,,,,,,,,,,,,,,,,,...

2. Shell Programming and Scripting

extracting uncommon part between two files

Hi, I need to extract the uncommon (better say incremental) part from 2 files say file_1 and file_2. file_2 contains everything that is in file_1. That is file_2 has been created internally somehow : cat file_1 temp_file > file_2 My objective is to extract the temp_file part from...

3. Shell Programming and Scripting

extracting data from files..

frnds, I m having prob woth doing some 2-3 task simultaneously... what I want is... I have lots ( lacs ) of files in a dir... I want.. these info from arround 2-3 months files filename convention is - abc20080403sdas.xyz ( for todays files ) I want 1. total no of files for 1 dec...

4. UNIX for Dummies Questions & Answers

Extracting data from many compressed files

I have a large number (50,000) of pretty large compressed files and I need only certain lines of data from them (each relevant line contains a certain key word). Each file contains 300 such lines. The individual file names are indexed by file number (file_name.1, file_name.2, ... ,...

5. UNIX for Dummies Questions & Answers

Finding and Extracting uniq data in multiple files

Hi, I have several files that look like this: File1.txt Data1 Data2 Data20 File2.txt Data1 Data5 Data10 File3.txt Data1 Data2 Data17 File4.txt

6. Shell Programming and Scripting

awk - extracting data from a series of files

Hi, I am trying to extract data from multiple output files. I am able to extract the data from a single output file by using the following awk commands: awk '/ test-file*/{print;m=0}' out1.log > out1a.txt awk '/ test-string/{m=1;c=0}m&&++c==3{print $2 " " $3 " " $4 ;m=0}' out1.log >...

7. UNIX for Dummies Questions & Answers

Extracting data from PDF files into CSV file

Hi, I have several hundreds of PDFfiles number 01.pdf, 02.pdf, 03.pdf, etc in one folder. These are vey long documentd with a lot of information (text, tables, figures, etc). I need to extract the information asociated with one disease in particular (Varicella). The information I need to...

8. Shell Programming and Scripting

Extracting Delimiter 'TAG' Data From log files

Hi I am trying to extract data from within a log file and output format to a new file for further manipulation can someone provide script to do this? For example I have a file as below and just want to extract all delimited variances of tag 32=* up to the delimiter "|" and output to a new file...

9. Programming

Python script for extracting data using two files

Hello, I have two files. File 1 is a list of interested IDs Ex1 Ex2 Ex3File 2 is the original file with over 8000 columns and 20 millions rows and is a compressed file .gz Ex1 xx xx xx xx .... Ex2 xx xx xx xx .... Ex2 xx xx xx xx ....Now I need to extract the information for all the IDs of...

10. Shell Programming and Scripting

Extracting data from specific rows and columns from multiple csv files

I have a series of csv files in the following format eg file1 Experiment Name,XYZ_07/28/15, Specimen Name,Specimen_001, Tube Name, Control, Record Date,7/28/2015 14:50, $OP,XYZYZ, GUID,abc, Population,#Events,%Parent All Events,10500, P1,10071,95.9 Early Apoptosis,1113,11.1 Late...

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Perl - extracting data from .csv files

Discussion started by: kregh99

2. Shell Programming and Scripting

extracting uncommon part between two files

Discussion started by: sabyasm

3. Shell Programming and Scripting

extracting data from files..

Discussion started by: clx

4. UNIX for Dummies Questions & Answers

Extracting data from many compressed files

Discussion started by: Boltzmann

5. UNIX for Dummies Questions & Answers

Finding and Extracting uniq data in multiple files

Discussion started by: Fahmida

6. Shell Programming and Scripting

awk - extracting data from a series of files

Discussion started by: p_sun

7. UNIX for Dummies Questions & Answers

Extracting data from PDF files into CSV file

Discussion started by: Xterra

8. Shell Programming and Scripting

Extracting Delimiter 'TAG' Data From log files

Discussion started by: Buddyluv

9. Programming

Python script for extracting data using two files

Discussion started by: nans

10. Shell Programming and Scripting

Extracting data from specific rows and columns from multiple csv files

Discussion started by: pawannoel