Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

awk - extracting data from a series of files

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting awk - extracting data from a series of files
# 1  
Old 03-24-2011
awk - extracting data from a series of files
Hi,
I am trying to extract data from multiple output files.

I am able to extract the data from a single output file by
using the following awk commands:
Code:
awk '/ test-file*/{print;m=0}' out1.log > out1a.txt
awk '/ test-string/{m=1;c=0}m&&++c==3{print  $2  "  "  $3   "  "  $4  ;m=0}' out1.log > out1b.txt
awk '/ test-string/{m=1;c=0}m&&++c==13{print  $2  "  "  $3   "  "  $4  ;m=0}' out1.log > out1c.txt

Now, how to extract data from multiple output files, say out1.log, out2.log, out3.log in a single step?

Any suggestions on how to combine the second and third awk commands to a single command [as the commands output 'line 3' and 'line 13' respectively, after the common 'string'].

Thanks for all your help!!Smilie
Last edited by Franklin52; 03-25-2011 at 04:10 AM.. Reason: Please use code tags
# 2  
Old 03-24-2011
Try:
Code:
awk '/ test-string/{m=1;c=0}m&&++c==3{print $2 " " $3 " " $4}m&&c==13{print $2 " " $3 " " $4 ;m=0}' out1.log > out1c.txt

# 3  
Old 03-24-2011
Quote:
Originally Posted by bartus11
Try:
Code:
awk '/ test-string/{m=1;c=0}m&&++c==3{print $2 " " $3 " " $4}m&&c==13{print $2 " " $3 " " $4 ;m=0}' out1.log > out1c.txt

Thanks, Bartus. The code works. Any thoughts about extracting data from multiple output files??Smilie
# 4  
Old 03-24-2011
Try:
Code:
for i in out1 out2 out3; do 
awk '/ test-file*/{print;m=0}' $i.log > ${i}a.txt
awk '/ test-string/{m=1;c=0}m&&++c==3{print $2 " " $3 " " $4}m&&c==13{print $2 " " $3 " " $4 ;m=0}' $i.log > ${i}c.txt
done

# 5  
Old 03-24-2011
Quote:
Originally Posted by bartus11
Try:
Code:
for i in out1 out2 out3; do 
awk '/ test-file*/{print;m=0}' $i.log > ${i}a.txt
awk '/ test-string/{m=1;c=0}m&&++c==3{print $2 " " $3 " " $4}m&&c==13{print $2 " " $3 " " $4 ;m=0}' $i.log > ${i}c.txt
done

Thanks Bartus!!

The code works well, and it writes out out1a.txt, out2a.txt, out2a.txt. If I need only one output (for the first awk command, out1a.txt) with all the data extracted sequentially into it from out1.log, out2.log and out3.log, what do I need to do? When I replace '${i}' with out1a.txt, only data from out1.log gets extracted. Thank You!Smilie
# 6  
Old 03-24-2011
Code:
rm -f outa.txt
for i in out1 out2 out3; do 
   awk '/ test-file*/{print;m=0}' $i.log >> outa.txt
   awk '/ test-string/{m=1;c=0}m&&++c==3{print $2 " " $3 " " $4}m&&c==13{print $2 " " $3 " " $4 ;m=0}' $i.log > ${i}c.txt
done

# 7  
Old 03-24-2011
Bug
Quote:
Originally Posted by Chubler_XL
Code:
rm -f outa.txt
for i in out1 out2 out3; do 
   awk '/ test-file*/{print;m=0}' $i.log >> outa.txt
   awk '/ test-string/{m=1;c=0}m&&++c==3{print $2 " " $3 " " $4}m&&c==13{print $2 " " $3 " " $4 ;m=0}' $i.log > ${i}c.txt
done

Hi Chubler,
It doesn't do the trick. I guess it is overwriting the first output. Any other suggestions? Thank youSmilie
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Problem in extracting data using cut/awk command

Hi Everyone, I have a very simple problem and i am stuck in that from last 8 days. I tried many attempts, googled my query but all in vain. I have a text file named "test.txt" In that suppose i have contents like: Java: 1 Object oriented programming language 2 Concepts of Abstraction... (5 Replies)
Discussion started by: Abhijeet Anand
5 Replies

2. Shell Programming and Scripting

Extracting part of data from files

Hi All, I have log files as below. log1.txt <table name="content_analyzer" primary-key="id"> <type="global" /> </table> <table name="content_analyzer2" primary-key="id"> <type="global" /> </table> Time taken: 1.008 seconds ID = gd54321bbvbvbcvb <table name="content_analyzer"... (7 Replies)
Discussion started by: ROCK_PLSQL
7 Replies

3. Programming

Python script for extracting data using two files

Hello, I have two files. File 1 is a list of interested IDs Ex1 Ex2 Ex3File 2 is the original file with over 8000 columns and 20 millions rows and is a compressed file .gz Ex1 xx xx xx xx .... Ex2 xx xx xx xx .... Ex2 xx xx xx xx ....Now I need to extract the information for all the IDs of... (4 Replies)
Discussion started by: nans
4 Replies

4. Programming

Resample time series data with replacement any way to do this in awk or just bash script

I have some time series data that I need to resample or downsample at some specific time intervals. The firs column is time in decimal hours. I am tryiong to resample this data every 3 minutse. So I need a data value ever 0.05. Here is the example data and as you can see, there time slot for 0.1500... (3 Replies)
Discussion started by: malandisa
3 Replies

5. Shell Programming and Scripting

Using Awk for extracting data in specific format

please help me writing a awk script 001_r.pdb 0.0265185 001_r.pdb 0.0437049 001_r.pdb 0.0240642 001_r.pdb 0.0310264 001_r.pdb 0.0200482 001_r.pdb 0.0146746 001_r.pdb 0.0351344 001_r.pdb 0.0347856 001_r.pdb 0.036119 001_r.pdb 1.49 002_r.pdb 0.0281011 002_r.pdb 0.0319908 002_r.pdb... (5 Replies)
Discussion started by: phoenix_nebula
5 Replies

6. UNIX for Dummies Questions & Answers

Extracting data from many compressed files

I have a large number (50,000) of pretty large compressed files and I need only certain lines of data from them (each relevant line contains a certain key word). Each file contains 300 such lines. The individual file names are indexed by file number (file_name.1, file_name.2, ... ,... (1 Reply)
Discussion started by: Boltzmann
1 Replies

7. UNIX for Dummies Questions & Answers

Iterate a min/max awk script over time-series temperature data

I'm trying to iterate a UNIX awk script that returns min/max temperature data for each day from a monthly weather data file (01_weath.dat). The temperature data is held in $5. The temps are reported each minute so each day contains 1440 temperature enteries. The below code has gotten me as far as... (5 Replies)
Discussion started by: jgourley
5 Replies

8. Shell Programming and Scripting

calculate the average of time series data using AWK

Hi, I have two time series data (below) merged into a file. t1 and t2 are in unit of second I want to calculate the average of V1 every second and count how many times "1" in V2 is occur within a second Input File: t1 V1 t2 V2 10.000000... (5 Replies)
Discussion started by: nica
5 Replies

9. Shell Programming and Scripting

extracting data from files..

frnds, I m having prob woth doing some 2-3 task simultaneously... what I want is... I have lots ( lacs ) of files in a dir... I want.. these info from arround 2-3 months files filename convention is - abc20080403sdas.xyz ( for todays files ) I want 1. total no of files for 1 dec... (1 Reply)
Discussion started by: clx
1 Replies

10. Shell Programming and Scripting

Perl - extracting data from .csv files

PROJECT: Extracting data from an employee timesheet. The timesheets are done in excel (for user ease) and then converted to .csv files that look like this (see color code key below): ,,,,,,,,,,,,,,,,,,, 9/14/2003,<-- Week Ending,,,,,,,,,,,,,,,,,, Craig Brennan,,,,,,,,,,,,,,,,,,,... (3 Replies)
Discussion started by: kregh99
3 Replies
Login or Register to Ask a Question