Extraction of data from multiple text files, and creation of a chart


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Extraction of data from multiple text files, and creation of a chart
# 1  
Old 07-02-2009
Lightbulb Extraction of data from multiple text files, and creation of a chart

Hello dear friends,

My problem as explained below seems really basic.
Fact is that I'm totally new to programming, and have only a week to produce a script ( CShell or Perl ? ) to perform this action.

While searching on the forums, I found a command that could help me, but I don't know how to use it properly :s.

But first of all . . . the context.
I have 7300 text files, each containing the same type of information.
Below is shown how the text files are written :


3 PARAMETERS ON 05JAN01.
1 AZU1 STA X -0.247297943318988E+07 +- 0.241394876950605E-02
2 AZU1 STA Y -0.467133800760033E+07 +- 0.374164031593362E-02
3 AZU1 STA Z 0.355810775092403E+07 +- 0.257172727253398E-02
2 1 0.571958818322482E+00
3 1 -0.659378461245129E+00
3 2 -0.847505892381651E+00
AZU1 ANTENNA LC 0.1636 0.0000 0.0000 !up north east (m)


In order to analyze all the data, I need to extract the bold values, and put each one in a different column of an Excel file (or equivalent) or in a text file ( but then with the three values on a line, and maybe try to create a csv file ).

To finish, I have a file like this in a different folder . . . . and there's 3 or 4 levels of difference in the arborescence. As :

desktop/data_2005/05JAN01/out/solution/05JAN01AZU1_fix.txt
desktop/data_2005/05JAN02/out/solution/05JAN02AZU1_fix.txt
and it goes on for the whole year . . .

The idea would be to use the 'grep' command I guess :
I tried with this "egrep "1|2|3" < 05JAN01AZU1_fix.stacov.txt | awk '{ print $5 }'>> output.txt" but it doesn't work properly. It did output cleanly the 3 values that I want when I changed the "1|2|3" to "10|20|30" and did the changed the values 1,2,3 to 10,20,30before the three values I wanted to extract.

So, basically . . . I have a text file for each day of the year, containing three values that I want to put in three separate columns of an excel file, or in the same line of the csv file.

I guess that I have to find the way to extract correctly my values, but also that I have to write a kind of loop, to go through all the folders(days) of the year.

CShell or Perl ?

Does anyone has one or more ideas to help me go further ?
If you do, I'll be pleased if you drop it there Smilie
Thanks . . .

P.S :I don't know who wrote the rules of the forum . . . the point n°15 "No smoking in the forums" made me smile . . . Smilie
# 2  
Old 07-02-2009
Don't use cshell - sounds like you are in university... Do you have bash or ksh or even Bourne? There are very valid reasons for avoiding csh.

Here is a non- Cshell answer
Code:
find /desktop/data_2005  -name '*AZU1_fix.txt' |
while read filename
do
     awk '{ printf("%s ", $5)} END {printf("\n")} ' $filename
done | tr -s ' ' ',' > data_2005.csv

# 3  
Old 07-05-2009
Thank you very much Jim, it does work perfectly Smilie
I still have to search to understand fully how it works . . .Smilie I mean the syntax and everything . . . I would like to output the dates e.g. "05JAN01" in the first column aswell . . .

I do notice a difference between the values of the txt files and the values in the csv . . .
For instance when the first value is " -0.247297943318988E+07" (see the exemple in my first post ), it becomes "-2472979.90616501" (decimal part, it seems). I don't know yet if I actually need that precision . . . I'm checking this.

Any ideas ?

Your code will help me A LOT . . . Thank you again !
Ackheron.

Last edited by ackheron; 07-05-2009 at 08:46 AM..
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

In PErl script: need to read the data one file and generate multiple files based on the data

We have the data looks like below in a log file. I want to generat files based on the string between two hash(#) symbol like below Source: #ext1#test1.tale2 drop #ext1#test11.tale21 drop #ext1#test123.tale21 drop #ext2#test1.tale21 drop #ext2#test12.tale21 drop #ext3#test11.tale21 drop... (5 Replies)
Discussion started by: Sanjeev G
5 Replies

2. UNIX for Beginners Questions & Answers

Extraction of data between parenthesis using multiple pattern

i want to extract all data with in parenthesis from a file by passing a pattern from another file.i have some sql statements in my file and i want to extract those ddl by refering to a pattern which is in another file and before writting into file i need some transformation to do.Basically i want... (1 Reply)
Discussion started by: raj121
1 Replies

3. Shell Programming and Scripting

Read multiple text files and copy data to csv

hi i need to extract lines from multiple files to a csv file. for example, i have these 3 files file1.txt date:29dec1980 caller:91245824255 called:8127766 file2.txt date:11apr2014 caller:9155584558 called:8115478 file3.txt date:25jun2015 caller:445225552 called:8117485 (30 Replies)
Discussion started by: lp.descamps
30 Replies

4. Shell Programming and Scripting

Column extraction from multiple files to multiple files

I have roughly ~30 .txt files in a directory which all have unique names. These files all contain text arranged in columns separated by whitespace (example file: [#YY MM DD hh mm WDIR WSPD GST WVHT DPD APD MWD PRES ATMP WTMP DEWP VIS TIDE #yr mo dy hr mn degT m/s m/s m sec ... (5 Replies)
Discussion started by: aozgaa
5 Replies

5. UNIX for Dummies Questions & Answers

Need help for data extraction if files

Hello all, I want to extract some particular data from a files and than add all the values . but i m not able to cut the particular word(USU-INOCT and USU-OUTOCT) as it is coming not in column. and than able to add values coming in it . can anyone help me Please cat <file name> ... (7 Replies)
Discussion started by: anamdev
7 Replies

6. UNIX for Dummies Questions & Answers

Using AWK: Extract data from multiple files and output to multiple new files

Hi, I'd like to process multiple files. For example: file1.txt file2.txt file3.txt Each file contains several lines of data. I want to extract a piece of data and output it to a new file. file1.txt ----> newfile1.txt file2.txt ----> newfile2.txt file3.txt ----> newfile3.txt Here is... (3 Replies)
Discussion started by: Liverpaul09
3 Replies

7. UNIX for Dummies Questions & Answers

Data storing and chart

I apologize if this has been posted before but I have been doing a lot of searching and I have not found any real answers, this is quite possibly due to that I may not be searching for the right terms or something. Little bit of info about my situation is that I'm a newbie as you probably guessed... (1 Reply)
Discussion started by: ummSomeNameHere
1 Replies

8. Shell Programming and Scripting

Selective extraction of data from a files

Hi, I would like to seek for methods to do selective extraction of line froma file. The scenario as follows: I have a file with content: message a received on 11:10:00 file size: 10 bytes send by abc message b received on 11:20:00 file size: 10 bytes send by abc (3 Replies)
Discussion started by: dwgi32
3 Replies

9. UNIX for Advanced & Expert Users

extraction of data from a text file which follows certain pattern

hi everybody, i have a file, in it I need to extract some data that follows a particular pattern.. For example: my file contains like now running Speak225 sep 22 mon 16:34:05 2008 -------------------------------- ... (4 Replies)
Discussion started by: mohkris
4 Replies
Login or Register to Ask a Question