|
|
Sponsored Content
Top Forums
Shell Programming and Scripting
Extraction of data from multiple text files, and creation of a chart
Post 302330729 by ackheron on Thursday 2nd of July 2009 07:00:02 AM
07-02-2009
Extraction of data from multiple text files, and creation of a chart
Hello dear friends,
My problem as explained below seems really basic.
Fact is that I'm totally new to programming, and have only a week to produce a script ( CShell or Perl ? ) to perform this action.
While searching on the forums, I found a command that could help me, but I don't know how to use it properly :s.
But first of all . . . the context.
I have 7300 text files, each containing the same type of information.
Below is shown how the text files are written :
3 PARAMETERS ON 05JAN01.
1 AZU1 STA X -0.247297943318988E+07 +- 0.241394876950605E-02
2 AZU1 STA Y -0.467133800760033E+07 +- 0.374164031593362E-02
3 AZU1 STA Z 0.355810775092403E+07 +- 0.257172727253398E-02
2 1 0.571958818322482E+00
3 1 -0.659378461245129E+00
3 2 -0.847505892381651E+00
AZU1 ANTENNA LC 0.1636 0.0000 0.0000 !up north east (m)
In order to analyze all the data, I need to extract the bold values, and put each one in a different column of an Excel file (or equivalent) or in a text file ( but then with the three values on a line, and maybe try to create a csv file ).
To finish, I have a file like this in a different folder . . . . and there's 3 or 4 levels of difference in the arborescence. As :
desktop/data_2005/05JAN01/out/solution/05JAN01AZU1_fix.txt
desktop/data_2005/05JAN02/out/solution/05JAN02AZU1_fix.txt
and it goes on for the whole year . . .
The idea would be to use the 'grep' command I guess :
I tried with this "egrep "1|2|3" < 05JAN01AZU1_fix.stacov.txt | awk '{ print $5 }'>> output.txt" but it doesn't work properly. It did output cleanly the 3 values that I want when I changed the "1|2|3" to "10|20|30" and did the changed the values 1,2,3 to 10,20,30before the three values I wanted to extract.
So, basically . . . I have a text file for each day of the year, containing three values that I want to put in three separate columns of an excel file, or in the same line of the csv file.
I guess that I have to find the way to extract correctly my values, but also that I have to write a kind of loop, to go through all the folders(days) of the year.
CShell or Perl ?
Does anyone has one or more ideas to help me go further ?
If you do, I'll be pleased if you drop it there
Thanks . . .
P.S :I don't know who wrote the rules of the forum . . . the point n°15 "No smoking in the forums" made me smile . . .
My problem as explained below seems really basic.
Fact is that I'm totally new to programming, and have only a week to produce a script ( CShell or Perl ? ) to perform this action.
While searching on the forums, I found a command that could help me, but I don't know how to use it properly :s.
But first of all . . . the context.
I have 7300 text files, each containing the same type of information.
Below is shown how the text files are written :
3 PARAMETERS ON 05JAN01.
1 AZU1 STA X -0.247297943318988E+07 +- 0.241394876950605E-02
2 AZU1 STA Y -0.467133800760033E+07 +- 0.374164031593362E-02
3 AZU1 STA Z 0.355810775092403E+07 +- 0.257172727253398E-02
2 1 0.571958818322482E+00
3 1 -0.659378461245129E+00
3 2 -0.847505892381651E+00
AZU1 ANTENNA LC 0.1636 0.0000 0.0000 !up north east (m)
In order to analyze all the data, I need to extract the bold values, and put each one in a different column of an Excel file (or equivalent) or in a text file ( but then with the three values on a line, and maybe try to create a csv file ).
To finish, I have a file like this in a different folder . . . . and there's 3 or 4 levels of difference in the arborescence. As :
desktop/data_2005/05JAN01/out/solution/05JAN01AZU1_fix.txt
desktop/data_2005/05JAN02/out/solution/05JAN02AZU1_fix.txt
and it goes on for the whole year . . .
The idea would be to use the 'grep' command I guess :
I tried with this "egrep "1|2|3" < 05JAN01AZU1_fix.stacov.txt | awk '{ print $5 }'>> output.txt" but it doesn't work properly. It did output cleanly the 3 values that I want when I changed the "1|2|3" to "10|20|30" and did the changed the values 1,2,3 to 10,20,30before the three values I wanted to extract.
So, basically . . . I have a text file for each day of the year, containing three values that I want to put in three separate columns of an excel file, or in the same line of the csv file.
I guess that I have to find the way to extract correctly my values, but also that I have to write a kind of loop, to go through all the folders(days) of the year.
CShell or Perl ?
Does anyone has one or more ideas to help me go further ?
If you do, I'll be pleased if you drop it there
Thanks . . .
P.S :I don't know who wrote the rules of the forum . . . the point n°15 "No smoking in the forums" made me smile . . .
|
|
9 More Discussions You Might Find Interesting
1. UNIX for Advanced & Expert Users
hi everybody,
i have a file, in it I need to extract some data that follows a particular pattern..
For example: my file contains like
now running Speak225
sep 22 mon 16:34:05 2008
--------------------------------
... (4 Replies)
Discussion started by: mohkris
4 Replies
2. Shell Programming and Scripting
Hi,
I would like to seek for methods to do selective extraction of line froma file. The scenario as follows:
I have a file with content:
message a
received on 11:10:00
file size: 10 bytes
send by abc
message b
received on 11:20:00
file size: 10 bytes
send by abc (3 Replies)
Discussion started by: dwgi32
3 Replies
3. UNIX for Dummies Questions & Answers
I apologize if this has been posted before but I have been doing a lot of searching and I have not found any real answers, this is quite possibly due to that I may not be searching for the right terms or something. Little bit of info about my situation is that I'm a newbie as you probably guessed... (1 Reply)
Discussion started by: ummSomeNameHere
1 Replies
4. UNIX for Dummies Questions & Answers
Hi,
I'd like to process multiple files. For example:
file1.txt
file2.txt
file3.txt
Each file contains several lines of data. I want to extract a piece of data and output it to a new file.
file1.txt ----> newfile1.txt
file2.txt ----> newfile2.txt
file3.txt ----> newfile3.txt
Here is... (3 Replies)
Discussion started by: Liverpaul09
3 Replies
5. UNIX for Dummies Questions & Answers
Hello all,
I want to extract some particular data from a files and than add all the values .
but i m not able to cut the particular word(USU-INOCT and USU-OUTOCT) as it is coming not in column. and than able to add values coming in it .
can anyone help me Please
cat <file name>
... (7 Replies)
Discussion started by: anamdev
7 Replies
6. Shell Programming and Scripting
I have roughly ~30 .txt files in a directory which all have unique names. These files all contain text arranged in columns separated by whitespace (example file:
[#YY MM DD hh mm WDIR WSPD GST WVHT DPD APD MWD PRES ATMP WTMP DEWP VIS TIDE
#yr mo dy hr mn degT m/s m/s m sec ... (5 Replies)
Discussion started by: aozgaa
5 Replies
7. Shell Programming and Scripting
hi
i need to extract lines from multiple files to a csv file.
for example, i have these 3 files
file1.txt
date:29dec1980
caller:91245824255
called:8127766
file2.txt
date:11apr2014
caller:9155584558
called:8115478
file3.txt
date:25jun2015
caller:445225552
called:8117485 (30 Replies)
Discussion started by: lp.descamps
30 Replies
8. UNIX for Beginners Questions & Answers
i want to extract all data with in parenthesis from a file by passing a pattern from another file.i have some sql statements in my file and i want to extract those ddl by refering to a pattern which is in another file and before writting into file i need some transformation to do.Basically i want... (1 Reply)
Discussion started by: raj121
1 Replies
9. Shell Programming and Scripting
We have the data looks like below in a log file.
I want to generat files based on the string between two hash(#) symbol like below
Source:
#ext1#test1.tale2 drop
#ext1#test11.tale21 drop
#ext1#test123.tale21 drop
#ext2#test1.tale21 drop
#ext2#test12.tale21 drop
#ext3#test11.tale21 drop... (5 Replies)
Discussion started by: Sanjeev G
5 Replies
LEARN ABOUT REDHAT
set_color
set_color(1) fish set_color(1) NAME
set_color - set_color - set the terminal color set_color - set the terminal color Synopsis set_color [-v --version] [-h --help] [-b --background COLOR] [COLOR] Description Change the foreground and/or background color of the terminal. COLOR is one of black, red, green, brown, yellow, blue, magenta, purple, cyan, white and normal. o -b, --background Set the background color o -c, --print-colors Prints a list of all valid color names o -h, --help Display help message and exit o -o, --bold Set bold or extra bright mode o -u, --underline Set underlined mode o -v, --version Display version and exit Calling set_color normal will set the terminal color to whatever is the default color of the terminal. Some terminals use the --bold escape sequence to switch to a brighter color set. On such terminals, set_color white will result in a grey font color, while set_color --bold white will result in a white font color. Not all terminal emulators support all these features. This is not a bug in set_color but a missing feature in the terminal emulator. set_color uses the terminfo database to look up how to change terminal colors on whatever terminal is in use. Some systems have old and incomplete terminfo databases, and may lack color information for terminals that support it. Download and install the latest version of ncurses and recompile fish against it in order to fix this issue. Version 1.23.1 Sun Jan 8 2012 set_color(1)