Column extraction from multiple files to multiple files


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Column extraction from multiple files to multiple files
# 1  
Old 10-18-2012
Column extraction from multiple files to multiple files

I have roughly ~30 .txt files in a directory which all have unique names. These files all contain text arranged in columns separated by whitespace (example file:
[#YY MM DD hh mm WDIR WSPD GST WVHT DPD APD MWD PRES ATMP WTMP DEWP VIS TIDE
#yr mo dy hr mn degT m/s m/s m sec sec deg hPa degC degC degC nmi ft
2012 05 31 23 50 60 0.8 0.9 0.04 99.00 2.77 999 1016.7 7.1 5.3 999.0 99.0 99.00
2012 06 01 00 50 30 1.8 2.2 0.04 99.00 2.77 999 1016.8 6.0 5.3 999.0 99.0 99.00
2012 06 01 01 50 42 0.7 1.3 0.13 99.00 3.98 999 1016.4 6.4 4.8 999.0 99.0 99.00
2012 06 01 02 50 105 2.3 2.6 0.04 99.00 3.46 999 1016.5 5.3 4.6 999.0 99.0 99.00
...
...
...
)
I am looking to extract columns from these files (ie: PRES, which runs from the 54-59 char in each line) so that i can find their average value accross each file. I would like these outputs to remain distinct and indentifiable to the original file.
In the end, I am looking to extract data from the files with the following names:
4500172012.txt
4500182012.txt
4500262012.txt
4500272012.txt
4500282012.txt
4500362012.txt
4500372012.txt
4500382012.txt
4500462012.txt
4500472012.txt
4500482012.txt
4500562012.txt
4500572012.txt
4500582012.txt
4500662012.txt
4500672012.txt
4500682012.txt
4500762012.txt
4500772012.txt
4500782012.txt
4500862012.txt
4500872012.txt
4500882012.txt
4501262012.txt
4501272012.txt
4501282012.txt

I am looking to extract the columns PRES, ATMP, and WTMP (char 54-59, 62-65, 68-71 respectively).


How can I do this?

Thank you for any help you can provide.
# 2  
Old 10-18-2012
Columns PRES, ATMP, and WTMP can be extracted and written to another file using below code:-

Code:
echo "PRES ATMP WTMP FILE" > final_extract.txt

for file in *2012.txt
do
   cat ${file} | egrep -v "^#|PRES" | awk -v FNAME=$file ' { print $13 " " $14 " " $15 " " FNAME } ' >> final_extract.txt
done

Added file name to final_extract.txt just in case if you want to know from which file the values were extracted.
# 3  
Old 10-18-2012
bipinajith,
This seems to take all the files and output the result into a single file. I need the results to all be mapped to unique files, separate for each file. In this case, it is fine if the original file is overwritten (i have copies of the orginal files in multiple directories). Therein lies the difficulty for me, a noobie with scripting.

---------- Post updated at 05:04 PM ---------- Previous update was at 05:02 PM ----------

Basically, I need some way of knowing which file my data originally came from. If there is some other way of doing this I am not aware of, then that could work as well.
# 4  
Old 10-18-2012
Code:
 
for fl in *2012.txt
do
  awk 'BEGIN{OFS="\t";print "PRES","ATMP","WTMP"}$1 !~ /#/ {print $13,$14,$15}' $fl > new_$fl
done


Last edited by rdrtx1; 10-18-2012 at 07:40 PM..
# 5  
Old 10-18-2012
This works perfectly. Thank you!
# 6  
Old 10-18-2012
rdrtx1,
That works perfectly. Thank you!
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

How to copy a column of multiple files and paste into new excel file (next to column)?

I have data of an excel files as given below, file1 org1_1 1 1 2.5 100 org1_2 1 2 5.5 98 org1_3 1 3 7.2 88 file2 org2_1 1 1 2.5 100 org2_2 1 2 5.5 56 org2_3 1 3 7.2 70 I have multiple excel files as above shown. I have to copy column 1, column 4 and paste into a new excel file as... (26 Replies)
Discussion started by: dineshkumarsrk
26 Replies

2. Shell Programming and Scripting

Grep strings on multiple files and output to multiple files

Hi All, I want to use egrep on multiple files and the results should be output to multiple files. I am using the below code in my shell script(working in Ksh shell). However with this code I am not attaining the desired results. #!/bin/ksh ( a="/path/file1" b="path/file2" for file in... (4 Replies)
Discussion started by: am24
4 Replies

3. UNIX for Dummies Questions & Answers

Stack data from multiple files into one, with variable column files

Hello Gurus, Im new to scripting. Got struck with a file merge issue in Unix. Was looking for some direction and stumbled upon this site. I saw many great posts and replies but couldnt find a solution to my issue. Greatly appreciate any help.. I have three csv files -> Apex_10_Latest.csv,... (1 Reply)
Discussion started by: wamshi
1 Replies

4. Shell Programming and Scripting

Select multiple column from multiple files

Hi Friends, $ cat test1.txt emeka:1438 shelley:1439 dmeyer:1440 kurtarn:1441 abdul:1442 $ cat test2.txt 1:a 2:b 3:c 4:d $ cat test3.txt cat:dog:bat man:hot:cold (5 Replies)
Discussion started by: Jewel
5 Replies

5. Shell Programming and Scripting

Run one script on multiple files and print out multiple files.

How can I Run one script on multiple files and print out multiple files. FOR EXAMPLE i want to run script.pl on 100 files named 1.txt ....100.txt under same directory and print out corresponding file 1.gff ....100.gff.THANKS (4 Replies)
Discussion started by: grace_shen
4 Replies

6. UNIX for Dummies Questions & Answers

Run one script on multiple files and print out multiple files.

How can I run the following command on multiple files and print out the corresponding multiple files. perl script.pl genome.gff 1.txt > 1.gff However, there are multiples files of 1.txt, from 1----100.txt Thank you so much. No duplicate posting! Continue here. (0 Replies)
Discussion started by: grace_shen
0 Replies

7. Shell Programming and Scripting

awk, multiple files input and multiple files output

Hi! I'm new in awk and I need some help. I have a folder with a lot of files and I need that awk do something in each file and print a new file with the output. The input file name should be modified when I print the outpu files. Thanks in advance for help! :-) ciao (5 Replies)
Discussion started by: gabrysfe
5 Replies

8. Shell Programming and Scripting

combine multiple files by column into one files already sorted!

I have multiple files; each file contains a certain data in a column view simply i want to combine all those files into one file in columns example file1: a b c d file 2: 1 2 3 4 file 3: G (4 Replies)
Discussion started by: ahmedamro
4 Replies

9. UNIX for Dummies Questions & Answers

Using AWK: Extract data from multiple files and output to multiple new files

Hi, I'd like to process multiple files. For example: file1.txt file2.txt file3.txt Each file contains several lines of data. I want to extract a piece of data and output it to a new file. file1.txt ----> newfile1.txt file2.txt ----> newfile2.txt file3.txt ----> newfile3.txt Here is... (3 Replies)
Discussion started by: Liverpaul09
3 Replies

10. Shell Programming and Scripting

Extraction of data from multiple text files, and creation of a chart

Hello dear friends, My problem as explained below seems really basic. Fact is that I'm totally new to programming, and have only a week to produce a script ( CShell or Perl ? ) to perform this action. While searching on the forums, I found a command that could help me, but I don't know... (2 Replies)
Discussion started by: ackheron
2 Replies
Login or Register to Ask a Question