Passing multiple files to awk


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Passing multiple files to awk
# 1  
Old 07-05-2012
Passing multiple files to awk

Hi all,

I have a load of files in the format e.g.

a_1.out
a_300.out
a_20.out

etc

I would like to numeric sort them in ascending order by the number in the file name,
then pass them into awk for manipulation. How do I do this?
# 2  
Old 07-05-2012
Code:
ls a_*.out > file_list
awk -F"[_.]" '{print $2" "$0}' file_list|sort -n|awk '{print $2}' > temp
mv temp > file_list
while read f
do
 awk <your-program> "$f"
done < file_list


Last edited by elixir_sinari; 07-06-2012 at 03:50 AM..
This User Gave Thanks to elixir_sinari For This Post:
# 3  
Old 07-05-2012
Or a bit shorter:
Code:
ls a_*.out | sort -t_ -k2n |
while read f
do 
  awk... "$f"
done

This User Gave Thanks to Scrutinizer For This Post:
# 4  
Old 07-06-2012
Thanks for your response.

The only problem is the above solutions don't work.

My awk program needs the list of files to operate on all in the one go, not passed in one by one. So I put the list of files all on the one line in a file and read them in from that file using a while loop. The list must have been too long because it didn't work.

FYI my awk program is

Code:
$ awk '{arr[$1]=arr[$1] " " $2}END{for(i in arr)print i,arr[i]}' file1 file2 etc


Last edited by radoulov; 07-06-2012 at 09:13 AM..
# 5  
Old 07-06-2012
Quick solution, you could try:
Code:
ls a_*.out | sort -t_ -k2n |
while read f
do 
  cat "$f"
done |
awk '{arr[$1]=arr[$1] " " $2}END{for(i in arr)print i,arr[i]}'

--
Looking at your awk script, if it uses too much memory you could perform a sort on the input and then use a simpler script..

Last edited by Scrutinizer; 07-06-2012 at 04:36 PM..
This User Gave Thanks to Scrutinizer For This Post:
# 6  
Old 07-07-2012
try this
Code:
awk '{arr[$1]=arr[$1] " " $2}END{for(i in arr)print i,arr[i]}' `ls a_*.out | sort -t_ -k2n`


Last edited by Scrutinizer; 07-07-2012 at 05:03 AM..
# 7  
Old 07-07-2012
@raj, that will work if there are not too many files AND if those files do not contain special characters and/or spaces....
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Passing multiple files to awk for processing in bash script

Hi, I'm using awk command in bash script. I'm able to pass multiple files to awk for processing.The code i can use is as below(sample code) #!/bin/bash awk -F "," 'BEGIN { ... ... ... }' file1 file2 file3 In the above code i'm passing the file names manually and it is fine till my... (7 Replies)
Discussion started by: shree11
7 Replies

2. Shell Programming and Scripting

Passing multiple variable to awk

Hi , can I pass more then one variable to awk using -v option? (4 Replies)
Discussion started by: Anupam_Halder
4 Replies

3. Shell Programming and Scripting

ksh passing to awk multiple dyanamic variables awk -v

Using ksh to call a function which has awk script embedded. It parses a long two element list file, filled with text numbers (I want column 2, beginning no sooner than line 45, that's the only known thing) . It's unknown where to start or end the data collection, dynamic variables will be used. ... (1 Reply)
Discussion started by: highnthemnts
1 Replies

4. Shell Programming and Scripting

awk, multiple files input and multiple files output

Hi! I'm new in awk and I need some help. I have a folder with a lot of files and I need that awk do something in each file and print a new file with the output. The input file name should be modified when I print the outpu files. Thanks in advance for help! :-) ciao (5 Replies)
Discussion started by: gabrysfe
5 Replies

5. Shell Programming and Scripting

perform 3 awk commands to multiple files in multiple directories

Hi, I have a directory /home/datasets/ which contains a bunch (720) of subdirectories called hour_1/ hour_2/ etc..etc.. in each of these there is a single text file called (hour_1.txt in hour_1/ , hour_2.txt for hour_2/ etc..etc..) and i would like to do some text processing in them. Each of... (20 Replies)
Discussion started by: amarn
20 Replies

6. UNIX for Dummies Questions & Answers

Using AWK: Extract data from multiple files and output to multiple new files

Hi, I'd like to process multiple files. For example: file1.txt file2.txt file3.txt Each file contains several lines of data. I want to extract a piece of data and output it to a new file. file1.txt ----> newfile1.txt file2.txt ----> newfile2.txt file3.txt ----> newfile3.txt Here is... (3 Replies)
Discussion started by: Liverpaul09
3 Replies

7. UNIX for Dummies Questions & Answers

best method of replacing multiple strings in multiple files - sed or awk? most simple preferred :)

Hi guys, say I have a few files in a directory (58 text files or somthing) each one contains mulitple strings that I wish to replace with other strings so in these 58 files I'm looking for say the following strings: JAM (replace with BUTTER) BREAD (replace with CRACKER) SCOOP (replace... (19 Replies)
Discussion started by: rich@ardz
19 Replies

8. Shell Programming and Scripting

Passing two files to awk

I want to pass the first file as input argument using -v then first get awk to go through mref then pert.txt awk -v mref="reference.txt" -f ../../A-Scripts/convert-zv-zp.awk pert.txt BEGIN { Version = "V04" if ( !mref ) { usage(Version) print "ERROR: mref not... (1 Reply)
Discussion started by: kristinu
1 Replies

9. Shell Programming and Scripting

extract multiple cloumns from multiple files; skip rows and include filenames; awk

Hello, I am trying to write a bash shell script that does the following: 1.Finds all *.txt files within my directory of interest 2. reads each of the files (25 files) one by one (tab-delimited format and have the same data format) 3. skips the first 10 rows of the file 4. extracts and... (4 Replies)
Discussion started by: manishabh
4 Replies

10. Shell Programming and Scripting

passing multiple files as parameters

hi all i am etl guy i have shell script that i use to do processing on file. the problem is that i want it to use on multiple files at same time is there any way of passing the file name since my all my filename start with samename like abc* p,ease let me know (4 Replies)
Discussion started by: er_zeeshan05
4 Replies
Login or Register to Ask a Question