Visit Our UNIX and Linux User Community


How to make AWK process an input file many many times?


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting How to make AWK process an input file many many times?
# 1  
Old 05-14-2010
How to make AWK process an input file many many times?

By "many many times" I mean the times the input file is to be processed is unknown beforehand, it will be known when awk finishes processing the input file for the first time.

So my question is: how to start over again from the first record of the input file when AWK finishes processing the last record(before reaching "END")?
# 2  
Old 05-14-2010
Can you give a practical example to describe for what this is needed?
Do you need a demon that processes a log file continously for example or is it just for theoretical purpose?
Else you could pipe a tail -f into awk.

A tad more description needed.
# 3  
Old 05-14-2010
OK.
Here's an example: I have a file that has 30 records. for the first time this file is processed, I will do "group by" for the records, it might remain only 10 records after that, and I will save the 10 records to an array. And then I want to process the original file(30 records) 10 times (once for each record that was saved in the first processing, each record is something like a parameter here.)
# 4  
Old 05-14-2010
Still somewhat abstract - to answer abstract you can just fill 1 array with your matching pattern lines and another array with all lines. In the END block you can walk through both arrays onto each other by as many elements that are in the 1st array.

An example input file with an example of expected output would help to help.
# 5  
Old 05-14-2010
Hi
Do you mean to say

Code:
awk '.......' file file file file

Instead of writing the file n number of times, you want a way to control it?

Guru.
# 6  
Old 05-14-2010
If you need to process the same data multiple times and the data is not that large (GBs), you'd better read it into an array (or multiple arrays) and process it in memory after reading the input.

If the data size does not fit into memory, you can manipulate the ARGV array (just like I showed you in one of your previous posts).
# 7  
Old 05-14-2010
Hi, Guru
I would take zaxxon's suggestion and read the file once and save the records to an array and process them as many times as needed.
or just put "ARGV[ARGC++] = ARGV[ARGC-1]"(thanks for radoulov's code) at the end of each processing.

---------- Post updated at 04:38 AM ---------- Previous update was at 04:35 AM ----------

Quote:
Originally Posted by radoulov
If you need to process the same data multiple times and the data is not that large (GBs), you'd better read it into an array (or multiple arrays) and process it in memory after reading the input.

If the data size does not fit into memory, you can manipulate the ARGV array (just like I showed you in one of your previous posts).
Smilie Thank you.

Previous Thread | Next Thread
Test Your Knowledge in Computers #105
Difficulty: Easy
The programming interface, file abstraction, built-in networking and persistent background processing are all features and capabilities supported by a Unix or Linux OS.
True or False?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to check same process running 10 times?

At run time Without knowing job name how to check the job running in specific user "ABCD" ,If the running job duplicate more then 10 then script it self send alert message to the users with the process ID name so that will kill the processed to avoid hung issue ,tried below script please check and... (15 Replies)
Discussion started by: Kalia
15 Replies

2. Shell Programming and Scripting

make loop from input file

Hi Guys, I have file A.txt PP97 PP66 PP87 PP66 PP47 PP57 PP44 PP20 PP66 PP16 PP13 PP51 PP68 PP70 PP75 PP30 (2 Replies)
Discussion started by: asavaliya
2 Replies

3. Shell Programming and Scripting

The scripts not able to make the file to size 0, every times it go back to its original size

#!/bin/sh ########################################################################################################## #This script is being used for AOK application for cleaning up the .out files and zip it under logs directory. # IBM # Created #For pdocap201/pdoca202 .out files for AOK #1.... (0 Replies)
Discussion started by: mridul10_crj
0 Replies

4. Shell Programming and Scripting

Reading a file several times with awk

Hi everyone, I was wondering if it's possible to read a file ("file2" in my example) more than once. In this example I want to print file2 entirely for each lines of file1: awk -F$'\t' '{ print $0 while ((getline < "file2") > 0) { print "\t"$0 } }' file1 It... (4 Replies)
Discussion started by: anthony.cros
4 Replies

5. Shell Programming and Scripting

How to read the input from a file in background process?

Hi I have a script to run some other scripts automatically. But while running the script it should take the input value from a text file instead of taking from the keyboard. Please find last two lines of the script below. Here ans should be taken from a text file ineerly without displaying this... (1 Reply)
Discussion started by: shirdi
1 Replies

6. Shell Programming and Scripting

How to make the parent process to wait for the child process

Hi All, I have two ksh script. 1st script calls the 2nd script and the second script calls an 'C' program. I want 1st script to wait until the 'C' program completes. I cant able to get the process id for the 'C' program (child process) to make the 1st script to wait for the second... (7 Replies)
Discussion started by: sennidurai
7 Replies

7. UNIX for Dummies Questions & Answers

make 1 new file using records of 5 input files

can you make me an awk script that will: read the 1st record from each input file, file1 file2 file3 file4 and file5 and write them to file6. then..... read the 2nd record from each input file, file1 file2 file3 file4 and file5 and append them to file6. then..... read the 3rd record... (1 Reply)
Discussion started by: kenneth.mcbride
1 Replies

8. Shell Programming and Scripting

Can process substitution be used as an input file to another program?

Hey, I was trying to figure out how to launch a program from the command line, and it works if you pass it a config file. I was thinking about writing a script to dynamically create the config file and pass it to the command using something like command substitution (so I don't actually have to... (3 Replies)
Discussion started by: bj0
3 Replies

9. Shell Programming and Scripting

how to start a process and make it sleep for 5 mins and then kill that process

how to start a process and make it sleep for 5 mins and then kill that process (6 Replies)
Discussion started by: shrao
6 Replies

10. Shell Programming and Scripting

How to make my command work at all times

hi all, This is a very basic question. I want to make the command work at all times. i'm working on Suse-Linux and "clear" command is used to clear the contents of screen. I want to use only "cls" instead of "clear" command. i tried alias cls=clear , but its working only for a temporary... (3 Replies)
Discussion started by: wxwidgets
3 Replies

Featured Tech Videos