awk: too many output files created from while loop

Login or Register for Dates, Times and to Reply

Thread Tools Search this Thread
Top Forums Shell Programming and Scripting awk: too many output files created from while loop
# 1  
awk: too many output files created from while loop

I am using awk to read lines from a CSV file then put data into other files. These other files are named using the value of a certain column. Column 7 is a name such as "att" or "charter" . I want to end up with file names with the value of column 7 appended to them, like this:


I am getting these files alright, but I am ending up with a couple extra files as if column 7 was not populated.


The last part of the file name after the 3rd "-" comes from the $7 variable which I have as part of the output file name. I don't understand why my script is creating these files that end in "-.cfg" since $7 is always populated with data. Also, I instruct the script not to run through the while loop when $4 equals "d" and $7 equals "none".

Here is a snippet of my CSV file, notice that line 17 has "d" in $4 and "none" in $7.

1,k,,,Keystone, RENO DOWNTOWN,charter1,,
2,k,,,Keystone, SPARKS NORTH,charter1,,
7,k,,,Keystone, WASHOE MED CENTER,att1,,
17,k,,d,Keystone, RENO BRISAS,none,,

Here is my code. Is there something I am doing wrong? I am getting the data I need but I am ending up with extra data that I don't want which makes me think there is a better way of doing this.

BEGIN {FS = ",";read_header=0}
# Read the header row that has the field keys
{if(read_header==1) { 
#read values of variables to replace
    for (i=1; i<=num_fields; i=i+1) value[i]=$i;

    {if($4 == "d") {
      output_file = "no routes.txt";
        while (getline < "nodata.cfg" > 0) {
        for (i=1; i<=num_fields; i=i+1) gsub(header[i],value[i]);
        print $0 >> output_file;

    {if($4 != "d" && $7 != "none") {
        output_file = "stockton-1-migrate-"$7".cfg";
        output_file2 = "stockton-2-migrate-"$7".cfg";
        while (getline < "mls-migration-template-s1.cfg" > 0) {
            for (i=1; i<=num_fields; i=i+1) gsub(header[i],value[i]);
            print $0 >> output_file;
        while (getline < "mls-migration-template-s2.cfg" > 0) {
            for (i=1; i<=num_fields; i=i+1) gsub(header[i],value[i]);
            print $0 >> output_file2;
{if(read_header==0) {
#read the names of variables to replace
    for (i=1; i<=NF; i=i+1) {header[i]=$i}

Last edited by dodgerfan78; 10-08-2012 at 10:59 PM.. Reason: modify title
# 2  
I would have to say that somewhere in your data you have at least one record where field 4 is not a 'd' and field 7 is empty. What is produced when you run this across your data?

awk -F , '$4 != "d" && !$7 '   data-file

If it generates even one line, that is the cause of your problem.
# 3  

Thanks. I ran that command as you suggested and it gave back no input. I am using a spreadsheet editor to make the CSV file so I am positive that there is data in column 7 for every row. It seems to me that my loop somewhere is clearing the value of column 7 then re-reading the dataset against the template.
This User Gave Thanks to dodgerfan78 For This Post:
# 4  
Thanks for running the test -- too painful, if not impossible, to ask to see all of the data Smilie

I didn't see this before...

   if($4 == "d") {
      output_file = "no routes.txt";
        while (getline < "nodata.cfg" > 0) {
        for (i=1; i<=num_fields; i=i+1) gsub(header[i],value[i]);
        print $0 >> output_file;

You've already processed the record, and using getline you'll read to the end of the file and when you drop into the next block you'll always match because $7 will be nil.

Adding a 'next' will cause awk to read the next record and start over.

Last edited by agama; 10-09-2012 at 12:14 AM.. Reason: correction -- clarification
# 5  
That works! thank you agama! Smilie I am just learning awk, so I will keep that "next" command in mind next time.

P.S. and thanks for the clarification, that is what i was suspecting but didn't understand how it worked
# 6  
You can also simplify the if statement in the next section:

if($4 != "d" && $7 != "none")

can become

if( $7 != "none" )

because you already processed a 'd' record above and skipped the rest of the processing. It's minor, but makes it more efficient and that could help if your input is huge.
Login or Register for Dates, Times and to Reply

Previous Thread | Next Thread
Thread Tools Search this Thread
Search this Thread:
Advanced Search

Test Your Knowledge in Computers #998
Difficulty: Medium
Hooksafe is a hypervisor-based light system that safeguards a computer's kernel from rootkit attacks.
True or False?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Output Multiple Files in the For Loop

while IFS= read -r line do # sV for version detection nmap -T4 -Pn -v -sS "$line" > "text/$line" done < <(grep '' $file) Hi, where line represents the IP. I am using NMAP to do scanning. How can I set to execute that command in the loop several concurrently at a time instead of one... (5 Replies)
Discussion started by: alvinoo
5 Replies

2. Shell Programming and Scripting

Print awk output in same line ,For loop

My code is something like below. #/bin/bash for i in `ps -ef | grep pmon | grep -v bash | grep -v grep | grep -v perl | grep -v asm | grep -v MGMT|awk '{print $1" "$8}'` do echo $i ORACLE_SID=`echo $line | awk '{print $2}'` USERNAME=`echo $line | awk '{print $1}'` done ============= But... (3 Replies)
Discussion started by: tapia
3 Replies

3. Shell Programming and Scripting

How to give full access to output files created by .sh script run via crontab?

Hi Expert, I have a .sh script in my home/new_dir/ This script creates number of output files at home/new_dir/email, home/new_dir/logs dir. I am running this script using crontab (owner root). Now this output files are getting created with rw-r----- 1 root root So if i... (2 Replies)
Discussion started by: Jeet1982
2 Replies

4. UNIX for Dummies Questions & Answers

Print each output of loop in new column using awk or shell

I have this output from a loop a11 1,2 3,4 5,6 7,8 12,8 5,4 3,6 a12 10,11 12,13 15,18 20,22 a13 ... (3 Replies)
Discussion started by: maryre89
3 Replies

5. Shell Programming and Scripting

awk output error while loop through array

Have built this script, the output is what I needed, but NR 6 is omitted. Why? Is it an error? I am using Gawk. '{nr=$2;f = $1} END{for (i=1;i<=f;i++) if (nr != i) print i, nr }' input1.csv >output1.csvinput1.csv 1 9 3 5 4 1 7 6 8 5 10 6 output1.csv > with the missing line number 6. 6 is... (5 Replies)
Discussion started by: sdf
5 Replies

6. Shell Programming and Scripting

unexpected output within a for loop using awk

Hi all, after hours of playing around with this and scouring the web I decided to ask my fellow UNIX operators as I can't wrap my head around this. First off, I want to parse an input file with tabs (I could pull this off easily with different delimiters) but I was trying to make nicer... (2 Replies)
Discussion started by: Keepcase
2 Replies

7. Shell Programming and Scripting

for loop returns more output with awk

I need to get total number of hdisk not assigned to any VGs. PDC # lspv |grep None |awk '{print $1}' |wc 131 131 1099 So, it shows 131 hdisks. I need to look at the individual hdisk fget_config info like below: PDC # fget_config -Av |grep hdisk230 hdisk230 dac1 229... (4 Replies)
Discussion started by: Daniel Gate
4 Replies

8. Shell Programming and Scripting

Trouble with a file path as awk output in for loop

When I run the following command in the shell it works fine. It prints a city name and then a path for a file. ~$ for i in `awk -F':' '{print $0}' /home/knoppix/Desktop/data/subs | grep -m 1 $ city | sed "s/:/ /"` >do >echo $i >done Now, when I place it in this shell script (sh) it prints... (6 Replies)
Discussion started by: afroCluster
6 Replies

9. Shell Programming and Scripting

Using variables created sequentially in a loop while still inside of the loop [bash]

I'm trying to understand if it's possible to create a set of variables that are numbered based on another variable (using eval) in a loop, and then call on it before the loop ends. As an example I've written a script called question (The fist command is to show what is the contents of the... (2 Replies)
Discussion started by: DeCoTwc
2 Replies

10. Shell Programming and Scripting

total output from a file created in a while loop

I have a while loop which looks for files and then sets a variable to give me the record count of each file: current_members=`wc -l ${DATA_DIR}/$MEMBERS_FILENAME | nawk '{ printf "%d\n", $0}'` I am out putting the totals into a file: echo $current_members >> ../data/out_total_members.dat ... (7 Replies)
Discussion started by: Pablo_beezo
7 Replies

Featured Tech Videos