awk: too many output files created from while loop
I am using awk to read lines from a CSV file then put data into other files. These other files are named using the value of a certain column. Column 7 is a name such as "att" or "charter" . I want to end up with file names with the value of column 7 appended to them, like this:
I am getting these files alright, but I am ending up with a couple extra files as if column 7 was not populated.
stockton-migrate-1-.cfg
stockton-migrate-2-.cfg
The last part of the file name after the 3rd "-" comes from the $7 variable which I have as part of the output file name. I don't understand why my script is creating these files that end in "-.cfg" since $7 is always populated with data. Also, I instruct the script not to run through the while loop when $4 equals "d" and $7 equals "none".
Here is a snippet of my CSV file, notice that line 17 has "d" in $4 and "none" in $7.
Here is my code. Is there something I am doing wrong? I am getting the data I need but I am ending up with extra data that I don't want which makes me think there is a better way of doing this.
Code:
BEGIN {FS = ",";read_header=0}
# Read the header row that has the field keys
{if(read_header==1) {
#read values of variables to replace
for (i=1; i<=num_fields; i=i+1) value[i]=$i;
{if($4 == "d") {
output_file = "no routes.txt";
while (getline < "nodata.cfg" > 0) {
for (i=1; i<=num_fields; i=i+1) gsub(header[i],value[i]);
print $0 >> output_file;
}
close("nodata.cfg")
}
}
{if($4 != "d" && $7 != "none") {
output_file = "stockton-1-migrate-"$7".cfg";
output_file2 = "stockton-2-migrate-"$7".cfg";
while (getline < "mls-migration-template-s1.cfg" > 0) {
for (i=1; i<=num_fields; i=i+1) gsub(header[i],value[i]);
print $0 >> output_file;
}
close("mls-migration-template-s1.cfg")
while (getline < "mls-migration-template-s2.cfg" > 0) {
for (i=1; i<=num_fields; i=i+1) gsub(header[i],value[i]);
print $0 >> output_file2;
}
close("mls-migration-template-s2.cfg")
}
}
}
}
{if(read_header==0) {
#read the names of variables to replace
num_fields=NF;
for (i=1; i<=NF; i=i+1) {header[i]=$i}
read_header=1}}
Last edited by dodgerfan78; 10-08-2012 at 09:59 PM..
Reason: modify title
I would have to say that somewhere in your data you have at least one record where field 4 is not a 'd' and field 7 is empty. What is produced when you run this across your data?
Code:
awk -F , '$4 != "d" && !$7 ' data-file
If it generates even one line, that is the cause of your problem.
Thanks. I ran that command as you suggested and it gave back no input. I am using a spreadsheet editor to make the CSV file so I am positive that there is data in column 7 for every row. It seems to me that my loop somewhere is clearing the value of column 7 then re-reading the dataset against the template.
This User Gave Thanks to dodgerfan78 For This Post:
Thanks for running the test -- too painful, if not impossible, to ask to see all of the data
I didn't see this before...
Code:
if($4 == "d") {
output_file = "no routes.txt";
while (getline < "nodata.cfg" > 0) {
for (i=1; i<=num_fields; i=i+1) gsub(header[i],value[i]);
print $0 >> output_file;
}
close("nodata.cfg")
next;
}
You've already processed the record, and using getline you'll read to the end of the file and when you drop into the next block you'll always match because $7 will be nil.
Adding a 'next' will cause awk to read the next record and start over.
Last edited by agama; 10-08-2012 at 11:14 PM..
Reason: correction -- clarification
You can also simplify the if statement in the next section:
Code:
if($4 != "d" && $7 != "none")
can become
Code:
if( $7 != "none" )
because you already processed a 'd' record above and skipped the rest of the processing. It's minor, but makes it more efficient and that could help if your input is huge.
while IFS= read -r line
do
# sV for version detection
nmap -T4 -Pn -v -sS "$line" > "text/$line"
done < <(grep '' $file)
Hi,
where line represents the IP.
I am using NMAP to do scanning.
How can I set to execute that command in the loop several concurrently at a time instead of one... (5 Replies)
Hi Expert,
I have a .sh script in my home/new_dir/script.sh
This script creates number of output files at home/new_dir/email, home/new_dir/logs dir.
I am running this script using crontab (owner root).
Now this output files are getting created with
rw-r----- 1 root root
So if i... (2 Replies)
Have built this script, the output is what I needed, but NR 6 is omitted. Why? Is it an error? I am using Gawk.
'{nr=$2;f = $1} END{for (i=1;i<=f;i++) if (nr != i) print i, nr }' input1.csv >output1.csvinput1.csv
1 9
3 5
4 1
7 6
8 5
10 6
output1.csv > with the missing line number 6. 6 is... (5 Replies)
Hi all,
after hours of playing around with this and scouring the web I decided to ask my fellow UNIX operators as I can't wrap my head around this.
First off,
I want to parse an input file with tabs (I could pull this off easily with different delimiters) but I was trying to make nicer... (2 Replies)
I need to get total number of hdisk not assigned to any VGs.
PDC # lspv |grep None |awk '{print $1}' |wc
131 131 1099
So, it shows 131 hdisks.
I need to look at the individual hdisk fget_config info like below:
PDC # fget_config -Av |grep hdisk230
hdisk230 dac1 229... (4 Replies)
When I run the following command in the shell it works fine. It prints a city name and then a path for a file.
~$ for i in `awk -F':' '{print $0}' /home/knoppix/Desktop/data/subs | grep -m 1 $ city | sed "s/:/ /"`
>do
>echo $i
>done
Now, when I place it in this shell script (sh) it prints... (6 Replies)
I'm trying to understand if it's possible to create a set of variables that are numbered based on another variable (using eval) in a loop, and then call on it before the loop ends.
As an example I've written a script called question (The fist command is to show what is the contents of the... (2 Replies)
I have a while loop which looks for files and then sets a variable to give me the record count of each file:
current_members=`wc -l ${DATA_DIR}/$MEMBERS_FILENAME | nawk '{ printf "%d\n", $0}'`
I am out putting the totals into a file:
echo $current_members >> ../data/out_total_members.dat
... (7 Replies)
10-08-2012
