Shell Programming and Scripting

Hi All,

I'd like to create a specific output filename for AWK.

The file I am processing with AWK looks like:

Code:

output_081012.csv*
27*TEXT*1.0*2.0*3.0

where * is my delimeter and the first line of the file is the output filename i'd like to create

is there a way to assign an awk variable to the first line and then use that variable in the printf command to create the output file?

for instance

Code:

awk -f inputfile
BEGIN
{
FS='*'
if (FNR==1)
outputfile=$1
}
{
if {FNR==2}
printf("%s,%s,%s,%s,%s\n",$1,$2,$3,$4,$5) >>outputfile
}

Thanks!

Moderator's Comments:

Please view this code tag video for how to use code tags when posting code and data.

Well, you don't put that in the BEGIN -- that runs before any files are processed, not during. You can use OFS to simplify your printf into a print, too.

Code:

awk -F"*" -v OFS="," 'NR==1{F=$1; next} { print $1,$2,$3,$4,$5>F }' input

I think Corona688's keyboard is dropping characters today... I think he meant:

Code:

awk -v FS="*" -v OFS="," 'NR==1{F=$1; next} { print $1,$2,$3,$4,$5>F }' input

I noticed that you attempt at the code used FNR==1 instead of NR==1. If you intended to process multiple input files in a single call to awk and to have awk append to a different output file based on the first line of each input file, I think you want something like:

Code:

awk -v FS="*" -v OFS="," 'FNR==1 {
	if (F != "") close(F);
	F=$1
	next
}
	{ print $1,$2,$3,$4,$5>>F}' input_file1 input_file2 input file3 ...

Note also that you don't need the "*" at the end of the first line in your input file. (It doesn't hurt to have it, it just isn't needed for the script to work.)

-F"*" is perfectly valid. It's short-form for -v OFS="*", not to mention probably older.

Not that I mind you catching my other typos.

Another funny awk thing you might see sometimes is awk '{print $1}' VARNAME="asdf" filename which looks weird but is also a perfectly good way of setting a variable inside awk, and probably older than -v. Just remember that they're parsed the same time as filenames -- i.e. they won't be parsed before a BEGIN {} block. -v VAR=whatever, on the other hand, gets parsed before BEGIN {}.

I apologize, -F ERE is a synonym for -v FS=ERE (and it is documented in all of the man page including the POSIX/UNIX standards). I assume you had a typo above and meant FS rather than OFS. When I copied your solution into a file and tried it out, it failed; I must have screwed up something in the cut and paste.

Yes, I know that variables can also be set after the awk program on the command line. In fact you can intermix variable assignment operands and pathname operands. Variable assignments that appear here are processed after any commands specified the the awk program's BEGIN block and before any following file operands are read by the program. So you could have a command line like:

Code:

awk ' {$(NF+1)=F;print}' F=file1 file1 F=file2 file2

to cat files with the filename appended to each line in the file. This is documented in the POSIX standard but isn't mentioned on many vendor man pages.

I left out some details but I basically have a bunch of *.csv's that I am trying to collect together into one file. The format of each *.csv matches what I posted earlier, where the filename is the first record and the second line is the data. Is there a good way to add a header row at the top of the output file? For some reason I don't believe my shell is working the way it is supposed to, so I am resorting to calling awk once to create the output file with the header row and then on the second call to populate it. Either way, thanks for your help!

---------- Post updated at 04:58 PM ---------- Previous update was at 04:47 PM ----------

My awk script looks like:
BEGIN{
RS="\n"
FS="*"
OFS=","
ST1="Channel Number"
ST2="Channel Label"
ST3="Time at Max"
ST4="Time History Max"
ST5="Time at Min"
ST6="Time History Min"
ST7="Frequency at Max Response"
ST8="Max Response"
}
{
if (FNR==1)
outputfile=$1
print ST1 ST2 ST3 ST4 ST5 ST6 ST7 ST8 >outputfile
if (FNR==2)
print $1 $2 $3 $4 $5 $6 $7 $8 >>outputfile
}

I thought this would work but it doesn't

You're close. You have a few problems:

First, the expressions passed to print need to be separated by a comma.

Second, you print the headerline to outputfile twice (because you're missing a { } pair around the commands you want to run when FNR is 1.

Third, you aren't closing any of the output files you're opening. With a small number of files, it won't matter since all open files will be closed when you get to the end. But if you have a large number of files, you may run out of file descriptors.

The default value for RS is a <newline>, so you don't need to set it.

I've made a couple of other slight changes and reformatted to make it easier to read, but this is VERY similar to what you did:

Code:

BEGIN{
    FS="*"
    OFS=","
    ST1="Channel Number"
    ST2="Channel Label"
    ST3="Time at Max"
    ST4="Time History Max"
    ST5="Time at Min"
    ST6="Time History Min"
    ST7="Frequency at Max Response"
    ST8="Max Response"
}

FNR==1 {
    if (output file!="") close(outputfile)
    outputfile=$1
    print ST1,ST2,ST3,ST4,ST5,ST6,ST7,ST8 >outputfile
}
FNR==2 {
    print $1,$2,$3,$4,$5,$6,$7,$8 >>outputfile
}