Login or Register to Ask a Question and Join Our Community


UNIX for Dummies Questions & Answers

Use awk to pipe output from one file into multiple files

 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Use awk to pipe output from one file into multiple files
# 1  
Old 05-09-2012
Use awk to pipe output from one file into multiple files
Hi All. Thanks for your help in advance.

I have a requirement to examine the number of delimiters in each record of a file. If the record has the expected number of delimiters it should be passed into a 'good' file. If it does not, the record should be passed into a 'bad' file. I have been able to complete this task, but it requires passing over the file twice. With bigger files this may slow down performance. I currently am using:
Code:
cat ${landing_dir}/${lfile}|awk -F "${file_delimiter}" "NF !=${file_field_count} {print NR, \$0 }" >> $bad_file

The similar code is used for the 'good' file with slight modifications. Is there a way to do this in only one pass over the file?

When I try something along the following lines, the script gets hung.
Code:
cat ${landing_dir}/${lfile}|awk -F "${file_delimiter}" "NF !=${file_field_count} {print NR, \$0 }" >> $bad_file
awk -F "${file_delimiter}" "NF ==${file_field_count} {print}" > ${rootDir}/landing/Validation_EDITED/new_${lfile}

Thanks, again!

Moderator's Comments:
Mod Comment Please use next time code tags for your code and data
Last edited by vbe; 05-09-2012 at 01:42 PM..
# 2  
Old 05-09-2012
Code:
awk -F<delim> '{if(NF!=field_count) print NR,$0 > "badfile"; else print NR,$0 > "goodfile"}'

# 3  
Old 05-09-2012
Thanks shamrock... I am still having an issue

I Changed code to
PHP Code:
awk -F ${file_delimiter} '{if(NF!=${file_field_count}) print NR, $0 > $bad_file; else
                                        print > ${rootDir}/landing/Validation_EDITED/new_${lfile}}' ${lfile} 
the error is
Code:
awk: RESPONSE_20120502094600.txt
awk:                                 ^ syntax error

# 4  
Old 05-09-2012
You need to convert shell variables into awk variables with the "-v" switch before doing this operation...
Code:
awk -F ${file_delimiter} -vbad=$bad_file -vgood=${rootDir}/landing/Validation_EDITED/new_${lfile} -vfc=${file_field_count} '{if(NF!=fc) print NR,$0 > bad; else print > good}' ${lfile}

This User Gave Thanks to shamrock For This Post:
codestar1
# 5  
Old 05-09-2012
${lfile} is the file being parsed. Do I need to pass that as a variable as well?

I keep getting this error:

Code:
awk: (FILENAME=RESPONSE_2012.txt FNR=1) fatal: expression for `>' redirection has null string value

---------- Post updated at 04:18 PM ---------- Previous update was at 03:07 PM ----------

Shamrock, thank you!! you provided the correct code. I discovered my variable that provided the delimiter type was coming as a null string. Once that was fixed, it worked fine.

Thanks Again!
# 6  
Old 05-09-2012
Depending on the flavor of the awk, two passes may be more efficient. The original awk would flush output (among other extra work) for each record if you do redirection inside awk, making it slower if you go with the one pass approach.
# 7  
Old 05-10-2012
cat ${landing_dir}/${lfile}|awk -F "${file_delimiter}" -v bad=${bad_file} -v good=${rootDir}/landing/Validation_EDITED/new_${lfile} -v fc=${file_field_count} '{if(NF!=fc) {print NR,$0 > bad;} else {{print > good}}}' ${lfile}

binlib, I ended up going this route. Not sure what you mean by flushing out the extra work. Could you please explain a little more?

Thanks for your input
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Pipe or combine output of three awk commands

What is the correct syntax to pipe or run three awk commands? Basically, using the output of the first awk as input in the second. Then using the output of the second awk in the third. Thank you :). awk 'FNR==NR {E; next }$3 in E {print $3, $5}' panel_genes.txt RefSeqGene.txt > update.txt |... (3 Replies)
Discussion started by: cmccabe
3 Replies

2. Shell Programming and Scripting

Awk: Print Error While Redirecting output in multiple Files

Hi, I have a following code in which I am unable to redirect to multiple files. Can anybody please help with some corrections awk -F, '{ if ( substr($1,26,2)=="02" && substr($1,184,14)=="MTSCC_VALFIRST") { array1++ array2++ array3++ } else if (substr($1,26,2)=="03" &&... (4 Replies)
Discussion started by: siramitsharma
4 Replies

3. Shell Programming and Scripting

Pipe awk's output to sed for deletion

Hi Friends, I am using a command that prints certain lines from a file. For ex: cat input abc chr1 456 def chr1 789 ghi chr1 999 jjj chr1 777 jhk chr7 914 My command awk '{if($2=="chr1" && $3>=456 && $3<=999) {print $0}}' OFS="\t" input Output being printed is abc chr1 456 (7 Replies)
Discussion started by: jacobs.smith
7 Replies

4. UNIX for Dummies Questions & Answers

Using multiple pipe output

I have a script that finds all sffs and extracts them into .fastq file types. What I need to do is change the .fastq to .fasta using the below script. How can I change the input.fastq and output.fasta to mirror the file's name? Would I use an array and use the default iterator? #!/bin/bash ... (3 Replies)
Discussion started by: jrymer
3 Replies

5. Shell Programming and Scripting

awk, multiple files input and multiple files output

Hi! I'm new in awk and I need some help. I have a folder with a lot of files and I need that awk do something in each file and print a new file with the output. The input file name should be modified when I print the outpu files. Thanks in advance for help! :-) ciao (5 Replies)
Discussion started by: gabrysfe
5 Replies

6. UNIX for Dummies Questions & Answers

awk to match multiple regex and create separate output files

Howdy Folks, I have a list that looks like this: (file2.txt) AAA BBB CCC DDD and there are 24 of these short words. I am matching these patterns to another file with 755795 lines (file1.txt). I have this code for matching: awk -v f2=file2.txt ' BEGIN { while(... (2 Replies)
Discussion started by: heecha
2 Replies

7. UNIX for Dummies Questions & Answers

Using AWK: Extract data from multiple files and output to multiple new files

Hi, I'd like to process multiple files. For example: file1.txt file2.txt file3.txt Each file contains several lines of data. I want to extract a piece of data and output it to a new file. file1.txt ----> newfile1.txt file2.txt ----> newfile2.txt file3.txt ----> newfile3.txt Here is... (3 Replies)
Discussion started by: Liverpaul09
3 Replies

8. Shell Programming and Scripting

pipe'ing grep output to awk

This script is supposed to find out if tomcat is running or not. #!/bin/sh if netstat -a | grep `grep ${1}: /tomcat/bases | awk -F: '{print $3}'` > /dev/null then echo Tomcat for $1 running else echo Tomcat for $1 NOT running fi the /tomcat/bases is a file that... (2 Replies)
Discussion started by: ziggy25
2 Replies

9. UNIX for Dummies Questions & Answers

single output of awk script processing multiple files

Helllo UNIX Forum :) Since I am posting on this board, yes, I am new to UNIX! I read a copy of "UNIX made easy" from 1990, which felt like a making a "computer-science time jump" backwards ;) So, basically I have some sort of understanding what the basic concept is. Problem Description:... (6 Replies)
Discussion started by: Kasimir
6 Replies

10. UNIX for Dummies Questions & Answers

pipe output to two files

I am using grep and I want the output to go into two files without going to the screen. I used tee to get the output into two files, but it is also putting the output on the screen which i do not want. Can this be fixed. (2 Replies)
Discussion started by: NobluesFDT
2 Replies
Login or Register to Ask a Question