Visit The New, Modern Unix Linux Community


How to pass a field from awk in a pipe?


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting How to pass a field from awk in a pipe?
# 1  
How to pass a field from awk in a pipe?

Thanks in advance : )

I try for a long time searching for a way to split a large gzip csv file into many gzip files (except for the last sub-file which is to joint the next big file's children.) All the subfiles are to be named by the field.

But I only managed to split them into the uncompressed subfiles.

Code:
zcat bigfile | awk ‘{ print $0 >> $1 }’ FS=”,”

If I can pass $1 outside awk and if gz could append lines then it will save me a lot of i/o time and capacity. One of my gzip file is 124 gb.

I.e. something like

Code:
zcat file | awk ‘{ print $0 }’ FS=”,” |gzip >> "$PassOut1".gz

If you kwon what I mean.

Last edited by Scott; 08-17-2010 at 03:42 AM.. Reason: Please use code tags
# 2  
may be you can try something like this.

First write the uncompressed data in subfile1 and when you are about to strat the subfile2, run gzip for subfile1.
This User Gave Thanks to kumaran_5555 For This Post:
# 3  
I am new to this forum. Thanks Kumaran. I am not sure how to do that, i.e. testing the change of $1. Also the source data is not sorted by $1 so it's hard to tell tell if I get more of the same over the entire bigfile.
# 4  
we need sample data and criteria to split the file.(where and when to split the file)
# 5  
Code:
mkdir subfolder
zcat file | awk -F, '{print $0 > "subfolder/" $1}' 
gzip subfolder/*

This User Gave Thanks to rdcwayx For This Post:
# 6  
Thanks rdcwayx!! Thats neat, but the file is not gz until it's all split.

The data goes like this, with a lot of fields

Code:
VTLJ.J	2-Jan-96	08:07:07.310	2	Quote
VTLJ.J	2-Jan-96	09:30:00.320	2	Quote
BKLJ.J	2-Jan-96	10:56:38.660	2	Quote
LJ.C	3-Jan-96	09:23:34.070	2	Quote

If I can pass $1 outside awk and if gz could be appended line then it will save me a lot of i/o time and capacity. One of my gzip file is 124 gb.

I.e. something like

Code:
Zcat file | awk { print $0 }' FS=, |gzip >> $PassOut1

If you kwon what I mean.

That, is I like to do 1 pass of the data and split the bigfile into

File VTLI.J.gz
Code:
VTLJ.J	2-Jan-96	08:07:07.310	2	Quote
VTLJ.J	2-Jan-96	09:30:00.320	2	Quote

File BKLI.J.gz
Etc

Last edited by Scott; 08-17-2010 at 04:52 AM.. Reason: Code tags, please...
# 7  
First, I don't see "," in your input file.

Second, Do you mean the big file has been sessioned or sorted by column 1

For example, if column 1 is VTLJ.J, and after I see BKLJ.J, there should be no line with VTLJ.J.

You need confirm it first.

If it is not, you have to wait until awk go through the whole file, then gzip the output files.

Previous Thread | Next Thread
Thread Tools Search this Thread
Search this Thread:
Advanced Search

Test Your Knowledge in Computers #584
Difficulty: Easy
Binary files generally requires less space and are more efficient to process.
True or False?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Cat a script, pipe it, then pass arguments to it?

suppose i have a perl script that is normally run this way: ./checkdisk.pl -H hostname -w 40 -c 80 but, for whatever reason, i cannot run the script directly as it should. But i can cat it through pipe. How can i pass the arguments "-H hostname -w 40 -c 80"? so this is what i'm doing,... (6 Replies)
Discussion started by: SkySmart
6 Replies

2. Shell Programming and Scripting

Function in one-linef and pass arguments in a pipe

I need to declare a function, this function will contain a script, this script cannot be in a file but must be piped. and then, for the script to run, i need to pass arguments to it. everything has to be on one line. so i'm basically looking for a one-liner here's what i'm doing: myfunc ()... (3 Replies)
Discussion started by: SkySmart
3 Replies

3. Shell Programming and Scripting

awk to parse field and include the text of 1 pipe in field 4

I am trying to parse the input in awk to include the |gc= in $4 but am not able to. The below is close: awk so far: awk '{sub(/\|]+]++/, ""); print }' input.txt Input chr1 955543 955763 AGRN-6|pr=2|gc=75 0 + chr1 957571 957852 AGRN-7|pr=3|gc=61.2 0 + chr1 970621 ... (7 Replies)
Discussion started by: cmccabe
7 Replies

4. Shell Programming and Scripting

Pass the value from pipe

am trying to pass the date calculated to variable so that i can use the same for my further reporting and tracking of the changes as per the variable value. System = SunOS Release = 5.9 KernelID = Generic_122300-61 date '+%m %d %Y' | { read MONTH DAY YEAR DAY=`expr "$DAY" - 1` case... (5 Replies)
Discussion started by: pradeep84in
5 Replies

5. Shell Programming and Scripting

Pivoting csv field from pipe delimited file

Hello All, Thanks for taking time to read through the thread and for providing any possible solution. I am trying to pivot a comma separated field in a pipe delimited file. Data looks something like this: Field1|Field2 123|345,567,789 234|563,560 345|975,098,985,397,984 456|736 Desired... (8 Replies)
Discussion started by: svks1985
8 Replies

6. Shell Programming and Scripting

How to pass parameter to pipe command row-wise?

Hi, I have a list of parameter in a file. I want to pass them one by one to piped command and syntax is like <command> <parameter> <command continues> How to achieve that ? Thanks (1 Reply)
Discussion started by: ezee
1 Replies

7. Shell Programming and Scripting

Pass awk field to a command line executed within awk

Hi, I am trying to pass awk field to a command line executed within awk (need to convert a timestamp into formatted date). All my attempts failed this far. Here's an example. It works fine with timestamp hard-codded into the command echo "1381653229 something" |awk 'BEGIN{cmd="date -d... (4 Replies)
Discussion started by: tuxer
4 Replies

8. UNIX for Dummies Questions & Answers

add new 'date field' in a pipe delimited file

i need to add a new field in a pipe delimited line. the field will be the current date today. aa|a|s|w|1 as|oiy|oiy|oiy|2 given that all lines are uniformed in the number of fields i want it to look like this:\ aa|a|s|w|1|20120126 as|oiy|oiy|oiy|2|20120126 please help :) (3 Replies)
Discussion started by: kokoro
3 Replies

9. Shell Programming and Scripting

AWK: Pattern match between 2 files, then compare a field in file1 as > or < field in file2

First, thanks for the help in previous posts... couldn't have gotten where I am now without it! So here is what I have, I use AWK to match $1 and $2 as 1 string in file1 to $1 and $2 as 1 string in file2. Now I'm wondering if I can extend this AWK command to incorporate the following: If $1... (4 Replies)
Discussion started by: right_coaster
4 Replies

10. UNIX for Dummies Questions & Answers

Replacing a field in pipe delimited TEXT File

Hi, I want to replace a field in a text delimited file with the actual number of records in the same file. HDR|ABCD|10-13-2008 to 10-19-2008.txt|10-19-2008|XYZ DTL|0|5464-1|0|02-02-2008|02-03-2008||||F||||||||| DTL|1|5464-1|1|02-02-2008|02-03-2008|1||JJJ... (3 Replies)
Discussion started by: ravi0435
3 Replies

Featured Tech Videos