How to pass a field from awk in a pipe?

Login or Register to Ask a Question and Join Our Community

How to pass a field from awk in a pipe?

Tags

Top Forums Shell Programming and Scripting How to pass a field from awk in a pipe?

Prev Next

08-17-2010

Registered User

5, 0

Join Date: Aug 2010

Last Activity: 17 August 2010, 9:00 AM EDT

Posts: 5

Thanks Given: 3

Thanked 0 Times in 0 Posts

How to pass a field from awk in a pipe?

Thanks in advance : )

I try for a long time searching for a way to split a large gzip csv file into many gzip files (except for the last sub-file which is to joint the next big file's children.) All the subfiles are to be named by the field.

But I only managed to split them into the uncompressed subfiles.

Code:

zcat bigfile | awk ‘{ print $0 >> $1 }’ FS=”,”

If I can pass $1 outside awk and if gz could append lines then it will save me a lot of i/o time and capacity. One of my gzip file is 124 gb.

I.e. something like

Code:

zcat file | awk ‘{ print $0 }’ FS=”,” |gzip >> "$PassOut1".gz

If you kwon what I mean.

Last edited by Scott; 08-17-2010 at 03:42 AM.. Reason: Please use code tags

Kingsley

View Public Profile for Kingsley

Find all posts by Kingsley

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Cat a script, pipe it, then pass arguments to it?

suppose i have a perl script that is normally run this way: ./checkdisk.pl -H hostname -w 40 -c 80 but, for whatever reason, i cannot run the script directly as it should. But i can cat it through pipe. How can i pass the arguments "-H hostname -w 40 -c 80"? so this is what i'm doing,...

2. Shell Programming and Scripting

Function in one-linef and pass arguments in a pipe

I need to declare a function, this function will contain a script, this script cannot be in a file but must be piped. and then, for the script to run, i need to pass arguments to it. everything has to be on one line. so i'm basically looking for a one-liner here's what i'm doing: myfunc ()...

3. Shell Programming and Scripting

awk to parse field and include the text of 1 pipe in field 4

I am trying to parse the input in awk to include the |gc= in $4 but am not able to. The below is close: awk so far: awk '{sub(/\|]+]++/, ""); print }' input.txt Input chr1 955543 955763 AGRN-6|pr=2|gc=75 0 + chr1 957571 957852 AGRN-7|pr=3|gc=61.2 0 + chr1 970621 ...

4. Shell Programming and Scripting

Pass the value from pipe

am trying to pass the date calculated to variable so that i can use the same for my further reporting and tracking of the changes as per the variable value. System = SunOS Release = 5.9 KernelID = Generic_122300-61 date '+%m %d %Y' | { read MONTH DAY YEAR DAY=`expr "$DAY" - 1` case...

5. Shell Programming and Scripting

Pivoting csv field from pipe delimited file

Hello All, Thanks for taking time to read through the thread and for providing any possible solution. I am trying to pivot a comma separated field in a pipe delimited file. Data looks something like this: Field1|Field2 123|345,567,789 234|563,560 345|975,098,985,397,984 456|736 Desired...

6. Shell Programming and Scripting

How to pass parameter to pipe command row-wise?

Hi, I have a list of parameter in a file. I want to pass them one by one to piped command and syntax is like <command> <parameter> <command continues> How to achieve that ? Thanks

7. Shell Programming and Scripting

Pass awk field to a command line executed within awk

Hi, I am trying to pass awk field to a command line executed within awk (need to convert a timestamp into formatted date). All my attempts failed this far. Here's an example. It works fine with timestamp hard-codded into the command echo "1381653229 something" |awk 'BEGIN{cmd="date -d...

8. UNIX for Dummies Questions & Answers

add new 'date field' in a pipe delimited file

i need to add a new field in a pipe delimited line. the field will be the current date today. aa|a|s|w|1 as|oiy|oiy|oiy|2 given that all lines are uniformed in the number of fields i want it to look like this:\ aa|a|s|w|1|20120126 as|oiy|oiy|oiy|2|20120126 please help :)

9. Shell Programming and Scripting

AWK: Pattern match between 2 files, then compare a field in file1 as > or < field in file2

First, thanks for the help in previous posts... couldn't have gotten where I am now without it! So here is what I have, I use AWK to match $1 and $2 as 1 string in file1 to $1 and $2 as 1 string in file2. Now I'm wondering if I can extend this AWK command to incorporate the following: If $1...

10. UNIX for Dummies Questions & Answers

Replacing a field in pipe delimited TEXT File

Hi, I want to replace a field in a text delimited file with the actual number of records in the same file. HDR|ABCD|10-13-2008 to 10-19-2008.txt|10-19-2008|XYZ DTL|0|5464-1|0|02-02-2008|02-03-2008||||F||||||||| DTL|1|5464-1|1|02-02-2008|02-03-2008|1||JJJ...

Login or Register to Ask a Question