Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

awk help

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting awk help
# 1  
Old 03-17-2011
awk help
i have a file like

Code:
 
col1, col2, col3, col4, col5, col6,col7,col8
a,b,c,d,e,1,t,11
b,e,,w,,1,,
c,r,c,d,e,1,t,11
d,f,c,g,w,1,e,22
e,f,c,d,e,1,w,33
f,e,f,r,t,1,t,22
g,h,t,d,e,2,s,22
h,t,,,e,2,f,
a,r,,,w,2,,
f,r,r,d,e,2,f,44
e,b,h,d,t,2,d,55
a,r,h,d,e,2,g,66
a,b,c,d,e,2,44
a,b,c,d,e,2,,
a,b,c,d,e,2,
a,b,c,d,e,2,55

I am concenered with specific cols -- (col6, col8) -- (1,11)

what i need to do is make out a new file out of this containing unique records of (col6,col8) excluding spaces in col8.

output of the above sample is:

Code:
 
1,11
1,22
1,33
2,44
2,55
2,66

this removes the duplicates, but how to ignore the blank of col8,

i tried this
Code:
 
nawk -F, '!a[$6,$8]++ {print $6 "," $8}' sample.dat

but it gives this output
Code:
 col6,col8
,
1,11
1,
1,22
1,33
2,22
2,
2,44
2,55
2,66

what is that extra , after first line and how can i ignore blanks of col8.
pls help

Moderator's Comments:
Mod Comment Please use a more descriptive subject title. "awk help" is not useful. Say what awk help you want.
Last edited by Scott; 03-17-2011 at 05:30 PM..
# 2  
Old 03-17-2011
Could this help you?
Code:
awk -F"," '$6!="" && $8!="" && !a[$6FS$8] {a[$6FS$8]++;print $6FS$8}' inputfile

OR
Code:
#!/usr/bin/perl

use strict;
my (@fields,%hash);

while (<DATA>) {
chomp;
@fields=split(/,/);
if ($fields[5] !="" && $fields[7]!="") {$hash{$fields[5].",".$fields[7]}++;}
}

print $_,"\n" foreach (sort(keys(%hash)));

__DATA__
a,b,c,d,e,1,t,11
b,e,,w,,1,,
c,r,c,d,e,1,t,11
d,f,c,g,w,1,e,22
e,f,c,d,e,1,w,33
f,e,f,r,t,1,t,22
g,h,t,d,e,2,s,22
h,t,,,e,2,f,
a,r,,,w,2,,
f,r,r,d,e,2,f,44
e,b,h,d,t,2,d,55
a,r,h,d,e,2,g,66
a,b,c,d,e,2,44
a,b,c,d,e,2,,
a,b,c,d,e,2,
a,b,c,d,e,2,55

Last edited by pravin27; 03-17-2011 at 04:47 AM..
This User Gave Thanks to pravin27 For This Post:
mukeshguliao
# 3  
Old 03-17-2011
Hi mukeshguliao,

Excluding blank in Col6 and Col8
Code:
awk -F"," '$6 && $8~/./{print $6","$8}'

If it is needed to exclude blank only in Col 8:
Code:
awk -F"," '$8~/./{print $6","$8}'

Regards
# 4  
Old 03-17-2011
yes this works for the sample i provided...
but cant figure out it is failing to skip the last col's blank record in my original file..

here is the chunk of file
Code:
ASSETCLASS|ASSETTYPE|SETTLEMENTDATE|HOLIDAYCENTERNAMES|VALUATIONBASIS|RIGHTVALUATIONBASIS|CONTRACTUALREUSERIGHTS|APPLICABLEPARTY|RATENAME|BPSPREAD|BPSPREADEFFECTIVE|MATURITYDATE|CURRENCY|COUNTRY|IDENTIFIERS|SECUCREDITRATING|ISSUERS|FROMTIME|TOTIME|VALUATIONPERC|DESCRIPTION|REHYPOTYPE|REHYPORIGHTS|EXTID|ORGID|ICI_ID
Government Bonds|Govt-United States|T1|New York City|98|0|1|P||||||||||0|1|98|BLACK DIAMOND DBL C88|Bilateral|Y|88|1449041|2530016690
Government Bonds|Govt-United States|T1|New York City|98|0|1|P||||||||||1|10|95|BLACK DIAMOND DBL C88|Bilateral|Y|88|1449041|2530016690
Government Bonds|Govt-United States|T1|New York City|98|0|1|P||||||||||5|10|90|BLACK DIAMOND DBL C88|Bilateral|Y|88|1449041|2530016690
Mortgage Backed Securities|MBS-US FNMA (FANNIE MAE)|T1|New York City|98|0|1|C||||||||||0|99|98|TMC MTGE FD C2047|Bilateral|Y|2047|6110230|2530084990
Government Bonds|Govt-United States|T1|New York City|98|0|1|C||||||||||0|1|98|BLACK DIAMOND DBL C88|Bilateral|Y|88|1449041|2530016690
Government Bonds|Govt-United States|T1|New York City|98|0|1|C||||||||||5|10|90|BLACK DIAMOND DBL C88|Bilateral|Y|88|1449041|2530016690
Government Bonds|Govt-United States|T1|New York City|98|0|1|C||||||||||1|10|95|BLACK DIAMOND DBL C88|Bilateral|Y|88|1449041|2530016690
Mortgage Backed Securities|MBS-US FNMA (FANNIE MAE)|T1|New York City|98|0|1|C||||||||||0|0|98|TMC MTGE FD C2047|Bilateral|Y|2047|6110230|2530084990
Mortgage Backed Securities|MBS-US FHLMC (FREDDIE MAC)|T1|New York City|98|0|1|C||||||||||0|99|98|TMC MTGE FD C2047|Bilateral|Y|2047|6110230|2530084990
Government Bonds|Govt-France|T2|Target|98|0|1|C||||||||||1|10|98|NORDEA BK DENMARK C2044|Bilateral|Y|2044|640|ENORBKNRX
Agency Bonds|Agency-US FHLB (FED HOME LOAN BK)|T1|New York City|98|0|1|C||||||||||1|5|97|BR (NRRIT) C16240|Bilateral|Y|16240|6591472|2530028790
Agency Bonds|Agency-US FHLMC (FREDDIE MAC)|T1|New York City|98|0|1|P||||||||||0|1|95|SOWOOD ALPHA LP C5326|Bilateral|Y|5326|6599026|2530081290
Government Bonds|Govt-France|T2|Target|98|0|1|C||||||||||0|1|98|PENSFORMAGOGPSYK C5374|Bilateral|Y|5374|6541495|C265909
Government Bonds|Govt-France|T2|Target|98|0|1|C||||||||||1|10|95|PENSFORMAGOGPSYK C5374|Bilateral|Y|5374|6541495|C265909
Government Bonds|Govt-France|T2|Target|98|0|1|C||||||||||10|0|90|PENSFORMAGOGPSYK C5374|Bilateral|Y|5374|6541495|C265909
Government Bonds|Govt-Great Britain|T2|London|98|0|1|P||||||||||10|0|95|WELL37N9 C45423|Bilateral|Y|45423|7100736|
Government Bonds|Govt-Great Britain|T2|London|98|0|1|P||||||||||5|10|97|WELL37N9 C45423|Bilateral|Y|45423|7100736|
Government Bonds|Govt-France|T2|Target|98|0|1|C||||||||||0|1|100|WELL37N9 C45423|Bilateral|Y|45423|7100736|

and here is what i am executing as per you:
Code:
 
nawk -F"|"  '$24!="" && $26!="" && !a[$24FS$26] {a[$24FS$26]++;print $24FS$26}' file.dat

output i am getting is very close but blanks are not ignored!!? Smilie
Code:
 
EXTID|ICI_ID
88|2530016690
2047|2530084990
2044|ENORBKNRX
16240|2530028790
5326|2530081290
5374|C265909
45423|

Last edited by Scott; 03-17-2011 at 05:31 PM.. Reason: Mode code tags
# 5  
Old 03-17-2011
Try in general with:
Code:
awk -F"|" '$NF~/./{print $(NF-2)","$NF}' inputfile
EX TID,ICI_ID
88,2530016690
88,2530016690
88,2530016690
2047,2530084990
88,2530016690
88,2530016690
88,2530016690
2047,2530084990
2047,2530084990
2044,ENORBKNRX
16240,2530028790
5326,2530081290
5374,C265909
5374,C265909
5374,C265909

Or more precisely using fixed columns numbers:
Code:
awk -F"|" '$26~/./{print $24","$26}' inputfile
EX TID,ICI_ID
88,2530016690
88,2530016690
88,2530016690
2047,2530084990
88,2530016690
88,2530016690
88,2530016690
2047,2530084990
2047,2530084990
2044,ENORBKNRX
16240,2530028790
5326,2530081290
5374,C265909
5374,C265909
5374,C265909

Regards
This User Gave Thanks to cgkmal For This Post:
mukeshguliao
# 6  
Old 03-17-2011
I think you have space in the last column.
try this,
Code:
awk -F"|"  '$24 ~/[[:alnum:]]/ && $26~/[[:alnum:]]/ && !a[$24FS$26] {a[$24FS$26]++;print $24FS$26}' file.dat

# 7  
Old 03-17-2011
@pravin
there is no space though.. its just delimited with |. New line after that..

this doesnt return anything at all Smilie
Code:
awk -F"|"  '$24 ~/[[:alnum:]]/ && $26~/[[:alnum:]]/ && !a[$24FS$26] {a[$24FS$26]++;print $24FS$26}' file.dat

@cgkmal
i think this should work, but
still returns lines with blanks of $26

getting output
Code:
awk -F"|" '$26~/./{print $24","$26}' OTCeg.dat
EXTID,ICI_ID
88,2530016690
88,2530016690
88,2530016690
2047,2530084990
88,2530016690
88,2530016690
88,2530016690
2047,2530084990
2047,2530084990
2044,ENORBKNRX
16240,2530028790
5326,2530081290
5374,C265909
5374,C265909
5374,C265909
45423,
45423,
45423,

Last edited by mukeshguliao; 03-17-2011 at 07:05 AM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk output yields error: awk:can't open job_name (Autosys)

Good evening, Im newbie at unix specially with awk From an scheduler program called Autosys i want to extract some data reading an inputfile that comprises jobs names, then formating the output to columns for example 1. This is the inputfile: $ more MapaRep.txt ds_extra_nikira_usuarios... (18 Replies)
Discussion started by: alexcol
18 Replies

2. Shell Programming and Scripting

Pass awk field to a command line executed within awk

Hi, I am trying to pass awk field to a command line executed within awk (need to convert a timestamp into formatted date). All my attempts failed this far. Here's an example. It works fine with timestamp hard-codded into the command echo "1381653229 something" |awk 'BEGIN{cmd="date -d... (4 Replies)
Discussion started by: tuxer
4 Replies

3. Shell Programming and Scripting

Passing awk variable argument to a script which is being called inside awk

consider the script below sh /opt/hqe/hqapi1-client-5.0.0/bin/hqapi.sh alert list --host=localhost --port=7443 --user=hqadmin --password=hqadmin --secure=true >/tmp/alerts.xml awk -F'' '{for(i=1;i<=NF;i++){ if($i=="Alert id") { if(id!="") if(dt!=""){ cmd="sh someScript.sh... (2 Replies)
Discussion started by: vivek d r
2 Replies

4. Shell Programming and Scripting

HELP with AWK one-liner. Need to employ an If condition inside AWK to check for array variable ?

Hello experts, I'm stuck with this script for three days now. Here's what i need. I need to split a large delimited (,) file into 2 files based on the value present in the last field. Samp: Something.csv bca,adc,asdf,123,12C bca,adc,asdf,123,13C def,adc,asdf,123,12A I need this split... (6 Replies)
Discussion started by: shell_boy23
6 Replies

5. Shell Programming and Scripting

awk command to compare a file with set of files in a directory using 'awk'

Hi, I have a situation to compare one file, say file1.txt with a set of files in directory.The directory contains more than 100 files. To be more precise, the requirement is to compare the first field of file1.txt with the first field in all the files in the directory.The files in the... (10 Replies)
Discussion started by: anandek
10 Replies

6. Shell Programming and Scripting

Comparison and editing of files using awk.(And also a possible bug in awk for loop?)

I have two files which I would like to compare and then manipulate in a way. File1: pictures.txt 1.1 1.3 dance.txt 1.2 1.4 treehouse.txt 1.3 1.5 File2: pictures.txt 1.5 ref2313 1.4 ref2345 1.3 ref5432 1.2 ref4244 dance.txt 1.6 ref2342 1.5 ref2352 1.4 ref0695 1.3 ref5738 1.2... (1 Reply)
Discussion started by: linuxkid
1 Replies

7. Shell Programming and Scripting

Problem with awk awk: program limit exceeded: sprintf buffer size=1020

Hi I have many problems with a script. I have a script that formats a text file but always prints the same error when i try to execute it The code is that: { if (NF==17){ print $0 }else{ fields=NF; all=$0; while... (2 Replies)
Discussion started by: fate
2 Replies

8. Shell Programming and Scripting

awk: assign variable with -v didn't work in awk filter

I want to filter 2nd column = 2 using awk $ cat t 1 2 2 4 $ VAR=2 #variable worked in print $ cat t | awk -v ID=$VAR ' { print ID}' 2 2 # but variable didn't work in awk filter $ cat t | awk -v ID=$VAR '$2~/ID/ { print $0}' (2 Replies)
Discussion started by: honglus
2 Replies

9. Shell Programming and Scripting

scripting/awk help : awk sum output is not comming in regular format. Pls advise.

Hi Experts, I am adding a column of numbers with awk , however not getting correct output: # awk '{sum+=$1} END {print sum}' datafile 2.15291e+06 How can I getthe output like : 2152910 Thank you.. # awk '{sum+=$1} END {print sum}' datafile 2.15079e+06 (3 Replies)
Discussion started by: rveri
3 Replies

10. Shell Programming and Scripting

Awk problem: How to express the single quote(') by using awk print function

Actually I got a list of file end with *.txt I want to use the same command apply to all the *.txt Thus I try to find out the fastest way to write those same command in a script and then want to let them run automatics. For example: I got the file below: file1.txt file2.txt file3.txt... (4 Replies)
Discussion started by: patrick87
4 Replies
Login or Register to Ask a Question