Sponsored Content
Full Discussion: Pivot Column using awk
Top Forums Shell Programming and Scripting Pivot Column using awk Post 302932997 by morbid_angel on Tuesday 27th of January 2015 12:26:40 AM
Old 01-27-2015
Pivot Column using awk

Hello everyone

I have a 20M file which is having the below sample layout

Code:
1111,ABC,100
1111,DEF,200
1111,XYZ,300
4444,LMN,100
4444,AKH,500
4444,WCD,400
2222,ABC,100
7777,DEF,300
7777,WCD,300

I need to covert this to below format
Output

Code:
Party_ID|ABC|AKH|DEF|LMN|WCD|XYZ
1111|100|0|200|0|0|300
2222|100|0|0|0|0|0
4444|0|500|0|100|400|0
7777|0|0|300|0|300|0

How can I make this through UNIX and it should be pretty faster.My current script take huge time to pivot for 20M records

Last edited by Don Cragun; 01-27-2015 at 04:25 AM.. Reason: Add CODE tags.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

for each different entry in column 1 extract maximum values from column 2 in unix/awk

Hello, I have 2 columns (1st column has multiple entries but the corresponding values in the column 2 may be the same or different.) however I want to extract unique values for each entry in column 1 by assigning the max value from column 2 SDF4 -0.211654 SDF4 0.978068 ... (1 Reply)
Discussion started by: Diya123
1 Replies

2. Shell Programming and Scripting

Help newbie: transposing column into row (pivot)

Hi, I have a file in this format. Name | organization John | INT Abby| DOM John | DOM John | MIX Jason | INT Anna | DOM Abby |MIX I want the output to look this. Name | organization John | INT, DOM, MIX Abby | DOM, MIX Jason | INT Anna | DOM (5 Replies)
Discussion started by: sirrtuan
5 Replies

3. Shell Programming and Scripting

AWK script to create max value of 3rd column, grouping by first column

Hi, I need an awk script (or whatever shell-construct) that would take data like below and get the max value of 3 column, when grouping by the 1st column. clientname,day-of-month,max-users ----------------------------------- client1,20120610,5 client2,20120610,2 client3,20120610,7... (3 Replies)
Discussion started by: ckmehta
3 Replies

4. Shell Programming and Scripting

awk Print New Column For Every Two Lines and Match On Multiple Column Values to print another column

Hi, My input files is like this axis1 0 1 10 axis2 0 1 5 axis1 1 2 -4 axis2 2 3 -3 axis1 3 4 5 axis2 3 4 -1 axis1 4 5 -6 axis2 4 5 1 Now, these are my following tasks 1. Print a first column for every two rows that has the same value followed by a string. 2. Match on the... (3 Replies)
Discussion started by: jacobs.smith
3 Replies

5. Shell Programming and Scripting

Pivot using awk

Hi, I am writing a code to basically pivot the data. awk -v var1="" -v var2="" -v var3="" -v var4="" -v var5="" -v Disp=0\ 'BEGIN {FS=":"; OFS="|";}\ /^Pattern1/ {var1=$2;Disp=0;} \ /^Pattern2/ {var2=$2;} \ /^Pattern3/ {var3=$2;} \ /^Pattern4/ {var4=$2;} \ /^Pattern5/... (5 Replies)
Discussion started by: tostay2003
5 Replies

6. Shell Programming and Scripting

Problems with awk (fatal error) and paste (two variables into one column-by-column)

Hello, I have a script extracting columns of useful numbers from a data file, and manipulating the numbers with awk commands. I have problems with my script... 1. There are two lines assigning numbers to $BaseForAveraging. If I use the commented line (the first one) and let the second one... (9 Replies)
Discussion started by: vgbraymond
9 Replies

7. Shell Programming and Scripting

Pivot data using awk

Hi My Input is like below DELETE|MPI|AUD_UPD_AGENT|MPISYS INSERT|MPI|AUD_UPD_AGENT|MPISYS SELECT|MPI|AUD_UPD_AGENT|MPISYS UPDATE|MPI|AUD_UPD_AGENT|MPISYS DELETE|MPI|BDYMOD|MPISYS INSERT|MPI|BDYMOD|MPISYS SELECT|MPI|BDYMOD|MPISYS UPDATE|MPI|BDYMOD|MPISYS DELETE|MPI|BDYMOD_DESC|MPISYS... (4 Replies)
Discussion started by: dineshaila
4 Replies

8. Shell Programming and Scripting

Awk- Pivot Table Averages

Hi everyone, Has anyone figured out yet how to do pivot table averages using AWK. I didn't see anything with regards to doing averages. For example, suppose you have the following table with various individuals and their scores in round1 and round2: SAMPLE SCORE1 SCORE2 British ... (6 Replies)
Discussion started by: Geneanalyst
6 Replies

9. Shell Programming and Scripting

awk to Sum columns when other column has duplicates and append one column value to another with Care

Hi Experts, Please bear with me, i need help I am learning AWk and stuck up in one issue. First point : I want to sum up column value for column 7, 9, 11,13 and column15 if rows in column 5 are duplicates.No action to be taken for rows where value in column 5 is unique. Second point : For... (1 Reply)
Discussion started by: as7951
1 Replies

10. Shell Programming and Scripting

awk script to append suffix to column when column has duplicated values

Please help me to get required output for both scenario 1 and scenario 2 and need separate code for both scenario 1 and scenario 2 Scenario 1 i need to do below changes only when column1 is CR and column3 has duplicates rows/values. This inputfile can contain 100 of this duplicated rows of... (1 Reply)
Discussion started by: as7951
1 Replies
Vend::Payment::PSiGate(3pm)				User Contributed Perl Documentation			       Vend::Payment::PSiGate(3pm)

NAME
Vend::Payment::PSiGate - Interchange PSiGate Support SYNOPSIS
&charge=psigate or [charge mode=psigate param1=value1 param2=value2] PREREQUISITES
Net::SSLeay or LWP::UserAgent and Crypt::SSLeay Only one of these need be present and working. DESCRIPTION
The Vend::Payment::PSiGate module implements the psigate() routine for use with Interchange. It is compatible on a call level with the other Interchange payment modules. To enable this module, place this directive in "interchange.cfg": Require module Vend::Payment::PSiGate This must be in interchange.cfg or a file included from it. Make sure CreditCardAuto is off (default in Interchange demos). The mode can be named anything, but the "gateway" parameter must be set to "psigate". To make it the default payment gateway for all credit card transactions in a specific catalog, you can set in "catalog.cfg": Variable MV_PAYMENT_MODE psigate It uses several of the standard settings from Interchange payment. Any time we speak of a setting, it is obtained either first from the tag/call options, then from an Interchange order Route named for the mode, then finally a default global payment variable, For example, the "id" parameter would be specified by: [charge mode=psigate id=YourPSiGateMerchantID] or Route psigate id YourPSiGateMerchantID or Variable MV_PAYMENT_ID YourPSiGateMerchantID The active settings are: id Your PSiGate.com account ID, supplied by PSiGate.com when you sign up. Global parameter is MV_PAYMENT_ID. referer A valid referering url (match this with your setting on PSiGate). Global parameter is MV_PAYMENT_REFERER. transaction The type of transaction to be run. Valid values are: Interchange PSiGate Note ---------------- ----------------- --------- auth '1', PreAuth sale '0', Sale settle '2', PostAuth void '9', Void remap This remaps the form variable names to the ones needed by PSiGate. See the "Payment Settings" heading in the Interchange documentation for use. test Set this to "TRUE" if you wish to operate in test mode, i.e. set the PSiGate "x_Test_Request" query paramter to TRUE.i Examples: Route psigate test TRUE or Variable MV_PAYMENT_TEST TRUE or [charge mode=psigate test=TRUE] Troubleshooting Try the instructions above, then enable test mode. A test order should complete. Disable test mode, then test in various PSiGate error modes by using the credit card number 4111 1111 1111 1111. Then try a sale with the card number "4111 1111 1111 1111" and a valid expiration date. The sale should be denied, and the reason should be in [data session payment_error]. If nothing works: o Make sure you "Require"d the module in interchange.cfg: Require module Vend::Payment::PSiGate o Make sure either Net::SSLeay or Crypt::SSLeay and LWP::UserAgent are installed and working. You can test to see whether your Perl thinks they are: perl -MNet::SSLeay -e 'print "It works "' or perl -MLWP::UserAgent -MCrypt::SSLeay -e 'print "It works "' If either one prints "It works." and returns to the prompt you should be OK (presuming they are in working order otherwise). o Check the error logs, both catalog and global. o Make sure you set your payment parameters properly. o Try an order, then put this code in a page: <XMP> [calc] my $string = $Tag->uneval( { ref => $Session->{payment_result} }); $string =~ s/{/{ /; $string =~ s/,/, /g; return $string; [/calc] </XMP> That should show what happened. o If all else fails, Red Hat and other consultants are available to help with integration for a fee. BUGS
There is actually nothing *in* Vend::Payment::PSiGate. It changes packages to Vend::Payment and places things there. AUTHORS
Mark Stosberg <mark@summersault.com>, based on original code by Mike Heins <mheins@redhat.com>. CREDITS
Jeff Nappi <brage@cyberhighway.net> Paul Delys <paul@gi.alaska.edu> webmaster@nameastar.net Ray Desjardins <ray@dfwmicrotech.com> Nelson H. Ferrari <nferrari@ccsc.com> perl v5.14.2 2011-03-09 Vend::Payment::PSiGate(3pm)
All times are GMT -4. The time now is 10:42 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy