Need optimized awk/perl/shell to give the statistics for the Large delimited file


Login or Register to Reply

 
Thread Tools Search this Thread
# 1  
Old 09-13-2018
Need optimized awk/perl/shell to give the statistics for the Large delimited file

I have a file size is around 24 G with 14 columns, delimiter with "|"

My requirement- can anyone provide me the fastest and best to get the below results

Number of records of the file
First column and second Column- Unique counts

Thanks for your time
Karti

------ Post updated at 04:03 PM ------

Correction -

Number of records of the file
First column and second Column- Distinct column values , not the counts.
# 2  
Old 09-13-2018
Try:
Code:
awk -F\| '!A[$1]++{c1++}; !B[$2]++{c2++} END{print c1, c2, NR}' file

This User Gave Thanks to Scrutinizer For This Post:
kartikirans (09-13-2018)
# 3  
Old 09-13-2018
Thanks , I need to redirect the Distinct column1 and column2 to dis_col1.txt and dis_col2.txt files. File SIze is Huge ( 24 G). Appreciate for your quick reply and time
# 4  
Old 09-14-2018
something like:

Code:
awk -F\| '!a[$1]++ { print $1 > "dis_col1.txt"; } !b[$2]++ { print $2 > "dis_col2.txt"; } END { print NR; }' file

This User Gave Thanks to neutronscott For This Post:
kartikirans (09-18-2018)
Login or Register to Reply

|
Thread Tools Search this Thread
Search this Thread:
Advanced Search

More UNIX and Linux Forum Topics You Might Find Helpful
Perl script give answers by file Priy Shell Programming and Scripting 2 11-27-2013 05:21 AM
Removing dupes within 2 delimited areas in a large dictionary file gimley Shell Programming and Scripting 6 12-07-2012 08:50 AM
awk read one delimited file, search another delimited file dagamier Shell Programming and Scripting 4 11-19-2012 12:20 PM
AWK, Perl or Shell? Unique strings and their maximum values from 3 column data file rich@ardz Shell Programming and Scripting 4 02-22-2012 05:14 AM
Awk getting statistics of a grid file, alex2005 Shell Programming and Scripting 6 06-02-2011 07:33 AM
Script Optimization - large delimited file, for loop with many greps verge Shell Programming and Scripting 17 04-27-2011 09:15 PM
awk help - delimited file ashwin3086 Shell Programming and Scripting 8 10-30-2010 12:42 PM
Extracting a portion of data from a very large tab delimited text file Lucky Ali Shell Programming and Scripting 2 04-11-2010 11:55 AM
Create tab-delimited file of outputs - Perl shwang3 Shell Programming and Scripting 1 04-06-2010 07:49 PM
Large pipe delimited file that I need to add CR/LF every n fields clintrpeterson Shell Programming and Scripting 2 10-15-2009 02:15 PM
AWK Shell Program to Split Large Files mkastin Shell Programming and Scripting 16 06-29-2009 11:51 PM
Trim String in 3rd Column in Tab Delimited File...SED/PERL/AWK? rickdini UNIX for Dummies Questions & Answers 6 09-29-2008 02:35 AM
Could someone give me an example of awk accessing array defined in Korn Shell? biglau Shell Programming and Scripting 2 06-19-2008 07:26 AM
Optimized way of doing the task in shell programming pcjandyala Shell Programming and Scripting 14 06-12-2008 02:54 AM