Home Man
Search
Today's Posts
Register

Expert-to-Expert. Learn advanced UNIX, UNIX commands, Linux, Operating Systems, System Administration, Programming, Shell, Shell Scripts, Solaris, Linux, HP-UX, AIX, OS X, BSD.

Need optimized awk/perl/shell to give the statistics for the Large delimited file

Tags
advanced, awk, delimited, delimited file, file, ics, large, perl, shell, statistics

Login to Reply

 
Thread Tools Search this Thread
# 1  
Old 1 Week Ago
Need optimized awk/perl/shell to give the statistics for the Large delimited file

I have a file size is around 24 G with 14 columns, delimiter with "|"

My requirement- can anyone provide me the fastest and best to get the below results

Number of records of the file
First column and second Column- Unique counts

Thanks for your time
Karti

------ Post updated at 04:03 PM ------

Correction -

Number of records of the file
First column and second Column- Distinct column values , not the counts.
# 2  
Old 1 Week Ago
Try:
Code:
awk -F\| '!A[$1]++{c1++}; !B[$2]++{c2++} END{print c1, c2, NR}' file

The Following User Says Thank You to Scrutinizer For This Useful Post:
kartikirans (1 Week Ago)
# 3  
Old 1 Week Ago
Thanks , I need to redirect the Distinct column1 and column2 to dis_col1.txt and dis_col2.txt files. File SIze is Huge ( 24 G). Appreciate for your quick reply and time
# 4  
Old 1 Week Ago
something like:

Code:
awk -F\| '!a[$1]++ { print $1 > "dis_col1.txt"; } !b[$2]++ { print $2 > "dis_col2.txt"; } END { print NR; }' file

The Following User Says Thank You to neutronscott For This Useful Post:
kartikirans (1 Week Ago)
Login to Reply

« Previous Thread | Next Thread »
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
UNIX/PERL script to convert XML file to pipe delimited format karthi1305561 Shell Programming and Scripting 15 11-06-2015 11:55 PM
Perl script give answers by file Priy Shell Programming and Scripting 2 11-27-2013 05:21 AM
Removing dupes within 2 delimited areas in a large dictionary file gimley Shell Programming and Scripting 6 12-07-2012 08:50 AM
Shell Script help for Quotation Delimited File nickytcom Shell Programming and Scripting 5 05-21-2012 02:18 PM
Script Optimization - large delimited file, for loop with many greps verge Shell Programming and Scripting 17 04-27-2011 09:15 PM
Extracting a portion of data from a very large tab delimited text file Lucky Ali Shell Programming and Scripting 2 04-11-2010 11:55 AM
Create tab-delimited file of outputs - Perl shwang3 Shell Programming and Scripting 1 04-06-2010 07:49 PM
Large pipe delimited file that I need to add CR/LF every n fields clintrpeterson Shell Programming and Scripting 2 10-15-2009 02:15 PM
Trim String in 3rd Column in Tab Delimited File...SED/PERL/AWK? rickdini UNIX for Dummies Questions & Answers 6 09-29-2008 02:35 AM
Optimized way of doing the task in shell programming pcjandyala Shell Programming and Scripting 14 06-12-2008 02:54 AM


All times are GMT -4. The time now is 07:29 AM.

Unix & Linux Forums Content Copyrightę1993-2018. All Rights Reserved.
UNIX.COM Login
Username:
Password:  
Show Password