The UNIX and Linux Forums
Forum Home
Linux and Unix Man Pages
Search Forums
Search Community Posts
Today's Posts
Quick Links
Man Pages
RedHat Commands
OpenSolaris Commands
Linux Commands
SunOS Commands
FreeBSD Commands
All UNIX Man Pages
All Linux Man Pages
Full Man Repository
Login or Register to Ask a Question and Join Our Community
Search Forums
Show Threads .
Show Posts
Tag Search
Advanced Search
Quick Links
Contact Us
Forum Rules
Today's Posts
FAQ
Pictures & Albums
All Albums
Miscellaneous
What is My IP
Whois
Mark Forums Read
Search Results
Showing results 1 to 2 of 2
Search took
0.00
seconds.
Search:
Posts Made By:
Kannan K
Forum:
UNIX for Advanced & Expert Users
01-07-2013
9
Performance problem with removing duplicates in a huge file (50+ GB)
3,800
Posted By
Kannan K
Sample Records
Sample records from file:
14480020180,A20180,A020180,143245765381,A00062,17284171796
14480020180,A20180,A020180,143245765381,A00062,17284171796
14480000127,A00127,A000127,143245730649,A00127, ...
Forum:
UNIX for Advanced & Expert Users
01-04-2013
9
Performance problem with removing duplicates in a huge file (50+ GB)
3,800
Posted By
Kannan K
Performance problem with removing duplicates in a huge file (50+ GB)
I'm trying to remove duplicate data from an input file with unsorted data which is of size >50GB and write the unique records to a new file.
I'm trying and already tried out a variety of options...
Showing results 1 to 2 of 2
Member Badges and Information Modal
×