File comaprsons for the Huge data files ( around 60G) - Need optimized and teh best way to do this

advanced, data, file, files, huge

Thread Tools Search this Thread
# 8  
Old 10-25-2018
Originally Posted by kartikirans
grep -F -x -v -f file2 file1 ?? or any other optimization command
sounds about right.
Just remember - whatever you do, comparing 60G files will be slow...
Test this on a smaller chunks to see if you're getting the desired results first.
# 9  
Old 10-31-2018
Hi kartikirans,

I'd be tempted to look at comm -3 ${file1} ${file2} this will suppress lines common to ${file1} and ${file2} later versions of comm don't require the files to be sorted.


# 10  
Old 10-31-2018
One additional question: what means "non-matching lines"?

- only Lines in file1 which are not in file2? or
- plus lines in file2 which are not in file1?


Thread Tools Search this Thread
Search this Thread:
Advanced Search

More UNIX and Linux Forum Topics You Might Find Helpful
Need Optimization shell/awk script to aggreagte (sum) for all the columns of Huge data file kartikirans UNIX for Advanced & Expert Users 2 10-23-2018 05:31 PM
The Fastest for copy huge data edydsuranta Solaris 11 09-17-2014 08:54 PM
Aggregation of huge data Ravichander Shell Programming and Scripting 8 04-07-2014 06:42 AM
Split a huge 7 GB File Based on Pattern into 4 files KishM UNIX for Dummies Questions & Answers 6 07-25-2013 09:18 AM
Disk is Full but really does not contain huge data kalpeer Red Hat 10 08-13-2012 05:11 AM
File comparison of huge files kaaliakahn UNIX for Dummies Questions & Answers 9 01-07-2012 10:39 PM
Copy huge data into vi editor alok.behria UNIX for Dummies Questions & Answers 18 08-31-2011 02:04 PM
Help- counting delimiter in a huge file and split data into 2 files lv99 Shell Programming and Scripting 7 03-01-2011 03:32 PM
Three Difference File Huge Data Comparison Problem. patrick87 Shell Programming and Scripting 4 10-22-2010 07:49 PM
Problem running Perl Script with huge data files ad23 Shell Programming and Scripting 4 07-09-2010 06:41 PM
Splitting the Huge file into several files... lakteja Shell Programming and Scripting 3 03-16-2010 12:13 PM
Split a huge data into few different files?! patrick87 Shell Programming and Scripting 7 11-02-2009 12:13 AM
insert a header in a huge data file without using an intermediate file deepaktanna Shell Programming and Scripting 10 02-23-2009 03:38 PM
How to extract data from a huge file? srsahu75 Shell Programming and Scripting 5 01-18-2008 05:06 AM
search and grab data from a huge file ting123 UNIX for Dummies Questions & Answers 1 06-06-2006 10:41 PM