File comaprsons for the Huge data files ( around 60G) - Need optimized and teh best way to do this

Tags
advanced, data, file, files, huge

 
Thread Tools Search this Thread
# 1  
Old 10-25-2018
File comaprsons for the Huge data files ( around 60G) - Need optimized and teh best way to do this

I have 2 large file (.dat) around 70 g, 12 columns but the data not sorted in both the files.. need your inputs in giving the best optimized method/command to achieve this and redirect the not macthing lines to the thrid file ( diff.dat)


File 1 - 15 columns
File 2 - 15 columns

Data is not in sorted order.
# 2  
Old 10-25-2018
What is this in method/command to achieve this?
Sample files and the desired output would help as well...
# 3  
Old 10-25-2018
sample look-

Code:
2036|001|021|92|570|2|422|1|0|0|0|570|0|0|12

Field separate - "|"

File 1 Size ( 60 G)
File 2 Size ( 61 g)
Note - data is not in the sorted order ( file1 and file2)

Requirement, I need to find the not matching lines and redirect those to new file "differnce.dat"

Last edited by vgersh99; 10-25-2018 at 11:19 AM..
# 4  
Old 10-25-2018
what constitutes "non-matching" lines?
Entire line or some key fields in file1 and 2 to match on?
You have to be clearer with your requirement statements.

Also, please use code tags when posting code/data samples.
# 5  
Old 10-25-2018
Thanks for the quick reply, Entire line...
# 6  
Old 10-25-2018
look into man grep with options -F and -f.
Or man fgrep
# 7  
Old 10-25-2018
grep -F -x -v -f file2 file1 ?? or any other optimization command

|
Thread Tools Search this Thread
Search this Thread:
Advanced Search

More UNIX and Linux Forum Topics You Might Find Helpful
Need Optimization shell/awk script to aggreagte (sum) for all the columns of Huge data file kartikirans UNIX for Advanced & Expert Users 2 10-23-2018 05:31 PM
awk does not work well with huge data? ariesto Shell Programming and Scripting 4 08-12-2014 12:05 AM
Aggregation of huge data Ravichander Shell Programming and Scripting 8 04-07-2014 06:42 AM
Split a huge 7 GB File Based on Pattern into 4 files KishM UNIX for Dummies Questions & Answers 6 07-25-2013 09:18 AM
Disk is Full but really does not contain huge data kalpeer Red Hat 10 08-13-2012 05:11 AM
File comparison of huge files kaaliakahn UNIX for Dummies Questions & Answers 9 01-07-2012 10:39 PM
Copy huge data into vi editor alok.behria UNIX for Dummies Questions & Answers 18 08-31-2011 02:04 PM
Help- counting delimiter in a huge file and split data into 2 files lv99 Shell Programming and Scripting 7 03-01-2011 03:32 PM
Three Difference File Huge Data Comparison Problem. patrick87 Shell Programming and Scripting 4 10-22-2010 07:49 PM
Problem running Perl Script with huge data files ad23 Shell Programming and Scripting 4 07-09-2010 06:41 PM
Splitting the Huge file into several files... lakteja Shell Programming and Scripting 3 03-16-2010 12:13 PM
Split a huge data into few different files?! patrick87 Shell Programming and Scripting 7 11-02-2009 12:13 AM
insert a header in a huge data file without using an intermediate file deepaktanna Shell Programming and Scripting 10 02-23-2009 03:38 PM
How to extract data from a huge file? srsahu75 Shell Programming and Scripting 5 01-18-2008 05:06 AM
search and grab data from a huge file ting123 UNIX for Dummies Questions & Answers 1 06-06-2006 10:41 PM