Compare 200,000 of rows in two text files


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Compare 200,000 of rows in two text files
# 1  
Old 04-03-2009
Compare 200,000 of rows in two text files

Friends,

I have two very large plain text files with pipe delimited as below.
Both files are not sorted.
Both files have 200,000 of rows.

FName|LName|Address|HPhNumber

Is perl or shell script feasible for this task?

Thanks,
Prashant
# 2  
Old 04-03-2009
This shell script may do what you are after?
Code:
sort file1 > file1.sorted
sort file2 > file2.sorted
sdiff file1.sorted file2.sorted

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Compare 2 csv files by columns, then extract certain columns of matcing rows

Hi all, I'm pretty much a newbie to UNIX. I would appreciate any help with UNIX coding on comparing two large csv files (greater than 10 GB in size), and output a file with matching columns. I want to compare file1 and file2 by 'id' and 'chain' columns, then extract exact matching rows'... (5 Replies)
Discussion started by: bkane3
5 Replies

2. UNIX for Dummies Questions & Answers

Take 100MB worth files from 200,000 Files

Hi, I have a process which creates almost 200K files. Each file ranging from 1kb to 5kb. This is almost 2GB of data in all files. I have a requirement where the business needs only 100MB worth of files. Is there a way to get files around 100MB (doesn't have to be exactly 100MB) from all the... (2 Replies)
Discussion started by: grep_me
2 Replies

3. UNIX for Dummies Questions & Answers

Extract unique combination of rows from text files

Hi Gurus, I have 100 tab-delimited text files each with 21 columns. I want to extract only 2nd and 5th column from each text file. However, the values in both 2bd and 5th column contain duplicate values but the combination of these values in a row are not duplicate. I want to extract only those... (3 Replies)
Discussion started by: Unilearn
3 Replies

4. Shell Programming and Scripting

Compare two fields in text files?

Hi, I have two text files, compare column one in both the files and if it matches then the output should contain the id in column one, the number and the description. Both the files are sorted. Is there a one liner to get this done, kindly help. Thank you File 1: NC_000964 92.33 ... (2 Replies)
Discussion started by: pulikoti
2 Replies

5. Shell Programming and Scripting

Script to compare two text files

i am working on a shell script and need help in the comparing part of it. for e.g. there two text files like this: file1.txt name1 name2 name3 file1 has to be comared with file2 defaultfile.txt name1 name2 name3 name4 and during comparision with defaultfile.txt if... (2 Replies)
Discussion started by: draghun9
2 Replies

6. UNIX for Dummies Questions & Answers

Compare two text files

Hello guys, I have file1 and file2, two text files containing various lines. I'm trying to find a way to compare file1 and file2: If the first 7 characters of a line in file2 match the first 7 characters of a line in file1, then do not do anything. Print out the lines of file1 (in file3,... (3 Replies)
Discussion started by: bobylapointe
3 Replies

7. Shell Programming and Scripting

Compare values in two files. For matching rows print corresponding values from File 1 in File2.

- I have two files (File 1 and File 2) and the contents of the files are mentioned below. - I am trying to compare the values of Column1 of File1 with Column1 of File2. If a match is found, print the corresponding value from Column2 of File1 in Column5 of File2. - I tried to modify and use... (10 Replies)
Discussion started by: Santoshbn
10 Replies

8. Shell Programming and Scripting

How to compare two text files

Hi Team, Could you please help me on below one .. etrademail1.txt etDefaultLogin=pdayanan mail=poojaaragam.dayanand@exchange.etr.comx employeeNumber=31567 etDefaultLogin=sudrupa mail=sudrupa.ayanand@exchange.etr.comx employeeNumber=318967 etDefaultLogin=gurathi (1 Reply)
Discussion started by: nivas_k2006
1 Replies

9. UNIX for Dummies Questions & Answers

Split 200.000 files into different subfolders

Dear UNIX-Community, can help me doing 2 things in Debian 5.0? 1.) Create 100 folders Format: ./0/0, ./0/1, ./0/2, ...,./0/9, ./1/0/, ..., ./9/9 2.) Move over 200.000 files into the subdirectories according to their last digits. 12398123.dat -> ./3/2/12398123.dat 48161.dat ->... (2 Replies)
Discussion started by: diadas
2 Replies

10. UNIX for Dummies Questions & Answers

compare text files

This may be the 3rd time I'm posting this question. I'm so new here that I'm not even sure how to post! I'm trying to compare two files but can't do a line by line comparison so comm and diff are out. I've been told that I would need to use the awk programing language. I've looked up what I... (14 Replies)
Discussion started by: jimmyflip
14 Replies
Login or Register to Ask a Question