Some lines to many in a file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Some lines to many in a file
# 1  
Old 07-15-2009
Some lines to many in a file

I need to find out which lines are in one file and are not in the other file.
Lines in files contain blank spaces as well.

File1:
Code:
some line here
some other line
some third line

File2:
Code:
some other line
some line here

The Result I Wont to get is:

File3:
Code:
some third line

I have done some searching thru the forum and I have found this:
Code:
grep -v -f File2 File1 > File3

That would solve the given example, BUT my files are about 120MB. Performing the opeartion on them results with "memory exousted" error.

I have tried spliting the file to smaller shunks so that every contains 100000 lines (~6MB), then I get two times (for every file) 30 files. That means that i have to make 30x30 operations. First operation that I started 40 min ago is still working..., and I dont expect positive result.

Last edited by Yogesh Sawant; 07-15-2009 at 05:38 AM.. Reason: added code tags
# 2  
Old 07-15-2009
try diff
# 3  
Old 07-15-2009
But diff compares like this:

some line here <-> some other line
some other line <-> some line here
some third line <->

Or?
# 4  
Old 07-15-2009
Code:
diff file1 file2

i hope that will help
# 5  
Old 07-15-2009
one of the ways:

Code:
grep -v -f file2 file1

# 6  
Old 07-15-2009
cmp file1 file2
# 7  
Old 07-15-2009
Loop line by line
Code:
#!/bin/ksh
while read line
do
      grep  "^${line}$"  file2 >/dev/null 2>&1 || echo "not in file2:$line"
done < file1

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Convert a horizontal lines to vertical lines in a csv file

Hi.. I need some help in converting the below horizontal lines to vertical lines format. can anyone help me on this. input file Hour,1,2,3,4,5 90RT,106,111,111,112,111 output file Hour,90RT 1,106 2,111 3,111 4,112 5,111 (3 Replies)
Discussion started by: Raghuram717
3 Replies

2. Shell Programming and Scripting

Find all lines in file such that each word on that line appears in at least n lines of the file

I have a file where every line includes four expressions with a caret in the middle (plus some other "words" or fields, always separated by spaces). I would like to extract from this file, all those lines such that each of the four expressions containing a caret appears in at least four different... (9 Replies)
Discussion started by: uncleMonty
9 Replies

3. UNIX for Advanced & Expert Users

SQL script with 86000 lines: new files with only 10000 lines (per file)

Hi this is my SQL script $ wc -l insert_into_customers.sql 85601 insert_into_customers.sqlI wish to cut this file into 9 files each 10000 lines (the last one less) $ wc -l insert_into_customers_00*.sql 10000 insert_into_customers_001.sql 10000 insert_into_customers_002.sql ... (1 Reply)
Discussion started by: slashdotweenie
1 Replies

4. Shell Programming and Scripting

Extract some lines from one file and add those lines to current file

hi, i have two files. file1.sh echo "unix" echo "linux" file2.sh echo "unix linux forums" now the output i need is $./file2.sh unix linux forums (3 Replies)
Discussion started by: snreddy_gopu
3 Replies

5. UNIX for Advanced & Expert Users

In a huge file, Delete duplicate lines leaving unique lines

Hi All, I have a very huge file (4GB) which has duplicate lines. I want to delete duplicate lines leaving unique lines. Sort, uniq, awk '!x++' are not working as its running out of buffer space. I dont know if this works : I want to read each line of the File in a For Loop, and want to... (16 Replies)
Discussion started by: krishnix
16 Replies

6. Shell Programming and Scripting

merging two .txt files by alternating x lines from file 1 and y lines from file2

Hi everyone, I have two files (A and B) and want to combine them to one by always taking 10 rows from file A and subsequently 6 lines from file B. This process shall be repeated 40 times (file A = 400 lines; file B = 240 lines). Does anybody have an idea how to do that using perl, awk or sed?... (6 Replies)
Discussion started by: ink_LE
6 Replies

7. Shell Programming and Scripting

Select lines in which column have value greater than some percent of total file lines

i have a file in following format 1 32 3 4 6 4 4 45 1 45 4 61 54 66 4 5 65 51 56 65 1 12 32 85 now here the total number of lines are 8(they vary each time) Now i want to select only those lines in which the values... (6 Replies)
Discussion started by: vaibhavkorde
6 Replies

8. Shell Programming and Scripting

Finding lines matching the Pattern and their previous lines in a file

Hi, I am trying to locate the occurences of certain pattern like 'Possible network disconnect' in a text file. I can get the actual lines matching the pttern using: grep -w 'Possible network disconnect' file_name. But I am more interested in getting the timing of these events which are... (7 Replies)
Discussion started by: sagarparadkar
7 Replies

9. UNIX for Dummies Questions & Answers

How get only required lines & delete the rest of the lines in file

Hiiii I have a file which contains huge data as a.dat: PDE 1990 1 9 18 51 28.90 24.7500 95.2800 118.0 6.1 0.0 BURMA event name: 010990D time shift: 7.3000 half duration: 5.0000 latitude: 24.4200 longitude: 94.9500 depth: 129.6000 Mrr: ... (7 Replies)
Discussion started by: reva
7 Replies

10. Shell Programming and Scripting

Extra/parse lines from a file between unque lines through the file

I need help to parse a file where there are many records, all of which are consistently separated by lines containing “^=============” and "^ End of Report". Example: ============= 1 2 3 4 End of record ============= 1 3 4 End of record Etc.... I only need specific lines... (5 Replies)
Discussion started by: jouuu
5 Replies
Login or Register to Ask a Question