Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

How to get unique of file1 from file2 and save the output?

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting How to get unique of file1 from file2 and save the output?
# 8  
Old 06-14-2013
Thanks on the reply,

Yes, but the result I need is unique of file1 that are not in file2.
# 9  
Old 06-14-2013
Sometimes you need "export LC_ALL=C" to get sort to do binary order for comm.
# 10  
Old 06-14-2013
richmac,
check this out, with the above data,


Code:
grep -v -w -f file2 file1
1
2

Enjoy..
# 11  
Old 06-14-2013
The comm and sort are large data stable, where grep gets slower with more stored lines and may blow up if it hits a 4G address limit putting file2 into VM. grep also has to check for regex at some stage, a waste on pure data if not a threat to data integrity; fgrep / grep -F is faster and more data-stable.

Awk and bash can hash search, which does not have speed problems with large files and can save the sorting step, but still has to put file2 into VM.
# 12  
Old 06-14-2013
Code:
diff file1 file2 --old-line-format="%L" --new-line-format="" --unchanged-line-format="" -h

or

diff file1 file2 --old-line-format="%L" --new-line-format="" --unchanged-line-format="" --speed-large-files

# 13  
Old 06-14-2013
Since diff does not assume order, it will search around for missing lines, even half heartedly, which might not scale well, performance-wise. It should be durable with large files, though.
# 14  
Old 06-14-2013
Hi All,

I tried
Code:
grep -v -w -f file2 file1 | xargs > result.txtthe

Problem is its already 2hours since i start run on that command. because as I said its a 2million records. and until now its not finish yet.

I tried also
Code:
diff file1 file2 --old-line-format="%L" --new-line-format="" --unchanged-line-format="" \
  -h  or  diff file1 file2 --old-line-format="%L" --new-line-format="" --unchanged-line-format="" \
  --speed-large-filesbut

Unfortunately the timeout still exist the result is still theysame as file1.
Smilie

Thanks so much on the reply. But still no luck. Smilie
Last edited by Scott; 06-14-2013 at 06:06 PM.. Reason: Added code tags, split code
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk to search field2 in file2 using range of fields file1 and using match to another field in file1

I am trying to use awk to find all the $2 values in file2 which is ~30MB and tab-delimited, that are between $2 and $3 in file1 which is ~2GB and tab-delimited. I have just found out that I need to use $1 and $2 and $3 from file1 and $1 and $2of file2 must match $1 of file1 and be in the range... (6 Replies)
Discussion started by: cmccabe
6 Replies

2. UNIX for Dummies Questions & Answers

Compare file1 and file2, print matching lines in same order as file1

I want to print only the lines in file2 that match file1, in the same order as they appear in file 1 file1 file2 desired output: I'm getting the lines to match awk 'FNR==NR {a++}; FNR!=NR && a' file1 file2 but they are in sorted order, which is not what I want: Can anyone... (4 Replies)
Discussion started by: pathunkathunk
4 Replies

3. Shell Programming and Scripting

Compare columns of multiple files and print those unique string from File1 in an output file.

Hi, I have multiple files that each contain one column of strings: File1: 123abc 456def 789ghi File2: 123abc 456def 891jkl File3: 234mno 123abc 456def In total I have 25 of these type of file. (5 Replies)
Discussion started by: owwow14
5 Replies

4. Shell Programming and Scripting

If file1 and file2 exist then

HI, I would like a little help on writing a if statement. What i have so far is: #!/bin/bash FILE1=path/to/file1 FILE2=path/to/file2 echo ${FILE1} ${FILE2} if ] then echo file1 and file2 not found else echo FILE ok fi (6 Replies)
Discussion started by: techy1
6 Replies

5. Shell Programming and Scripting

[awk] split file1 and save it as var from file2

I have 2 files: file_1: file_2: expected result: name file: "artV1" "artV2" etc. I have: but why don;t work save to file 'out'?? (3 Replies)
Discussion started by: ffresz
3 Replies

6. Shell Programming and Scripting

look for line from FILE1 at FILE2

Hi guys! I'm trying to write something to find each line of file1 into file2, if line is found return YES, if not found return NO. The result can be written to a new file. Can you please help me out? FILE1 INPUT: WATER CAR SNAKE (in reality this file has about 600 lines each with a... (2 Replies)
Discussion started by: demmel
2 Replies

7. UNIX for Dummies Questions & Answers

if matching strings in file1 and file2, add column from file1 to file2

I have very limited coding skills but I'm wondering if someone could help me with this. There are many threads about matching strings in two files, but I have no idea how to add a column from one file to another based on a matching string. I'm looking to match column1 in file1 to the number... (3 Replies)
Discussion started by: pathunkathunk
3 Replies

8. Shell Programming and Scripting

file1 newer then file2

Hello, I am new to shell scripting and i need to create a script with the following directions and I can not figure it out. Create a shell script called newest.bash that takes two filenames as input arguments ($1 and $2) and prints out the name of the newest file (i.e. the file with the... (1 Reply)
Discussion started by: mandylynn78
1 Replies

9. Shell Programming and Scripting

grep -f file1 file2

Wat does this command do? fileA is a subset of fileB..now, i need to find the lines in fileB that are not in fileA...i.e fileA - fileB. diff fileA fileB gives the ouput but the format looks no good.... I just need the contents alone not the line num etc. (7 Replies)
Discussion started by: vijay_0209
7 Replies

10. Shell Programming and Scripting

match value from file1 in file2

Hi, i've two files (file1, file2) i want to take value (in column1) and search in file2 if the they match print the value from file2. this is what i have so far. awk 'FILENAME=="file1"{ arr=$1 } FILENAME=="file2" {print $0} ' file1 file2 (2 Replies)
Discussion started by: myguess21
2 Replies
Login or Register to Ask a Question