Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Grep -v -f and sort|diff which way is faster
Top Forums Shell Programming and Scripting Grep -v -f and sort|diff which way is faster Post 302922969 by MadeInGermany on Wednesday 29th of October 2014 01:28:58 PM
Old 10-29-2014
The file after the -f is read into memory, so it should be
Code:
grep -v -f smallfile bigfile

Some grep versions are rather slow on this, and even the faster full-line-match is not a race car:
Code:
grep -v -x -f smallfile bigfile

In this case consider replacing it by awk:
Code:
awk 'FILENAME=="-" {s[$0]; next} !($0 in s)' - bigfile <smallfile

This User Gave Thanks to MadeInGermany For This Post:
ken6503
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Sort/Grep Question

Hello all, I have a test file that has the format: ..... O 3.694950 -.895050 1.480000 O 5.485050 .895050 1.480000 Ti -4.590000 4.590000 2.960000 Ti -2.295000 ... (5 Replies)
Discussion started by: aarondesk
5 Replies

2. UNIX for Dummies Questions & Answers

How to grep faster ?

Hi I have to grep for 2000 strings in a file one after the other.Say the file name is Snxx.out which has these strings. I have to search for all the strings in the file Snxx.out one after the other. What is the fastest way to do it ?? Note:The current grep process is taking lot of time per... (7 Replies)
Discussion started by: preethgideon
7 Replies

3. Shell Programming and Scripting

how to grep sort userids

hello folks i have a file that have data like /test/aa/123 /test/aa/xyz /test/bb/xyz /test/bb/123 in above lines i just wants to grep "aa" and "bb". Thanks, Bash (4 Replies)
Discussion started by: learnbash
4 Replies

4. Shell Programming and Scripting

[solved] Diff between two files by grep

My requiremeny is as follows, I have two files file a A BONES RD,NHILL,3418,VIC 37TH PARALLEL RD,DEEP LEAD,3385,VIC 4 AK RD,OAKEY,4401,QLD A & J FARRS RD,BARMOYA,4703,QLD A B PATTERSON DR,ARUNDEL,4214,QLD A BLAIRS RD,BUCKRABANYULE,3525,VIC file b A BONES... (12 Replies)
Discussion started by: feelmyfrd
12 Replies

5. Shell Programming and Scripting

grep from 3 lines and sort

Pseudo name=hdiskpower54 Symmetrix ID=000190101757 Logical device ID=0601 state=alive; policy=SymmOpt; priority=0; queued-IOs=0 ============================================================================== ---------------- Host --------------- - Stor - -- I/O Path - -- Stats --- ### HW... (7 Replies)
Discussion started by: Daniel Gate
7 Replies

6. Shell Programming and Scripting

Diff between grep .* file name and grep '.*' filename

Hi, Can anyone let me know what is difference between grep .* foo.c grep '.*' foo.c I am not able to understand what is exact difference. Thanks in advance (2 Replies)
Discussion started by: SasDutta
2 Replies

7. Homework & Coursework Questions

Grep and Sort

Use and complete the template provided. The entire template must be completed. If you don't, your post may be deleted! 1. The problem statement, all variables and given/known data: 1. Print the number of people that are in the /etc/passwd file with the name of George 2. Sort by name and... (8 Replies)
Discussion started by: Jagst3r21
8 Replies

8. Homework & Coursework Questions

awk with Grep and Sort

1. The problem statement, all variables and given/known data: Please bare in mind I am a complete novice to this and have very very basic knowledge so please keep any answers as simple as possible and explain in terms I will understand ahha :):) I have a text file of names and test scores... (1 Reply)
Discussion started by: jamesb18
1 Replies

9. UNIX for Dummies Questions & Answers

What is the faster way to grep from huge file?

Hi All, I am new to this forum and this is my first post. My requirement is like to optimize the time taken to grep the file with 40000 lines. There are two files FILEA(40000 lines) FILEB(40000 lines). The requirement is like this, both the file will be in the format below... (11 Replies)
Discussion started by: mad man
11 Replies

10. UNIX for Advanced & Expert Users

Need help for faster file read and grep in big files

I have a very big input file <inputFile1.txt> which has list of mobile no inputFile1.txt 3434343 3434323 0970978 85233 ... around 1 million records i have another file as inputFile2.txt which has some log detail big file inputFile2.txt afjhjdhfkjdhfkd df h8983 3434343 | 3483 | myout1 |... (3 Replies)
Discussion started by: reldb
3 Replies

LEARN ABOUT X11R4

zcmp

ZDIFF(1)						      General Commands Manual							  ZDIFF(1)

NAME

       zcmp, zdiff - compare compressed files

SYNOPSIS

       zcmp [ cmp_options ] file1 [ file2 ]
       zdiff [ diff_options ] file1 [ file2 ]

DESCRIPTION

       Zcmp  and  zdiff are used to invoke the cmp or the diff program on files compressed via gzip.  All options specified are passed directly to
       cmp or diff.  If only file1 is specified, it is compared to the uncompressed contents of file1.gz.  If two files are specified, their  con-
       tents  (uncompressed  if  necessary)  are  fed to cmp or diff.  The input files are not modified.  The exit status from cmp or diff is pre-
       served.

SEE ALSO

       cmp(1), diff(1), zmore(1), zgrep(1), znew(1), zforce(1), gzip(1), gzexe(1)

BUGS

       Messages from the cmp or diff programs may refer to file names such as "-" instead of to the file names specified.

																	  ZDIFF(1)
All times are GMT -4. The time now is 06:55 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy