I have a very large results file, and a list of filters
The above line works but is very slow and I'm wondering how to make it faster.
The items in my filterlist are only relevant to the first column. I don't care if any item in the filterlist matches any of the other columns for the datafile. The filterlist items may not always be found. Duplicate items will not happen in either filterlist.txt or datafile.txt
here are some simple examples to illustrate my file structure
filterlist.txt
datafile.txt
desiredoutput (I'm not worried which order they are in)
Is there some sed or awk commands I could use that would be faster? I was thinking about using awk to keep only the first column of the datafile to create a smaller file (?therefore faster for unix to work with?), then using grep to return the line numbers of my matches (storing in an array), and then using sed to display only those lines in the original datafile.txt file. But I can't find out how (of possible) sed can be used for multiple items in this case.
I know sed can print certain line numbers eg
Can sed be used with multiple values in this context? i.e. an array? could someone please provide an example for me? I often get confused with quotation marks when using variables inside other commands. Could sed be used to display lines 3,5,9,17?
Any solutions to these questions much appreciated. I'm using BASH.
---------- Post updated at 05:12 PM ---------- Previous update was at 04:54 PM ----------
I'll answer my own question by refering people to this example
look for this topic: Print lines matching value(s) in other file using awk
sorry can't post a url here as i'm new and it won't let me
I didn't have nawk on my system but I changed the command to gawk and it worked okay. Much faster than using grep -wf! (takes a couple of seconds with this solution vs hours with old method)
If anyone can still tell me if using sed is possible with multiple values, I'd really like to know.
Last edited by Franklin52; 12-20-2010 at 01:31 PM..
Dear Experts,
I have file called file1 in which i am greping a pattern after that i want to next 3 lines when that pattern is matched.
Ex:- file1
USA
UK
India
Africa
Hello
Asia
Europe
Australia
Hello
Peter
Robert
Jo
i want to next 3 lines after matching Hello... (12 Replies)
Hi folks
I am not allowed to install GNU grep on AIX.
Here my code excerpt:
grep_fatal () {
/usr/sfw/bin/gegrep -B4 -A2 "FATAL|QUEUE|SIGHUP"
}
Howto the same on AIX based machine?
from manual GNU grep
‘--after-context=num’
Print num lines of trailing context after... (4 Replies)
Hello everyone,
I have a little script below:
die "Usage infile outfile reGex" if @ARGV != 3;
($regex) = @ARGV;
open(F,$ARGV) or die "Can't open";
open(FOUT,"+>$ARGV") or die "Can't open";
while (<F>)
{
print FOUT if /$regex/.../$regex/;
}
No matter what I give $regex on the... (2 Replies)
Hi Guys,
I have file like below, I want to print all lines between test1231233 to its 10 occurrence(till line 41)
test1231233
qwe
qwe
qweq123
test1231233
qwe
qwe
qweq23
test1231233
qwe
qwe
qweq123
test1231233
qwe
qwe
qweq123131 (3 Replies)
Hello,
I have two files file 1 and file 2 each having result of a query on certain database tables and need to compare for Col1 in file1 with Col3 in file2, compare Col2 with Col4 and output the value of Col1 from File1 which is a) not present in Col3 of File2 b) value of Col2 is different from... (2 Replies)
Hi all,
I want to search for strings in file1 that can be found in file2 and print out the whole line when matching pattern is found.
I have used the below command, but this is not working for me, because it is writing out only the matching patterns from file2, not the whole line.
fgrep -o... (2 Replies)