'file2' can be any length, but it actually has to load all of file1 into memory at once! On a 32-bit computer, this may not even be possible even if you have gigs of memory due to 32-bit addresss limitations.
If the file's too large to fit in memory, you may have to process it in batches.
Code:
split -l 50000 file1 split
...will produce splitaa, splitbb, ...splitzz files of a few megs each which you can process like
Code:
for FILE in split*
do
grep -f "$FILE" file2
done > allmatches
These 3 Users Gave Thanks to Corona688 For This Post:
If there's not enough memory or address space for grep to store the whole file at once, how will awk? It might try for a while but I think it'll break down halfway through...
I am trying to use awk to find all the $2 values in file2 which is ~30MB and tab-delimited, that are between $2 and $3 in file1 which is ~2GB and tab-delimited.
I have just found out that I need to use $1 and $2 and $3 from file1 and $1 and $2of file2 must match $1 of file1 and be in the range... (6 Replies)
I need to know if file1 is a subset of file2 i.e all the contents of file1 are present in file2 or not.
Here is how i would do it.
Read line by line file1 and grep every line in file2 in a for loop. any failing grep would means that it is not a subset.
Is there a quicker or easier way... (3 Replies)
I want to print only the lines in file2 that match file1, in the same order as they appear in file 1
file1
file2
desired output:
I'm getting the lines to match
awk 'FNR==NR {a++}; FNR!=NR && a' file1 file2
but they are in sorted order, which is not what I want:
Can anyone... (4 Replies)
HI,
I would like a little help on writing a if statement.
What i have so far is:
#!/bin/bash
FILE1=path/to/file1
FILE2=path/to/file2
echo ${FILE1} ${FILE2}
if ]
then
echo file1 and file2 not found
else
echo FILE ok
fi (6 Replies)
Hi guys!
I'm trying to write something to find each line of file1 into file2, if line is found return YES, if not found return NO. The result can be written to a new file.
Can you please help me out?
FILE1 INPUT:
WATER
CAR
SNAKE
(in reality this file has about 600 lines each with a... (2 Replies)
I have very limited coding skills but I'm wondering if someone could help me with this. There are many threads about matching strings in two files, but I have no idea how to add a column from one file to another based on a matching string.
I'm looking to match column1 in file1 to the number... (3 Replies)
Hello, I am new to shell scripting and i need to create a script with the following directions and I can not figure it out.
Create a shell script called newest.bash that takes two filenames as input arguments ($1 and $2) and prints out the name of the newest file (i.e. the file with the... (1 Reply)
file1 has pgap500 500
file2 has bunch of data
cat file1 file2 > file3
cp file2 file3.dat then vi pgap500 500 onto 1st line
compare file3 and fil3.dat, they are not the same.
any idea ? the 1st line, i want to put pg500 xxx
---------- Post updated at 07:35 AM ---------- Previous... (2 Replies)
Wat does this command do?
fileA is a subset of fileB..now, i need to find the lines in fileB that are not in fileA...i.e fileA - fileB.
diff fileA fileB gives the ouput but the format looks no good....
I just need the contents alone not the line num etc. (7 Replies)
Hi, i've two files (file1, file2) i want to take value (in column1) and search in file2 if the they match print the value from file2.
this is what i have so far.
awk 'FILENAME=="file1"{ arr=$1 }
FILENAME=="file2"
{print $0}
' file1 file2 (2 Replies)
01-26-2011
