02-18-2008
Very promising awk script ...
Thanks otheus!
Nothing quite like a one-line cryptic awk script from a guru ... with a few minor typo corrections it shows excellent promise ... trying it with the giant files and the real data is going to need to wait for tomorrow. SWEET! (I'll post back here with some timing results.)
And thanks to to the other folks who replied - this is indeed an incredible resource!
Quote:
# FNR is the number of records in the current input file - it is reset
# when the next FILE is started but NR is the number of records processed
# so far and it is not reset ... so the first line effectively creates
# an associative array out of the lines in the first input file and marks
# them with a value of "1". Then the second line effectively examines
# the lines in the second file and sets the value to zero if it is there.
# Thus - by the time it finishes, only those lines in file A but NOT in
# file B will have a value of "1". And then we print those values.
awk ' NR==FNR { A[$0]=1; next; }
{ if ($0 in A) { A[$0]=0; } }
END { for (k in A) { if (A[k]==1) { print k; } } } ' $FILE1 $FILE2
-----------------------------------------------------------
The output from a test run follows:
FILE1:
1
2
3
4
5
FILE2:
5
3
1
AND THE OUTPUT IS:
4
2
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
I need help to parse a file where there are many records, all of which are consistently separated by lines containing “^=============” and "^ End of Report".
Example:
=============
1
2
3
4
End of record
=============
1
3
4
End of record
Etc....
I only need specific lines... (5 Replies)
Discussion started by: jouuu
5 Replies
2. Shell Programming and Scripting
Hi,
I have two files. 1st file has 1 column (huge file containing ~19200000 lines) and 2nd file has 2 columns (small file containing ~6000 lines).
#################################
huge_file.txt
a
a
ab
b
##################################
small_file.txt
a 1.5
b 2.5
ab ... (4 Replies)
Discussion started by: AshwaniSharma09
4 Replies
3. Shell Programming and Scripting
hi,
i have two files.
file1.sh
echo "unix"
echo "linux"
file2.sh
echo "unix linux forums"
now the output i need is
$./file2.sh
unix linux forums (3 Replies)
Discussion started by: snreddy_gopu
3 Replies
4. Shell Programming and Scripting
I have a configuration file that contains hundreds of these chunks. Each "chunk" is the section that begins with "define service {" and ends with "}".
define service {
check_command check_proc!java
hostgroup_name
service_description ... (5 Replies)
Discussion started by: SkySmart
5 Replies
5. Shell Programming and Scripting
I have a file that looks like this:
cat includes
CORP-CRASHTEST-BU
e:\crashplan\
CORP-TEST
/usr/openv/java
/usr/openv/logs
/usr/openv/man
CORP-LABS_TEST
/usr/openv/java
/usr/openv/logs
/usr/openv/man
What I want to do is make three new files with just those selections. So the three... (4 Replies)
Discussion started by: newbie2010
4 Replies
6. Shell Programming and Scripting
I really can't figure this one out.
I have 2 files, one file is a list of hostnames and the other is a list of their corresponding IPs:
fileA:
example.com
another.org
thirdie.net
fileB:
1.1.1.1
2.2.2.2
3.3.3.3
I want to create a fileC that looks like:
example.com 1.1.1.1... (2 Replies)
Discussion started by: zstar
2 Replies
7. UNIX for Dummies Questions & Answers
Hello All,
this is my first post so I don't know if I am doing this right.
I would like to append entries from a series of strings (contained in a text file) consecutively at the end of specifically labeled lines in another file.
As an example:
- the file that contains the values to be... (3 Replies)
Discussion started by: gus74
3 Replies
8. Shell Programming and Scripting
I have a bunch of file numbers in the file 'test':
I'm trying the above command to change all the instances of "H" to "Na+" in the file testsds.pdb at the line numbers indicated in the file 'test'. I've tried the following and various similar alternatives but nothing is working:
cat test |... (3 Replies)
Discussion started by: crunchgargoyle
3 Replies
9. Shell Programming and Scripting
HI,
I have 2 text files. file1 and file2.
file1.txt (There are no duplicates in this file)
1234
3232
4343
3435
6564
6767
1213
file2.txt
1234,wq,wewe,qwqw
1234,as,dfdf,dfdf
4343,asas,sdds,dsds
6767,asas,fdfd,fdffd
I need to search each number in file1.txt in file2.txt's 1st... (6 Replies)
Discussion started by: Little
6 Replies
10. Shell Programming and Scripting
I have a file where every line includes four expressions with a caret in the middle (plus some other "words" or fields, always separated by spaces). I would like to extract from this file, all those lines such that each of the four expressions containing a caret appears in at least four different... (9 Replies)
Discussion started by: uncleMonty
9 Replies
JOIN(1) User Commands JOIN(1)
NAME
join - join lines of two files on a common field
SYNOPSIS
join [OPTION]... FILE1 FILE2
DESCRIPTION
For each pair of input lines with identical join fields, write a line to standard output. The default join field is the first, delimited
by whitespace. When FILE1 or FILE2 (not both) is -, read standard input.
-a FILENUM
print unpairable lines coming from file FILENUM, where FILENUM is 1 or 2, corresponding to FILE1 or FILE2
-e EMPTY
replace missing input fields with EMPTY
-i, --ignore-case
ignore differences in case when comparing fields
-j FIELD
equivalent to `-1 FIELD -2 FIELD'
-o FORMAT
obey FORMAT while constructing output line
-t CHAR
use CHAR as input and output field separator
-v FILENUM
like -a FILENUM, but suppress joined output lines
-1 FIELD
join on this FIELD of file 1
-2 FIELD
join on this FIELD of file 2
--check-order
check that the input is correctly sorted, even if all input lines are pairable
--nocheck-order
do not check that the input is correctly sorted
--help display this help and exit
--version
output version information and exit
Unless -t CHAR is given, leading blanks separate fields and are ignored, else fields are separated by CHAR. Any FIELD is a field number
counted from 1. FORMAT is one or more comma or blank separated specifications, each being `FILENUM.FIELD' or `0'. Default FORMAT outputs
the join field, the remaining fields from FILE1, the remaining fields from FILE2, all separated by CHAR.
Important: FILE1 and FILE2 must be sorted on the join fields. E.g., use `sort -k 1b,1' if `join' has no options. If the input is not
sorted and some lines cannot be joined, a warning message will be given.
AUTHOR
Written by Mike Haertel.
REPORTING BUGS
Report join bugs to bug-coreutils@gnu.org
GNU coreutils home page: <http://www.gnu.org/software/coreutils/>
General help using GNU software: <http://www.gnu.org/gethelp/>
COPYRIGHT
Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.
SEE ALSO
The full documentation for join is maintained as a Texinfo manual. If the info and join programs are properly installed at your site, the
command
info coreutils 'join invocation'
should give you access to the complete manual.
GNU coreutils 7.1 July 2010 JOIN(1)