Sponsored Content
Top Forums Shell Programming and Scripting Faster Line by Line String/Date Comparison of 2 Files Post 302668799 by clx on Tuesday 10th of July 2012 03:11:32 AM
Old 07-10-2012
Try ..

Code:
 awk 'NR==FNR {r[$1]=$0;d_start[$1]=$2;d_end[$1]=$3;last_col[$1]=$7;next} ($1 in r) && d_start[$1] <= $3 && d_end[$1] >= $3 { print $0, d_start[$1], d_end[$1], last_col[$1]}' lookup_file *.files

I used multiple array which could be avoided. but since your look-up file is not so-big, its worth trying.

I see you files are not at one place, those needs to be found.

Code:
find /parent/path -type f | xargs awk 'NR==FNR {r[$1]=$0;d_start[$1]=$2;d_end[$1]=$3;last_col[$1]=$7;next} ($1 in r) && d_start[$1] <= $3 && d_end[$1] >= $3 { print $0, d_start[$1], d_end[$1], last_col[$1]}' lookup_file


I don't have a real system with me so can't test the performance on BIG files.

However, There could be surely other more efficient ways which might come-up soon.
This User Gave Thanks to clx For This Post:
 

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

line by line file comparison

I have sifted through many past posts but have not found a good solution. I have a 2 files that hold a filename and its cksum value like this: FILE1 contents: 45435347589 filename1 FILE1 is from the directory structure of one host and FILE2 is generated from a rsh: host2 cksum command. I... (1 Reply)
Discussion started by: newbreed1
1 Replies

2. Homework & Coursework Questions

Date comparison with 'string date having slashes and time zone' in Bash only

1. The problem statement, all variables and given/known data: I have standard web server log file. It contains different columns (like IP address, request result code, request type etc) including a date column with the format . I have developed a log analysis command line utility that displays... (1 Reply)
Discussion started by: TariqYousaf
1 Replies

3. Shell Programming and Scripting

String comparison: All lines in file but each within line

Hi folks, I'm trying to do a bit of extra monitoring on who is sending mail through our server and as I'm sure you'll understand, the log files aren't exactly small. As a note, I'm working to sort out a solution to this myself, so I'll keep editing the post when I get a chance. Here an example... (2 Replies)
Discussion started by: beddo
2 Replies

4. Solaris

Line too long error Replace string with new line line character

I get a file which has all its content in a single row. The file contains xml data containing 3000 records, but all in a single row, making it difficult for Unix to Process the file. I decided to insert a new line character at all occurrences of a particular string in this file (say replacing... (4 Replies)
Discussion started by: ducati
4 Replies

5. Shell Programming and Scripting

how to read the contents of two files line by line and compare the line by line?

Hi All, I'm trying to figure out which are the trusted-ips and which are not using a script file.. I have a file named 'ip-list.txt' which contains some ip addresses and another file named 'trusted-ip-list.txt' which also contains some ip addresses. I want to read a line from... (4 Replies)
Discussion started by: mjavalkar
4 Replies

6. Shell Programming and Scripting

Need help in column comparison & adding extra line to files

Hi, I wanted to check whether the x,y,z coordinates of two files are equal or not. At times, when one file is converted to another suitable file extension , there are some chances that the data mismatch would happen during the conversion. In order to avoid the data misfit, i would like to... (6 Replies)
Discussion started by: b@l@ji
6 Replies

7. Shell Programming and Scripting

Comparison of fields then increment a counter reading line by line in a file

Hi, i have a scenario were i should compare a few fields from each line then increment a variable based on that. Example file 989878|8999|Y|0|Y|N|V 989878|8999|Y|0|N|N|V 989878|8999|Y|2344|Y|N|V i have 3 conditions to check and increment a variable on every line condition 1 if ( $3... (4 Replies)
Discussion started by: selvankj
4 Replies

8. Shell Programming and Scripting

Add Date string at end of line

HOWTO Append Date String (MMYYYY) to End of Line file A.txt B.txt Output: A_052014.txt B_052014.txt (9 Replies)
Discussion started by: satish1222
9 Replies

9. Shell Programming and Scripting

awk - 2 files comparison without for loop - multi-line issue

Greetings Experts, I need to handle the views created over monthly retention tables for which every new table in YYYYMMDD format, there is equivalent view created and the older table which might be dropped, the view over it has to be re-created over a dummy table so that it doesn't fail.... (2 Replies)
Discussion started by: chill3chee
2 Replies
Devel::Refcount(3pm)					User Contributed Perl Documentation				      Devel::Refcount(3pm)

NAME
"Devel::Refcount" - obtain the REFCNT value of a referent SYNOPSIS
use Devel::Refcount qw( refcount ); my $anon = []; print "Anon ARRAY $anon has " . refcount($anon) . " reference "; my $otherref = $anon; print "Anon ARRAY $anon now has " . refcount($anon) . " references "; DESCRIPTION
This module provides a single function which obtains the reference count of the object being pointed to by the passed reference value. FUNCTIONS
$count = refcount($ref) Returns the reference count of the object being pointed to by $ref. COMPARISON WITH SvREFCNT This function differs from "Devel::Peek::SvREFCNT" in that SvREFCNT() gives the reference count of the SV object itself that it is passed, whereas refcount() gives the count of the object being pointed to. This allows it to give the count of any referent (i.e. ARRAY, HASH, CODE, GLOB and Regexp types) as well. Consider the following example program: use Devel::Peek qw( SvREFCNT ); use Devel::Refcount qw( refcount ); sub printcount { my $name = shift; printf "%30s has SvREFCNT=%d, refcount=%d ", $name, SvREFCNT($_[0]), refcount($_[0]); } my $var = []; printcount 'Initially, $var', $var; my $othervar = $var; printcount 'Before CODE ref, $var', $var; printcount '$othervar', $othervar; my $code = sub { undef $var }; printcount 'After CODE ref, $var', $var; printcount '$othervar', $othervar; This produces the output Initially, $var has SvREFCNT=1, refcount=1 Before CODE ref, $var has SvREFCNT=1, refcount=2 $othervar has SvREFCNT=1, refcount=2 After CODE ref, $var has SvREFCNT=2, refcount=2 $othervar has SvREFCNT=1, refcount=2 Here, we see that SvREFCNT() counts the number of references to the SV object passed in as the scalar value - the $var or $othervar respectively, whereas refcount() counts the number of reference values that point to the referent object - the anonymous ARRAY in this case. Before the CODE reference is constructed, both $var and $othervar have SvREFCNT() of 1, as they exist only in the current lexical pad. The anonymous ARRAY has a refcount() of 2, because both $var and $othervar store a reference to it. After the CODE reference is constructed, the $var variable now has an SvREFCNT() of 2, because it also appears in the lexical pad for the new anonymous CODE block. PURE-PERL FALLBACK An XS implementation of this function is provided, and is used by default. If the XS library cannot be loaded, a fallback implementation in pure perl using the "B" module is used instead. This will behave identically, but is much slower. Rate pp xs pp 225985/s -- -66% xs 669570/s 196% -- SEE ALSO
o Test::Refcount - assert reference counts on objects AUTHOR
Paul Evans <leonerd@leonerd.org.uk> perl v5.14.2 2011-11-15 Devel::Refcount(3pm)
All times are GMT -4. The time now is 03:18 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy