Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Faster Line by Line String/Date Comparison of 2 Files
Top Forums Shell Programming and Scripting Faster Line by Line String/Date Comparison of 2 Files Post 302668799 by clx on Tuesday 10th of July 2012 03:11:32 AM
Old 07-10-2012
Try ..

Code:
 awk 'NR==FNR {r[$1]=$0;d_start[$1]=$2;d_end[$1]=$3;last_col[$1]=$7;next} ($1 in r) && d_start[$1] <= $3 && d_end[$1] >= $3 { print $0, d_start[$1], d_end[$1], last_col[$1]}' lookup_file *.files

I used multiple array which could be avoided. but since your look-up file is not so-big, its worth trying.

I see you files are not at one place, those needs to be found.

Code:
find /parent/path -type f | xargs awk 'NR==FNR {r[$1]=$0;d_start[$1]=$2;d_end[$1]=$3;last_col[$1]=$7;next} ($1 in r) && d_start[$1] <= $3 && d_end[$1] >= $3 { print $0, d_start[$1], d_end[$1], last_col[$1]}' lookup_file


I don't have a real system with me so can't test the performance on BIG files.

However, There could be surely other more efficient ways which might come-up soon.
This User Gave Thanks to clx For This Post:
agentgrecko
 

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

line by line file comparison

I have sifted through many past posts but have not found a good solution. I have a 2 files that hold a filename and its cksum value like this: FILE1 contents: 45435347589 filename1 FILE1 is from the directory structure of one host and FILE2 is generated from a rsh: host2 cksum command. I... (1 Reply)
Discussion started by: newbreed1
1 Replies

2. Homework & Coursework Questions

Date comparison with 'string date having slashes and time zone' in Bash only

1. The problem statement, all variables and given/known data: I have standard web server log file. It contains different columns (like IP address, request result code, request type etc) including a date column with the format . I have developed a log analysis command line utility that displays... (1 Reply)
Discussion started by: TariqYousaf
1 Replies

3. Shell Programming and Scripting

String comparison: All lines in file but each within line

Hi folks, I'm trying to do a bit of extra monitoring on who is sending mail through our server and as I'm sure you'll understand, the log files aren't exactly small. As a note, I'm working to sort out a solution to this myself, so I'll keep editing the post when I get a chance. Here an example... (2 Replies)
Discussion started by: beddo
2 Replies

4. Solaris

Line too long error Replace string with new line line character

I get a file which has all its content in a single row. The file contains xml data containing 3000 records, but all in a single row, making it difficult for Unix to Process the file. I decided to insert a new line character at all occurrences of a particular string in this file (say replacing... (4 Replies)
Discussion started by: ducati
4 Replies

5. Shell Programming and Scripting

how to read the contents of two files line by line and compare the line by line?

Hi All, I'm trying to figure out which are the trusted-ips and which are not using a script file.. I have a file named 'ip-list.txt' which contains some ip addresses and another file named 'trusted-ip-list.txt' which also contains some ip addresses. I want to read a line from... (4 Replies)
Discussion started by: mjavalkar
4 Replies

6. Shell Programming and Scripting

Need help in column comparison & adding extra line to files

Hi, I wanted to check whether the x,y,z coordinates of two files are equal or not. At times, when one file is converted to another suitable file extension , there are some chances that the data mismatch would happen during the conversion. In order to avoid the data misfit, i would like to... (6 Replies)
Discussion started by: b@l@ji
6 Replies

7. Shell Programming and Scripting

Comparison of fields then increment a counter reading line by line in a file

Hi, i have a scenario were i should compare a few fields from each line then increment a variable based on that. Example file 989878|8999|Y|0|Y|N|V 989878|8999|Y|0|N|N|V 989878|8999|Y|2344|Y|N|V i have 3 conditions to check and increment a variable on every line condition 1 if ( $3... (4 Replies)
Discussion started by: selvankj
4 Replies

8. Shell Programming and Scripting

Add Date string at end of line

HOWTO Append Date String (MMYYYY) to End of Line file A.txt B.txt Output: A_052014.txt B_052014.txt (9 Replies)
Discussion started by: satish1222
9 Replies

9. Shell Programming and Scripting

awk - 2 files comparison without for loop - multi-line issue

Greetings Experts, I need to handle the views created over monthly retention tables for which every new table in YYYYMMDD format, there is equivalent view created and the older table which might be dropped, the view over it has to be re-created over a dummy table so that it doesn't fail.... (2 Replies)
Discussion started by: chill3chee
2 Replies

LEARN ABOUT OSX

locale::codes::langext

Locale::Codes::LangExt(3pm)				 Perl Programmers Reference Guide			       Locale::Codes::LangExt(3pm)

NAME

       Locale::Codes::LangExt - standard codes for language extension identification

SYNOPSIS

	  use Locale::Codes::LangExt;

	  $lext = code2langext('acm');		       # $lext gets 'Mesopotamian Arabic'
	  $code = langext2code('Mesopotamian Arabic'); # $code gets 'acm'

	  @codes   = all_langext_codes();
	  @names   = all_langext_names();

DESCRIPTION

       The "Locale::Codes::LangExt" module provides access to standard codes used for identifying language extensions, such as those as defined in
       the IANA language registry.

       Most of the routines take an optional additional argument which specifies the code set to use. If not specified, the default IANA language
       registry codes will be used.

SUPPORTED CODE SETS

       There are several different code sets you can use for identifying language extensions. A code set may be specified using either a name, or
       a constant that is automatically exported by this module.

       For example, the two are equivalent:

	  $lext = code2langext('acm','alpha');
	  $lext = code2langext('acm',LOCALE_LANGEXT_ALPHA);

       The codesets currently supported are:

       alpha
	   This is the set of three-letter (lowercase) codes from the IANA language registry, such as 'acm' for Mesopotamian Arabic.

	   This is the default code set.

ROUTINES

       code2langext ( CODE [,CODESET] )
       langext2code ( NAME [,CODESET] )
       langext_code2code ( CODE ,CODESET ,CODESET2 )
       all_langext_codes ( [CODESET] )
       all_langext_names ( [CODESET] )
       Locale::Codes::LangExt::rename_langext  ( CODE ,NEW_NAME [,CODESET] )
       Locale::Codes::LangExt::add_langext  ( CODE ,NAME [,CODESET] )
       Locale::Codes::LangExt::delete_langext  ( CODE [,CODESET] )
       Locale::Codes::LangExt::add_langext_alias  ( NAME ,NEW_NAME )
       Locale::Codes::LangExt::delete_langext_alias  ( NAME )
       Locale::Codes::LangExt::rename_langext_code  ( CODE ,NEW_CODE [,CODESET] )
       Locale::Codes::LangExt::add_langext_code_alias  ( CODE ,NEW_CODE [,CODESET] )
       Locale::Codes::LangExt::delete_langext_code_alias  ( CODE [,CODESET] )
	   These routines are all documented in the Locale::Codes::API man page.

SEE ALSO

       Locale::Codes
	   The Locale-Codes distribution.

       Locale::Codes::API
	   The list of functions supported by this module.

       http://www.iana.org/assignments/language-subtag-registry
	   The IANA language subtag registry.

AUTHOR

       See Locale::Codes for full author history.

       Currently maintained by Sullivan Beck (sbeck@cpan.org).

COPYRIGHT

	  Copyright (c) 2011-2012 Sullivan Beck

       This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

perl v5.16.2							    2012-10-11					       Locale::Codes::LangExt(3pm)
All times are GMT -4. The time now is 11:38 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy