Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: search & merg data from 3 files
Top Forums Shell Programming and Scripting search & merg data from 3 files Post 302635745 by Scrutinizer on Sunday 6th of May 2012 05:44:54 AM
Old 05-06-2012
Assuming that the fields have to be concatenated and then sorted and that the date field should be included, you can try with ksh93 ( version h or later )

Code:
#!/bin/ksh
printed=false
sort -n file_[123] |
while read line
do
  if [ -n "$line" ]; then
    read dd tt lat lon depth mag <<< "$line" 2>&1
    epoch_dd=$(printf "%(%s)T" "$dd $tt" ) 
    if (( epoch_dd < p_epoch_dd+60 && lat < p_lat+1 && lon < p_lon+1 && mag < p_mag+1 )); then
      if ! $printed; then 
        printf "%s\n" "$p_line"
        printed=true
      fi
      printf "%s\n" "$line"
    else
      printed=false
    fi
    p_epoch_dd=$epoch_dd p_lat=$lat p_lon=$lon p_depth=$depth p_mag=$mag p_line=$line
  fi
done > outfile

Probably a perl script would be faster, although ksh93 can be surprisingly fast. Note that this version of ksh93 does not need to call external programs for the date calculations..
Last edited by Scrutinizer; 05-29-2012 at 09:29 AM.. Reason: Changed depth to mag ( column 6 instead of 5) + test if line empty
This User Gave Thanks to Scrutinizer For This Post:
oreka18
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Merg files

i have: file1 contains: 123abc file2 contains: 123 abc i used: paste file1 file2 > file3, and the output looks like this: 123abc 123 abc i used: cat file3 | awk '{print $1, $2}' > file4, result: 123abc 123 my intention is to get file looks like this: 123abc123 abc when i... (9 Replies)
Discussion started by: tjmannonline
9 Replies

2. Shell Programming and Scripting

Help with complex merg of files with common field

Please help, I am new to shell Programming. I have three files each containg a unique text (key) field (e.g. ABCDEF, XCDUD as shown below), line return followed by some data of which there can be more then one instance. In addition, in some cases there may be no data but only a key field. Please... (18 Replies)
Discussion started by: gugs
18 Replies

3. UNIX for Dummies Questions & Answers

Search for & edit rows & columns in data file and pipe

Dear unix gurus, I have a data file with header information about a subject and also 3 columns of n rows of data on various items he owns. The data file looks something like this: adam peter blah blah blah blah blah blah car 01 30 200 02 31 400 03 57 121 .. .. .. .. .. .. n y... (8 Replies)
Discussion started by: tintin72
8 Replies

4. Shell Programming and Scripting

How to search & compare paragraphs between two files

Hello Guys, Greetings to All. I am stuck in my work here today while trying to comapre paragraphs between two files, I need your help on urgent basis, without your inputs I can not proceed. Kindly find some time to answer my question, I'll be grateful to you for ever. My detailed issue is as... (10 Replies)
Discussion started by: NARESH1302
10 Replies

5. Shell Programming and Scripting

Search & Replace in Multiple Files by reading a input file

Hi, I have a folder which contains multiple config.xml files and one input file, Please see the below format. Config Files format looks like :- Code: <application name="SAMPLE-ARCHIVE"> <NVPairs name="Global Variables"> <NameValuePair> ... (0 Replies)
Discussion started by: haiksuresh
0 Replies

6. Shell Programming and Scripting

Difference of two data files & writing to an outfile.

Hi Everyone, I have two files i.e. one file2 contains today's data and the other file1 contains Yesterday's data. The data in the files contains 226 columns and the data for the coulums separated by a Pipe "|" delimiter. Now, I have 4 Primary keys (coulumns) by which I have to compare file2 and... (10 Replies)
Discussion started by: filter
10 Replies

7. Shell Programming and Scripting

awk/sed to search & replace data in first column

Hi All, I need help in manipulating the data in first column in a file. The sample data looks like below, Mon Jul 18 00:32:52 EDT 2011,NULL,UAT Jul 19 2011,NULL,UAT 1] All field in the file are separated by "," 2] File is having weekly data extracted from database 3] For eg.... (8 Replies)
Discussion started by: gr8_usk
8 Replies

8. Shell Programming and Scripting

Script to search for a character in files in a Directory & remove it

Hi All, Am new to both Unix & this Forum - Need some help on a script that I am trying to write: In a Directory i have few text files which might or might not contain some text that I am trying to find. Once that text is found in any of the files, it needs to be removed from the file ... (6 Replies)
Discussion started by: rituparna_gupta
6 Replies

9. Shell Programming and Scripting

Search & Replace content of files using gsub in awk

Hi,I have 2 files master.txt & reference.txt as shown below & i require o/p as mentioned in file 3 using awk but content is not replacing properlymaster.txt:... (15 Replies)
Discussion started by: siramitsharma
15 Replies

10. Shell Programming and Scripting

Search & Replace in Multiple Files by reading a input file

I have a environment property file which contains: Input file: value1 = url1 value2 = url2 value3 = url3 and so on. I need to search all *.xml files under directory for value1 and replace it with url1. Same thing I have to do for all values mentioned in input file. I need script in unix bash... (7 Replies)
Discussion started by: Shamkamde
7 Replies

LEARN ABOUT LINUX

shcomp

SHCOMP(1)						    User Environment Utilities							 SHCOMP(1)

NAME

       shcomp - compile a ksh93 shell script

SYNOPSIS

       shcomp [ options ] [infile] [outfile]]

ORIGIN OF THIS MAN PAGE

       This man page was written by Christophe Martin <Schplurtz@free.fr> for the Debian GNU/Linux system. It is based on "shcomp --man" output.

DESCRIPTION

       Unless  -D  is  specified,  shcomp takes a shell script, infile, and creates a binary format file, outfile, that ksh93 can read and execute
       with the same effect as the original script.

       Since aliases are processed as the script is read, alias definitions whose value requires variable expansion will not work correctly.

       If -D is specified, all double quoted strings that are preceded by $ are output. These are the messages	that  need  to	be  translated	to
       locale specific versions for internationalization.

       If  outfile is omitted, then the results will be written to standard output.  If infile is also omitted, the shell script will be read from
       standard input.

OPTIONS

       -D, --dictionary
	      Generate a list of strings that need to be placed in a message catalog for internationalization.

       -n, --noexec
	      Displays warning messages for obsolete or non-conforming constructs.

       -v, --verbose
	      Displays input from infile onto standard error as it reads it.

       --man  Displays this man page on standard error and exits.

       --help Shows a short help message and exits.

EXIT STATUS

       0      Successful completion.

       >0     An error occurred.

DEBIAN WARNING

       On a Debian GNU/Linux system, more than one package may provide ksh. the output of shcomp can only be used by ksh93.  Any other ksh will be
       unable to run the compiled script.

EXAMPLE

       ( echo '#! /bin/ksh93' ; shcomp myscript ) > myscript.bin &&
       chmod 755 myscript.bin
       ./myscript.bin

SEE ALSO

       ksh(1)

IMPLEMENTATION

       version	       shcomp (AT&T Labs Research) 2003-03-02
       author	       David Korn <dgk@research.att.com>
       copyright       Copyright (c) 1982-2005 AT&T Corp.
       license	       http://www.opensource.org/licenses/cpl1.0.txt

KSH93								  Fri Jun 3 2005							 SHCOMP(1)
All times are GMT -4. The time now is 09:42 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy