Sponsored Content
Top Forums Shell Programming and Scripting search & merg data from 3 files Post 302635745 by Scrutinizer on Sunday 6th of May 2012 05:44:54 AM
Old 05-06-2012
Assuming that the fields have to be concatenated and then sorted and that the date field should be included, you can try with ksh93 ( version h or later )

Code:
#!/bin/ksh
printed=false
sort -n file_[123] |
while read line
do
  if [ -n "$line" ]; then
    read dd tt lat lon depth mag <<< "$line" 2>&1
    epoch_dd=$(printf "%(%s)T" "$dd $tt" ) 
    if (( epoch_dd < p_epoch_dd+60 && lat < p_lat+1 && lon < p_lon+1 && mag < p_mag+1 )); then
      if ! $printed; then 
        printf "%s\n" "$p_line"
        printed=true
      fi
      printf "%s\n" "$line"
    else
      printed=false
    fi
    p_epoch_dd=$epoch_dd p_lat=$lat p_lon=$lon p_depth=$depth p_mag=$mag p_line=$line
  fi
done > outfile

Probably a perl script would be faster, although ksh93 can be surprisingly fast. Note that this version of ksh93 does not need to call external programs for the date calculations..

Last edited by Scrutinizer; 05-29-2012 at 09:29 AM.. Reason: Changed depth to mag ( column 6 instead of 5) + test if line empty
This User Gave Thanks to Scrutinizer For This Post:
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Merg files

i have: file1 contains: 123abc file2 contains: 123 abc i used: paste file1 file2 > file3, and the output looks like this: 123abc 123 abc i used: cat file3 | awk '{print $1, $2}' > file4, result: 123abc 123 my intention is to get file looks like this: 123abc123 abc when i... (9 Replies)
Discussion started by: tjmannonline
9 Replies

2. Shell Programming and Scripting

Help with complex merg of files with common field

Please help, I am new to shell Programming. I have three files each containg a unique text (key) field (e.g. ABCDEF, XCDUD as shown below), line return followed by some data of which there can be more then one instance. In addition, in some cases there may be no data but only a key field. Please... (18 Replies)
Discussion started by: gugs
18 Replies

3. UNIX for Dummies Questions & Answers

Search for & edit rows & columns in data file and pipe

Dear unix gurus, I have a data file with header information about a subject and also 3 columns of n rows of data on various items he owns. The data file looks something like this: adam peter blah blah blah blah blah blah car 01 30 200 02 31 400 03 57 121 .. .. .. .. .. .. n y... (8 Replies)
Discussion started by: tintin72
8 Replies

4. Shell Programming and Scripting

How to search & compare paragraphs between two files

Hello Guys, Greetings to All. I am stuck in my work here today while trying to comapre paragraphs between two files, I need your help on urgent basis, without your inputs I can not proceed. Kindly find some time to answer my question, I'll be grateful to you for ever. My detailed issue is as... (10 Replies)
Discussion started by: NARESH1302
10 Replies

5. Shell Programming and Scripting

Search & Replace in Multiple Files by reading a input file

Hi, I have a folder which contains multiple config.xml files and one input file, Please see the below format. Config Files format looks like :- Code: <application name="SAMPLE-ARCHIVE"> <NVPairs name="Global Variables"> <NameValuePair> ... (0 Replies)
Discussion started by: haiksuresh
0 Replies

6. Shell Programming and Scripting

Difference of two data files & writing to an outfile.

Hi Everyone, I have two files i.e. one file2 contains today's data and the other file1 contains Yesterday's data. The data in the files contains 226 columns and the data for the coulums separated by a Pipe "|" delimiter. Now, I have 4 Primary keys (coulumns) by which I have to compare file2 and... (10 Replies)
Discussion started by: filter
10 Replies

7. Shell Programming and Scripting

awk/sed to search & replace data in first column

Hi All, I need help in manipulating the data in first column in a file. The sample data looks like below, Mon Jul 18 00:32:52 EDT 2011,NULL,UAT Jul 19 2011,NULL,UAT 1] All field in the file are separated by "," 2] File is having weekly data extracted from database 3] For eg.... (8 Replies)
Discussion started by: gr8_usk
8 Replies

8. Shell Programming and Scripting

Script to search for a character in files in a Directory & remove it

Hi All, Am new to both Unix & this Forum - Need some help on a script that I am trying to write: In a Directory i have few text files which might or might not contain some text that I am trying to find. Once that text is found in any of the files, it needs to be removed from the file ... (6 Replies)
Discussion started by: rituparna_gupta
6 Replies

9. Shell Programming and Scripting

Search & Replace content of files using gsub in awk

Hi,I have 2 files master.txt & reference.txt as shown below & i require o/p as mentioned in file 3 using awk but content is not replacing properlymaster.txt:... (15 Replies)
Discussion started by: siramitsharma
15 Replies

10. Shell Programming and Scripting

Search & Replace in Multiple Files by reading a input file

I have a environment property file which contains: Input file: value1 = url1 value2 = url2 value3 = url3 and so on. I need to search all *.xml files under directory for value1 and replace it with url1. Same thing I have to do for all values mentioned in input file. I need script in unix bash... (7 Replies)
Discussion started by: Shamkamde
7 Replies
shell_builtins(1)						   User Commands						 shell_builtins(1)

NAME
shell_builtins, case, for, foreach, function, if, repeat, select, switch, until, while - shell command interpreter built-in commands DESCRIPTION
The shell command interpreters csh(1), ksh(1), ksh93(1), and sh(1) have special built-in commands. The commands case, for, foreach, func- tion, if, repeat, select, switch, until, and while are commands in the syntax recognized by the shells. They are described in the Commands section of the manual pages of the respective shells. In ksh93(1), fc, hash, stop, suspend, times, and type are aliases by default. In ksh93, the following built-ins are bound to the /bin pathname by default and are invoked if the pathname search encounters an executable command of that name in the /bin or /usr/bin directory: cat, chown, getconf, head, mkdir, rmdir, tee, uniq, and wc. The remaining commands listed in the following table are built into the shells for reasons such as efficiency or data sharing between com- mand invocations. They are described on their respective manual pages. Command Shell ----------------------------------------------------------- ++**alias csh, ksh, ksh93 bg csh, ksh, ksh93, sh +*break csh, ksh, ksh93, sh builtin ksh93 case csh, ksh, ksh93, sh cat ksh93 cd csh, ksh, ksh93, sh chdir csh, sh chown ksh93 command ksh93 +*continue csh, ksh, ksh93, sh dirs csh disown ksh93 echo csh, ksh, ksh93, sh +*eval csh, ksh, ksh93, sh +*exec csh, ksh, ksh93, sh +*exit csh, ksh, ksh93, sh ++**export ksh, ksh93, sh false ksh, ksh93 fc ksh, ksh93 fg csh, ksh, ksh93, sh for ksh, ksh93, sh foreach csh function ksh, ksh93 getconf ksh93 getopts ksh, ksh93, sh glob csh goto csh hash ksh, ksh93, sh hashstat csh head ksh93 hist ksh93 history csh if csh, ksh, ksh93, sh jobs csh, ksh, ksh93, sh kill csh, ksh, ksh93, sh let ksh, ksh93, limit csh login csh, ksh, ksh93, sh logout csh mkdir ksh93 nice csh +*newgrp ksh, ksh93, sh nohup csh notify csh onintr csh popd csh print ksh, ksh93 printf ksh93 pushd csh pwd ksh, ksh93, sh read ksh, ksh93, sh ++**readonly ksh, ksh93, sh rehash csh repeat csh +*return ksh, ksh93, sh select ksh, ksh93 +set csh, ksh, ksh93, sh setenv csh shift csh, ksh, ksh93, sh sleep ksh93 source csh stop csh, ksh, ksh93, sh suspend csh, ksh, sh switch csh tee ksh93 test ksh, ksh93, sh time csh *times ksh, ksh93, sh *+trap ksh, ksh93, sh true ksh, ksh93 type ksh, ksh93, sh ++**typeset ksh, ksh93 ulimit ksh, ksh93, sh umask csh, ksh, ksh93, sh +unalias csh, ksh, ksh93 unhash csh uniq ksh93 unlimit csh +unset csh, ksh, ksh93, sh unsetenv csh until ksh, ksh93, sh *wait csh, ksh, ksh93, sh whence ksh, ksh93 while csh, ksh, ksh93, sh Bourne Shell, sh, Special Commands Input/output redirection is now permitted for these commands. File descriptor 1 is the default output location. When Job Control is enabled, additional Special Commands are added to the shell's environment. In addition to these built-in reserved command words, sh also uses: : No effect; the command does nothing. A zero exit code is returned. .filename Read and execute commands from filename and return. The search path specified by PATH is used to find the directory contain- ing filename. C shell, csh Built-in commands are executed within the C shell. If a built-in command occurs as any component of a pipeline except the last, it is exe- cuted in a subshell. In addition to these built-in reserved command words, csh also uses: : Null command. This command is interpreted, but performs no action. Korn Shell, ksh, Special Commands Input/Output redirection is permitted. Unless otherwise indicated, the output is written on file descriptor 1 and the exit status, when there is no syntax error, is zero. Commands that are preceded by one or two * (asterisks) are treated specially in the following ways: 1. Variable assignment lists preceding the command remain in effect when the command completes. 2. I/O redirections are processed after variable assignments. 3. Errors cause a script that contains them to abort. 4. Words, following a command preceded by ** that are in the format of a variable assignment, are expanded with the same rules as a variable assignment. This means that tilde substitution is performed after the = sign and word splitting and file name genera- tion are not performed. In addition to these built-in reserved command words, ksh also uses: * : [ arg ... ] The command only expands parameters. * .file [ arg ... ] Read the complete file then execute the commands. The commands are executed in the current shell environment. The search path specified by PATH is used to find the directory containing file. If any arguments arg are specified, they become the positional parameters. Otherwise, the positional parameters are unchanged. The exit status is the exit status of the last command executed. the loop termination test. Korn Shell, ksh93, Special Commands Input/Output redirection is permitted. Unless otherwise indicated, the output is written on file descriptor 1 and the exit status, when there is no syntax error, is zero. Except for :, true, false, echo, newgrp, and login, all built-in commands accept -- to indicate end of options. They also interpret the option --man as a request to display the manual page onto standard error and -? as a help request which prints a usage message on standard error. Commands that are preceded by one or two + are treated specially in the following ways: 1. Variable assignment lists preceding the command remain in effect when the command completes. 2. I/O redirections are processed after variable assignments. 3. Errors cause a script that contains them to abort. 4. They are not valid function names. 5. Words, following a command preceded by ++ that are in the format of a variable assignment, are expanded with the same rules as a variable assignment. This means that tilde substitution is performed after the = sign and field splitting and file name genera- tion are not performed. In addition to these built-in reserved command words, ksh93 also uses: : [ arg ... ] The command only expands parameters. .name [ arg ... ] If name is a function defined with the function name reserved word syntax, the function is executed in the cur- rent environment (as if it had been defined with the name() syntax.) Otherwise if name refers to a file, the file is read in its entirety and the commands are executed in the current shell environment. The search path specified by PATH is used to find the directory containing the file. If any arguments arg are specified, they become the positional parameters while processing the . command and the original positional parameters are restored upon completion. Otherwise the positional parameters are unchanged. The exit status is the exit status of the last command executed. SEE ALSO
Intro(1), alias(1), break(1), builtin(1), cd(1), chmod(1), csh(1), disown(1), echo(1), exec(1), exit(1), find(1), getoptcvt(1), getopts(1), glob(1), hash(1), history(1), jobs(1), kill(1), ksh(1), ksh93(1), let(1), limit(1), login(1), logout(1), newgrp(1), nice(1), nohup(1), print(1), printf(1), pwd(1), read(1), readonly(1), set(1), sh(1), shift(1), sleep(1), suspend(1), test(1)test(1), test(1B), time(1), times(1), trap(1), typeset(1), umask(1), wait(1), chdir(2), chmod(2), creat(2), umask(2), getopt(3C), profile(4), environ(5) SunOS 5.11 20 Nov 2007 shell_builtins(1)
All times are GMT -4. The time now is 10:54 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy