Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

search & merg data from 3 files

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting search & merg data from 3 files
# 1  
Old 05-05-2012
search & merg data from 3 files
i have 3 files which contains as below (example):
Code:
yy-mm-dd  hh:mm:sec   lat          lon         depth mag

[file_1]
Code:
2006-01-01 23:17:26.80  39.8405   41.8795   2.0  3.3
2006-01-06 00:10:26.80  39.9570   41.2130   5.0  3.3
2006-01-06 06:59:02.10  39.4099   44.6065  10.0  3.7

[file_2]
Code:
2006-01-06 13:49:52.70  38.5481   40.7437   6.0  2.5
2006-01-08 20:57:12.50  39.5189   40.8876  31.5  6.1
2006-01-09 01:50:09.10  39.1322   40.3405   5.0  3.4

[file_3]
Code:
2006-01-17 07:44:15.60  38.3730   41.4610   5.6  5.0
2006-01-17 08:30:16.80  38.3405   41.5901   9.7  3.6
2006-01-18 08:50:42.70  39.5115   44.1502   3.7  3.4

i want to search 2nd, 3rd,4th & 6th fields of 3 file instantaneously and find the the values that have 1 minute different in 2nd fields, 1 degree different in 3rd and 4th fields and less than 1 unit in 6th field; then write the records that have all conditions together in output_file.
Last edited by Scrutinizer; 05-05-2012 at 06:46 PM.. Reason: code tags
# 2  
Old 05-06-2012
Conditions based on what? the next record with the respective fields (same file)?
Or you have basically one file ( after joining all the files) and then compare?

from one min difference, I guess you would like to consider date field also. The files seems to be sorted on date-time. Is it true?
# 3  
Old 05-06-2012
Assuming that the fields have to be concatenated and then sorted and that the date field should be included, you can try with ksh93 ( version h or later )

Code:
#!/bin/ksh
printed=false
sort -n file_[123] |
while read line
do
  if [ -n "$line" ]; then
    read dd tt lat lon depth mag <<< "$line" 2>&1
    epoch_dd=$(printf "%(%s)T" "$dd $tt" ) 
    if (( epoch_dd < p_epoch_dd+60 && lat < p_lat+1 && lon < p_lon+1 && mag < p_mag+1 )); then
      if ! $printed; then 
        printf "%s\n" "$p_line"
        printed=true
      fi
      printf "%s\n" "$line"
    else
      printed=false
    fi
    p_epoch_dd=$epoch_dd p_lat=$lat p_lon=$lon p_depth=$depth p_mag=$mag p_line=$line
  fi
done > outfile

Probably a perl script would be faster, although ksh93 can be surprisingly fast. Note that this version of ksh93 does not need to call external programs for the date calculations..
Last edited by Scrutinizer; 05-29-2012 at 09:29 AM.. Reason: Changed depth to mag ( column 6 instead of 5) + test if line empty
This User Gave Thanks to Scrutinizer For This Post:
oreka18
# 4  
Old 05-06-2012
Quote:
Originally Posted by anchal_khare
Conditions based on what? the next record with the respective fields (same file)?
Or you have basically one file ( after joining all the files) and then compare?

from one min difference, I guess you would like to consider date field also. The files seems to be sorted on date-time. Is it true?
yes after merging the output file must be the same as input files

---------- Post updated at 07:45 AM ---------- Previous update was at 07:39 AM ----------

Quote:
Originally Posted by Scrutinizer
Assuming that the fields have to be concatenated and then sorted and that the date field should be included, you can try with ksh93 ( version u or later )

Code:
#!/bin/ksh
printed=false
sort -n file_[123] |
while read line
do
  read dd tt lat lon depth mag <<< "$line"
  epoch_dd=$(printf "%(%s)T" "$dd $tt" ) 
  if (( epoch_dd < p_epoch_dd+60 && lat < p_lat+1 && lon < p_lon+1 && depth < p_depth+1 )); then
    if ! $printed
    then 
      printf "%s\n" "$p_line"
      printed=true
    fi
    printf "%s\n" "$line"
  else
    printed=false
  fi
  p_epoch_dd=$epoch_dd p_lat=$lat p_lon=$lon p_depth=$depth p_mag=$mag p_line=$line
done > outfile

Probably a perl script would be faster, although ksh93 can be surprisingly fast. Note that this version of ksh93 does not need to call external programs for the date calculations..

thanks but there is a problem, after running script the output file contains some blank lines! and one more thing do it possible i put ### after each section?(the result with same values that was found)
Code:
[output_file]






2006-02-18 08:30:10.80 41.1901 38.8405 9.7 3.0
2006-02-18 08:30:16.80 41.1901 38.3405 9.7 3.2
2006-02-18 08:30:16.80 41.5901 38.3405 9.7 3.6

# 5  
Old 05-06-2012
I had changed depth to mag in the mean time (col 6 instead of col 5) plus an extra if the line is empty which should get rid of the empty lines..
Last edited by Scrutinizer; 05-06-2012 at 09:42 AM..
This User Gave Thanks to Scrutinizer For This Post:
oreka18
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Search & Replace in Multiple Files by reading a input file

I have a environment property file which contains: Input file: value1 = url1 value2 = url2 value3 = url3 and so on. I need to search all *.xml files under directory for value1 and replace it with url1. Same thing I have to do for all values mentioned in input file. I need script in unix bash... (7 Replies)
Discussion started by: Shamkamde
7 Replies

2. Shell Programming and Scripting

Search & Replace content of files using gsub in awk

Hi,I have 2 files master.txt & reference.txt as shown below & i require o/p as mentioned in file 3 using awk but content is not replacing properlymaster.txt:... (15 Replies)
Discussion started by: siramitsharma
15 Replies

3. Shell Programming and Scripting

Script to search for a character in files in a Directory & remove it

Hi All, Am new to both Unix & this Forum - Need some help on a script that I am trying to write: In a Directory i have few text files which might or might not contain some text that I am trying to find. Once that text is found in any of the files, it needs to be removed from the file ... (6 Replies)
Discussion started by: rituparna_gupta
6 Replies

4. Shell Programming and Scripting

awk/sed to search & replace data in first column

Hi All, I need help in manipulating the data in first column in a file. The sample data looks like below, Mon Jul 18 00:32:52 EDT 2011,NULL,UAT Jul 19 2011,NULL,UAT 1] All field in the file are separated by "," 2] File is having weekly data extracted from database 3] For eg.... (8 Replies)
Discussion started by: gr8_usk
8 Replies

5. Shell Programming and Scripting

Difference of two data files & writing to an outfile.

Hi Everyone, I have two files i.e. one file2 contains today's data and the other file1 contains Yesterday's data. The data in the files contains 226 columns and the data for the coulums separated by a Pipe "|" delimiter. Now, I have 4 Primary keys (coulumns) by which I have to compare file2 and... (10 Replies)
Discussion started by: filter
10 Replies

6. Shell Programming and Scripting

Search & Replace in Multiple Files by reading a input file

Hi, I have a folder which contains multiple config.xml files and one input file, Please see the below format. Config Files format looks like :- Code: <application name="SAMPLE-ARCHIVE"> <NVPairs name="Global Variables"> <NameValuePair> ... (0 Replies)
Discussion started by: haiksuresh
0 Replies

7. Shell Programming and Scripting

How to search & compare paragraphs between two files

Hello Guys, Greetings to All. I am stuck in my work here today while trying to comapre paragraphs between two files, I need your help on urgent basis, without your inputs I can not proceed. Kindly find some time to answer my question, I'll be grateful to you for ever. My detailed issue is as... (10 Replies)
Discussion started by: NARESH1302
10 Replies

8. UNIX for Dummies Questions & Answers

Search for & edit rows & columns in data file and pipe

Dear unix gurus, I have a data file with header information about a subject and also 3 columns of n rows of data on various items he owns. The data file looks something like this: adam peter blah blah blah blah blah blah car 01 30 200 02 31 400 03 57 121 .. .. .. .. .. .. n y... (8 Replies)
Discussion started by: tintin72
8 Replies

9. Shell Programming and Scripting

Help with complex merg of files with common field

Please help, I am new to shell Programming. I have three files each containg a unique text (key) field (e.g. ABCDEF, XCDUD as shown below), line return followed by some data of which there can be more then one instance. In addition, in some cases there may be no data but only a key field. Please... (18 Replies)
Discussion started by: gugs
18 Replies

10. UNIX for Dummies Questions & Answers

Merg files

i have: file1 contains: 123abc file2 contains: 123 abc i used: paste file1 file2 > file3, and the output looks like this: 123abc 123 abc i used: cat file3 | awk '{print $1, $2}' > file4, result: 123abc 123 my intention is to get file looks like this: 123abc123 abc when i... (9 Replies)
Discussion started by: tjmannonline
9 Replies
Login or Register to Ask a Question