Extract header data from one file and combine it with data from another file | Unix Linux Forums | Shell Programming and Scripting

  Go Back    


Shell Programming and Scripting Post questions about KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and OTHER shell scripts and shell scripting languages here.

Extract header data from one file and combine it with data from another file

Shell Programming and Scripting


Closed Thread    
 
Thread Tools Search this Thread Display Modes
    #1  
Old 12-03-2012
nex_asp nex_asp is offline
Banned
 
Join Date: Dec 2012
Last Activity: 21 May 2013, 2:14 AM EDT
Location: India
Posts: 70
Thanks: 20
Thanked 0 Times in 0 Posts
Extract header data from one file and combine it with data from another file

Hi, Great minds, I have some files, in fact header files, of CTD profiler, I tried a lot C programming, could not get output as I was expected, because my programming skills are very poor, finally, joined unix forum with the hope that, I may get what I want, from you people,

Here I have attached some text files which defines output, which actually I am looking for,,,

Thanks,

Cheers..
Attached Files
File Type: txt header.txt (316 Bytes, 30 views)
File Type: txt main.txt (552 Bytes, 24 views)
File Type: txt output_requirement.txt (2.0 KB, 22 views)

Last edited by nex_asp; 12-03-2012 at 01:28 AM.. Reason: to do code tag
Sponsored Links
    #2  
Old 12-03-2012
pamu pamu is offline
Registered User
 
Join Date: Mar 2012
Last Activity: 7 November 2014, 10:50 AM EST
Posts: 1,650
Thanks: 58
Thanked 478 Times in 474 Posts
Try


Code:
awk -F ":" 'NR==FNR{gsub("*","",$0);if($0 ~ /UpLoad Time/){split($0,P,"=")}else if(P[1]){A[++x]=$1;B[x]=$2};next}{if(FNR==1){printf P[1];FNV=P[2];
for(i=1;i<=x;i++){printf "\t%s", A[i];FNV=FNV"\t"B[i]};print $0}
if(FNR>1){print FNV"\t"$0}}' file2 file1

Sponsored Links
    #3  
Old 12-03-2012
nex_asp nex_asp is offline
Banned
 
Join Date: Dec 2012
Last Activity: 21 May 2013, 2:14 AM EDT
Location: India
Posts: 70
Thanks: 20
Thanked 0 Times in 0 Posts
Your code is working

How to I apply for loop for this
file 2 is .hdr
file 1 is .asc
for file in *.txt; do
for file in *.hdr; do
---
---
---
done
whether it works ..don't know much about shell scripting
    #4  
Old 12-03-2012
fpmurphy's Avatar
fpmurphy fpmurphy is offline Forum Staff  
who?
 
Join Date: Dec 2003
Last Activity: 22 November 2014, 10:34 AM EST
Location: /dev/ph
Posts: 4,834
Thanks: 64
Thanked 433 Times in 401 Posts
Use a simple for loop.

Code:
for file in *.hdr
do
    awk ...   $file
done

Sponsored Links
    #5  
Old 12-03-2012
nex_asp nex_asp is offline
Banned
 
Join Date: Dec 2012
Last Activity: 21 May 2013, 2:14 AM EDT
Location: India
Posts: 70
Thanks: 20
Thanked 0 Times in 0 Posts
Its not easy it seems..

from 2nd post,

see http://www.unix.com/members/31298.html

file base of asc and hdr must match then only it should add latitude, longitude, etc...to main file, solution given by pamu is working fine, but to process number of files I have to include for loop, so I had asked...if anybody knows any good book or link to learn shell scripting please put link, I am very poor, in shell scripting..I know only C...

---------- Post updated at 03:28 AM ---------- Previous update was at 03:07 AM ----------

actually header file is having more information when I tried with another file I got wrong result....

Pamu's script reads, full header file, it should not read actually it has to filter out only particular information only...

they are ship,cruise,station, lat,long,DEPTH(SONIC),DEPTH(CAST),sst, and cast


Code:
* Sea-Bird SBE19plus Data File: 
* FileName = D:\SK-296_CTD_portable\Hex_data\n12a001001.hex 
* Software Version 1.59 
* Temperature SN =  6336 
* Conductivity SN =  6336 
* System UpLoad Time = Jul 14 2012 18:12:59 
** SHIP : India-ship 
** CRUISE : 500
** STATION : n12a001 
** LAT : 21 12.1985 N 
** LONG : 89 27.016 E 
** DEPTH(SONIC) : 93 m 
** DEPTH(CAST) : 77.7 m 
** SST : 29 degC 
* ds 
* SBE 19plus V 2.3  SERIAL NO. 6336    14 Jul 2012 17:58:03 
* vbatt = 12.2, vlith =  8.6, ioper =  61.8 ma, ipump =  39.9 ma, 
* iext01 =   4.6 ma, iext2345 =  27.1 ma 
* status = not logging 
* number of scans to average = 1 
* samples = 103375, free = 3767104, casts = 11 
* mode = profile, minimum cond freq = 2700, pump delay = 60 sec 
* autorun = no, ignore magnetic switch = no 
* battery type = alkaline, battery cutoff =  7.5 volts 
* pressure sensor = strain gauge, range = 5076.0 
* SBE 38 = no, WETLABS = no, OPTODE = no, Gas Tension Device = no 
* Ext Volt 0 = yes, Ext Volt 1 = no 
* Ext Volt 2 = yes, Ext Volt 3 = no 
* Ext Volt 4 = yes, Ext Volt 5 = no 
* echo characters = yes 
* output format = raw HEX 
 
* S> 
* 19plus 
* dh 
* cast   1 13 Jul 2012 07:31:59 samples 1 to 8619, avg = 1, stop = mag switch

-----
still some texts are there...not including...

Pls help....

Last edited by Scott; 12-03-2012 at 03:15 PM.. Reason: Code tags
Sponsored Links
    #6  
Old 12-03-2012
pamu pamu is offline
Registered User
 
Join Date: Mar 2012
Last Activity: 7 November 2014, 10:50 AM EST
Posts: 1,650
Thanks: 58
Thanked 478 Times in 474 Posts
You can do something like this...


Code:
if ($0 ~ /SHIP/){A[++x]=$1;B[x]=$2}
if ($0 ~ /CRUISE/){A[++x]=$1;B[x]=$2}

Use this to implement your other headers also...

pamu
Sponsored Links
    #7  
Old 12-03-2012
Don Cragun's Avatar
Don Cragun Don Cragun is offline Forum Staff  
Moderator
 
Join Date: Jul 2012
Last Activity: 22 November 2014, 4:49 PM EST
Location: San Jose, CA, USA
Posts: 5,082
Thanks: 195
Thanked 1,701 Times in 1,444 Posts
Try saving the following script in a file (I used the name runner while testing it, but choose anything you like). Make it executable using chmod +x file_name and run it using ./runner header.txt main.txt to test it with the sample files you provided. The output this awk script produces match the output you requested (as long as you remove the trailing empty line at the end of main.txt) except that there are no spaces before tabs in the header line:
Code:
#!/bin/ksh
# Usage: runner hdr main [hdr main]...
if [ $# -lt 2 ] || [ $(($# % 2)) -ne 0 ]
then    printf "%s: Odd number of operands or less than two operands:\n" $0 >&2
        printf "Usage: %s hdr main [hdr main]...\n" "$0" >&2
        exit 1
fi
while [ $# -ge 2 ]
do      hf="$1"
        mf="$2"
        shift 2
        awk 'BEGIN{
                m["Jan"] = 1; m["Feb"] = 2;  m["Mar"] = 3;  m["Apr"] = 4
                m["May"] = 5; m["Jun"] = 6;  m["Jul"] = 7;  m["Aug"] = 8
                m["Sep"] = 9; m["Oct"] = 10; m["Nov"] = 11; m["Dec"] = 12
                FS = "[:=] "
                printf("System UpLoad\tTime\tSHIP\tCRUISE\tSTATION\tLAT\t%s",
                        "LON\tNATURE_OF_PR\tPROJ_NO\tINSTITUTE_CD\tST_DEPTH\n")
        }
        FNR>1 && FNR!=NR{
                printf("%s\t%s\t%s\t%s\t%s\n", hdr, $1, $2, $3, $4)
                next
        }       
        FNR==NR && /System UpLoad/{
                split($2, a, "  *")
                dt=sprintf("%02d/%02d/%02d", a[2], m[a[1]], a[3] % 100)
                tm=a[4]
                next
        }
        FNR==NR && /SHIP/{sh = $2;next}
        FNR==NR && /CRUISE/{cr = $2;next}
        FNR==NR && /STATION/{st = $2;next}
        FNR==NR && /LAT/{la = $2;next}
        FNR==NR && /LONG/{lo = $2;next}
        FNR==1 && NR>1{
                FS = "\t\t*"
                hdr = sprintf("%s\t%s\t%s\t%s\t%s\t%s\t%s",
                        dt, tm, sh, cr, st, la, lo)
                next;
        }' "$hf" "$mf"
done

You can invoke this script with any even number of files >=2 where the 1st file in each pair is in the format of header.txt and the 2nd file in each pair is in the format of main.txt.
The Following User Says Thank You to Don Cragun For This Useful Post:
nex_asp (12-03-2012)
Sponsored Links
Closed Thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
get data from files combine them to a file gc_sw Shell Programming and Scripting 4 12-21-2010 09:58 AM
Extract specific content from data and rename its header problem asking patrick87 Shell Programming and Scripting 10 03-25-2010 03:47 AM
Extract data based on match against one column data from a long list data patrick87 Shell Programming and Scripting 12 11-17-2009 04:27 AM
insert a header in a huge data file without using an intermediate file deepaktanna Shell Programming and Scripting 10 02-23-2009 03:38 PM
to find header in Mp3 file and retrieve data shashi Programming 2 09-12-2008 04:03 AM



All times are GMT -4. The time now is 11:53 AM.