Extract header data from one file and combine it with data from another file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Extract header data from one file and combine it with data from another file
# 8  
Old 12-03-2012
Yes your script is working

Hi
Don Cragun

I tried your script with new file its reading header file as well as main file, I am happy with your solution

How do I save it as file

How to give command..
# 9  
Old 12-03-2012
Quote:
Originally Posted by nex_asp
Hi
Don Cragun

I tried your script with new file its reading header file as well as main file, I am happy with your solution

How do I save it as file

How to give command..
Hi.
How do you save what as a file? It isn't clear to what "it" refers in you question?

If you mean that you want to save the output produced by the script for all input files as a single output file try:
Code:
./runner header.txt main.txt > output.txt

If you mean that you want the output produced by each set of input files to be stored in different output files, we can modify the script to take sets of three files (header file, main file, and output file) as operands.
# 10  
Old 01-19-2013
Sir as I got one small problem in appending header please help.

I modified as per my requirement
Code:
#!/bin/ksh
# Usage: runner hdr main [hdr main]...
if [ $# -lt 2 ] || [ $(($# % 2)) -ne 0 ]
then    printf "%s: Odd number of operands or less than two operands:\n" $0 >&2
        printf "Usage: %s hdr main [hdr main]...\n" "$0" >&2
        exit 1
fi
while [ $# -ge 2 ]
do      hf="$1"
        mf="$2"
        shift 2
        awk    'BEGIN{
                m["Jan"] = 1; m["Feb"] = 2;  m["Mar"] = 3;  m["Apr"] = 4
                m["May"] = 5; m["Jun"] = 6;  m["Jul"] = 7;  m["Aug"] = 8
                m["Sep"] = 9; m["Oct"] = 10; m["Nov"] = 11; m["Dec"] = 12
                FS = "[:=] "
  printf("Date\tTime\tLatitude\t%s","Longitude\tPrDM\tT090C\tC0S/m\tDepSM\tSal00\tDensity00\tSigma-é00\tOxsolML/L\tSvCM\n")
        }

    FNR>1 && FNR!=NR{printf("%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n", hdr, $1, $2, $3, $4, $5, $6, $7, $8, $9);next}
               
        FNR==NR && /System UpLoad/{split($2, a, "  *");dt=sprintf("%02d/%02d/%02d", a[2], m[a[1]], a[3] % 100);tm=a[4];next}

    FNR==NR && /NMEA Latitude/{lat = $2;next}

        FNR==NR && /NMEA Longitude/{lon = $2;next}

        FNR==1  && NR>1{FS = "\t\t";hdr = sprintf("%s\t%s\t%s\t%s",dt, tm, lat, lon)
    next
        }' "$hf" "$mf"
done

I attached data and header file also but latitude and longitude not coming, if I make
Code:
 FNR==NR && /NMEA Longitude/{lon = $2;next}

to
Code:
FNR==NR && /^NMEA Longitude/{lon = $2;next}

I can see time otherwise not

please tell me where I am doing wrong also
# 11  
Old 01-19-2013
If you need this in a hurry (as indicated by the three private messages you sent me Smilie ), you need to find someone else to help you. I just found out that I'm going to be tied up most of the rest of this weekend and at least until Wednesday afternoon next week.

I did look at your data enough to know that one of your problems is that the files you attached to message #10 in this thread are all terminated by carriage-return and newline characters instead of just a newline character.

If you change:
Code:
    FNR>1 && FNR!=NR{printf("%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n", hdr, $1, $2, $3, $4, $5, $6, $7, $8, $9);next}

in your script to:
Code:
                gsub(/]r/, "")
    FNR>1 && FNR!=NR{printf("%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n", hdr, $1, $2, $3, $4, $5, $6, $7, $8, $9);next}

that may get rid of the carriage return characters that are causing the latitude and longitude data to overwrite earlier fields in the output.

Another significant change is that in your first main.txt file, you had tabs separating fields. In your new main.txt file, there are no tab characters so all of your input lines in that file probably have only one field when you set FS to "\t\t".

Good luck...
# 12  
Old 01-20-2013
Code:
gsub(/]r/, "")

Sir I didn't find any changes from this sir

Last edited by Scrutinizer; 01-20-2013 at 06:38 AM.. Reason: code tags
# 13  
Old 01-20-2013
Try gsub(/\r/, "") instead (typo?).

And, as Don Cragun stated, your files are contaminated with windows style characters, repeatedly. Why don't you clean them before working on them or, even more important, posting them here? And, see same link, post consistent samples.

Last edited by RudiC; 01-20-2013 at 06:18 AM..
This User Gave Thanks to RudiC For This Post:
# 14  
Old 01-20-2013
Thanks but when you take output, say
Code:
sh script.sh file1 file2 > output

full header also coming

---------- Post updated at 05:25 AM ---------- Previous update was at 05:12 AM ----------

Code:
awk '{for(i=4;i<=NF;i++){printf "\t""%s", $i}printf "\n"}

' I tried to make pure tab separated from this but still having problem, I understood what you said, before using unix what I need to do, that cleaning part is complex for me.. can you please tell, how you clean windows style char

---------- Post updated at 05:48 AM ---------- Previous update was at 05:25 AM ----------

sometimes I use this command to convert dos to unix, but this I won't trust because sometimes it results wrong..

if any modification I have to do means please tell me..

Code:
cat dosfile | tr -d '\r' > unixfile && mv unixfile dosfile

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Bash to verify and validate file header and data type

The below bash is a file validation check executed that will verify the correct header count of 10 and the correct data type in each field of the tab-delimited file. The key has the data type of each field in it. My real data has 58 headers in it but only the header and next row need to be... (6 Replies)
Discussion started by: cmccabe
6 Replies

2. Shell Programming and Scripting

Combine data out of 3 files into one new file

Hi, How can I combine the data of of three files into one new file? I try to give as much informations as possible. The three existing files are called file1 file2 and file3 the new file should named output_combined. The size of the files will be around 900 words/lines each .. but always... (5 Replies)
Discussion started by: MyMemberName
5 Replies

3. Shell Programming and Scripting

How to combine data from 2 file ? How to print ?

Dear friends, I am just trying to print data from 2 file,namely file_1.txt and file_1.dat (specific column of ROW 1) file_1.txt 12 13 14 15 99 AMC 69 36 89 12 13 14 15 99 AMC 69 84 -12 12 13 14 ... (6 Replies)
Discussion started by: nex_asp
6 Replies

4. Shell Programming and Scripting

Extract data from XML file and write in CSV file

Hi friend i have input as following XML file <?xml version="1.0"?> <Document xmlns="urn:iso:std:iso:20022:tech:xsd:camt.054.001.02"> <BkToCstmrDbtCdtNtfctn> <GrpHdr><MsgId>LBP-RDJ-TE000000-130042430010001001</MsgId><CreDtTm>2013-01-04T03:21:30</CreDtTm></GrpHdr>... (3 Replies)
Discussion started by: mohan sharma
3 Replies

5. Shell Programming and Scripting

get data from files combine them to a file

hi people; this is my file1.txt:192.168.1.1 192.168.1.2 192.168.1.3 192.168.1.4 ... this is my file2.txt:portnames usernames maxusercap ... i want to write to file3.txt:l ./getports 192.168.1.1 'get all;l+;get . portnames;l-' l ./getports 192.168.1.1 'get all;l+;get . usernames;l-'... (4 Replies)
Discussion started by: gc_sw
4 Replies

6. Shell Programming and Scripting

Extract data from an XML file & write into a CSV file

Hi All, I am having an XML tag like: <detail sim_ser_no_1="898407109001000090" imsi_1="452070001000090"> <security>ADM1=????</security> <security>PIN1=????</security> <security>PIN2=????</security> ... (2 Replies)
Discussion started by: ss_ss
2 Replies

7. Shell Programming and Scripting

Extract specific content from data and rename its header problem asking

Input file 1: >pattern_5 GAATTCGTTCATGTAGGTTGASDASFGDSGRTYRYGHDGSDFGSDGGDSGSDGSDFGSDF ATTTAATTATGATTCATACGTCATATGTTATTATTCAATCGTATAAAATTATGTGACCTT SDFSDGSDFKSDAFLKJASLFJASKLFSJAKJFHASJKFHASJKFHASJKFHSJAKFHAW >pattern_1 AAGTCTTAAGATATCACCGTCGATTAGGTTTATACAGCTTTTGTGTTATTTAAATTTGAC... (10 Replies)
Discussion started by: patrick87
10 Replies

8. Shell Programming and Scripting

Extract data based on match against one column data from a long list data

My input file: data_5 Ali 422 2.00E-45 102/253 140/253 24 data_3 Abu 202 60.00E-45 12/23 140/23 28 data_1 Ahmad 256 7.00E-45 120/235 140/235 22 data_4 Aman 365 8.00E-45 15/65 140/65 20 data_10 Jones 869 9.00E-45 65/253 140/253 18... (12 Replies)
Discussion started by: patrick87
12 Replies

9. Shell Programming and Scripting

insert a header in a huge data file without using an intermediate file

I have a file with data extracted, and need to insert a header with a constant string, say: H|PayerDataExtract if i use sed, i have to redirect the output to a seperate file like sed ' sed commands' ExtractDataFile.dat > ExtractDataFileWithHeader.dat the same is true for awk and... (10 Replies)
Discussion started by: deepaktanna
10 Replies

10. Programming

to find header in Mp3 file and retrieve data

hi all, In an mp3 file , data is arranged in sequence of header and data ,how to retrieve data between two headers. Is the data between two headers fixed? because as per theory it says 1152 samples will be there , but dont knw how many bits one sample correspond to? it would help if any c... (2 Replies)
Discussion started by: shashi
2 Replies
Login or Register to Ask a Question