Formatting data in a raw file by using another mapping file | Unix Linux Forums | UNIX for Dummies Questions & Answers

  Go Back    


UNIX for Dummies Questions & Answers If you're not sure where to post a UNIX or Linux question, post it here. All UNIX and Linux newbies welcome !!

Formatting data in a raw file by using another mapping file

UNIX for Dummies Questions & Answers


Tags
file formats, shell script, standard output

Closed Thread    
 
Thread Tools Search this Thread Display Modes
    #1  
Old 01-28-2013
ravi4informatic ravi4informatic is offline
Registered User
 
Join Date: Jan 2013
Last Activity: 4 February 2013, 5:15 AM EST
Location: bangalore
Posts: 3
Thanks: 0
Thanked 0 Times in 0 Posts
Formatting data in a raw file by using another mapping file

Hi All,

i have a requirement where i need to format the input RAW file ( which is CSV) by using another mapping file(also CSV file). basically i am getting feed file with dynamic headers by using mapping file (in that target field is mapped with source filed) i have to convert the raw file into a standard format (target format).

currently i am using ETL tool to achieve this but it's taking long time.
i am trying to write generic shell script for this. could anyone help me in a high level view. i am new to scripting. the target file also be a CSV file.

thanks in advance

Ravi
Sponsored Links
    #2  
Old 01-28-2013
dexdex200 dexdex200 is offline
Registered User
 
Join Date: Nov 2010
Last Activity: 25 February 2014, 7:57 AM EST
Posts: 16
Thanks: 3
Thanked 0 Times in 0 Posts
Provide sample input and target files.
Sponsored Links
    #3  
Old 01-28-2013
DGPickett DGPickett is offline Forum Advisor  
Registered User
 
Join Date: Oct 2010
Last Activity: 16 September 2014, 1:48 PM EDT
Location: Southern NJ, USA (Nord)
Posts: 4,412
Thanks: 8
Thanked 540 Times in 518 Posts
I suspect we need some idea of what the one csv file's table does to the other csv file. For instance, an associative array can deal with a two column translate. PERL might be a good choice, as it has CSV handlers. Shell can only deal with "well behaved" CSV, e.g., no quoted commas. PERL can draw on many resources to deal with your problem, can call compiled code and has much of the speed of compiled code.
    #4  
Old 01-29-2013
ravi4informatic ravi4informatic is offline
Registered User
 
Join Date: Jan 2013
Last Activity: 4 February 2013, 5:15 AM EST
Location: bangalore
Posts: 3
Thanks: 0
Thanked 0 Times in 0 Posts
Hi all,

Thanks for the prompt response. Below is the sample data for both files

mapping file:

Header: AU_Template_Field,RawFile_Field

Data:
VENDOR_NAME,VENDOR NAME
SHIP_TO_PARTY_NAME, PARTY_NAME
SHIP_TO_PARTY_ADDRESS1,PARTY_ADDRESS1
SHIP_TO_PARTY_ADDRESS2,PARTY_ADDRESS2
SHIP_TO_PARTY_CITY,PARTY_CITY
SHIP_TO_PARTY_STATE,PARTY_STATE
SHIP_TO_PARTY_ZIP,PARTY_ZIP

Raw File:

Header: VENDOR NAME,PARTY_NAME,PARTY_ADDRESS1,PARTY_ADDRESS2,PARTY_CITY,PARTY_STATE,PARTY_ZIP

data: abc,xyz,aaa,lny,london,london,abc123

output file:

header: VENDOR_NAME,SHIP_TO_PARTY_NAME,SHIP_TO_PARTY_ADDRESS1,SHIP_TO_PARTY_ADDRESS2,SHIP_TO_PARTY_CITY,SHIP _TO_PARTY_STATE,SHIP_TO_PARTY_ZIP

the out file strcture is standard but raw file structure might vary each time that the reason business is providing mapping file to map the columns for target file.

i hope this explanation would be clear to all.

awaiting for the responses.

Thanks in advance

Ravi
Sponsored Links
    #5  
Old 01-29-2013
DGPickett DGPickett is offline Forum Advisor  
Registered User
 
Join Date: Oct 2010
Last Activity: 16 September 2014, 1:48 PM EDT
Location: Southern NJ, USA (Nord)
Posts: 4,412
Thanks: 8
Thanked 540 Times in 518 Posts
Put the input keys in a simple array, then for each record, put each field in an associative array keyed to the input name, then write the output using the unchanging ordered list of input names for each output field to fetch the fields from the associative array.
Sponsored Links
    #6  
Old 02-02-2013
ravi4informatic ravi4informatic is offline
Registered User
 
Join Date: Jan 2013
Last Activity: 4 February 2013, 5:15 AM EST
Location: bangalore
Posts: 3
Thanks: 0
Thanked 0 Times in 0 Posts
Hi,

Thanks for the prompt response. if you don't mind could you please provide an sample code for overview so that i can use to rewrite my script.

i am new to the scripting hence i am requesting to provide a sample snippet.

Thanks in advance.

Ravi
Sponsored Links
    #7  
Old 02-04-2013
DGPickett DGPickett is offline Forum Advisor  
Registered User
 
Join Date: Oct 2010
Last Activity: 16 September 2014, 1:48 PM EDT
Location: Southern NJ, USA (Nord)
Posts: 4,412
Thanks: 8
Thanked 540 Times in 518 Posts
Bash, PERL, awk, c, c++, JAVA, ???
Sponsored Links
Closed Thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Formatting file data to another file (control character related) hk6279 Shell Programming and Scripting 3 09-24-2012 03:17 AM
Mapping a data in a file and delete line in source file if data does not exist. kokoro UNIX for Dummies Questions & Answers 4 08-20-2012 03:10 PM
AWK/Shell script for formatting data in a file rajivnairfis Shell Programming and Scripting 1 05-14-2012 07:57 AM
How to make a CSV file from a Raw data Aditya.Gurgaon UNIX for Dummies Questions & Answers 4 12-06-2010 01:57 AM
formatting data file with awk or sed lego Shell Programming and Scripting 5 03-26-2010 10:11 PM



All times are GMT -4. The time now is 12:08 PM.