Normalize Data and write to a flat file


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Normalize Data and write to a flat file
# 8  
Old 05-18-2011
Thanks you danmero for the help.
# 9  
Old 05-19-2011
Here is a shell script solution:
Code:
#!/usr/bin/ksh
typeset -i mI=0
IFS='|'
set -A mTag $(head -1 inp_file)
sed '1d' inp_file | while read mLine; do
  set -A mValue ${mLine}
  mI=1
  while [[ ${mI} -le ${#mValue[*]} ]]; do
    if [[ "${mValue[$mI]}" != "" ]]; then
      for mTemp in $(echo ${mValue[$mI]} | sed 's/,/|/g'); do
        echo "${mValue[0]}|${mTag[$mI]}|${mTemp}"
      done
    fi
    mI=${mI}+1
  done
done

# 10  
Old 05-20-2011
Danmero,

Thank you very much, but the output is getting sorted, can you please help me so that the output is not sorted, and same as the orginal file.
Code:
awk '
BEGIN{
        FS=OFS="|";ORS = "\n";
        print "ID","Col_Name","Value","Count"
}
!(NR-2){
        split($0,l,FS);
        next
}
{
        for(o=1;++o<NF;)
        {
                split($o,_,",");
                for(O=0;O++<asort(_);)
                {
                       print toupper($1),toupper(l[o]),toupper(_[O]),O
                }
        }
}'

Thanks
Sunil

---------- Post updated at 05:01 PM ---------- Previous update was at 02:46 PM ----------

Danmero,

I have a few changes for the outfile, am not able to modify the awk , can you please help me in this.

I have highlighted the new change in brown color

Can anyone please help me with the below scenario.

I have a Flat file of the below format.

Code:
ID|Name|Level|Type|Zip|MAD|Risk|Band|Salesl|Dealer|CID|AType|CValue|LV|HV|DCode|TR|DU|NStartDate|Use rRole|WFlag|EOption|PName|NActivationDate|Os|Orig|Cus|OType|ORequired|DType
03| Prog|3||90001,90002,90003,90004-90006,90007,90008|881,866,862-839,828||99-99,88-88,11-19||||REG||||MOQA,MODD,9999,8229,8228,8227,8226||||Confffers||Leased|||FREE-TO-GO-2011,FREE-TO-GO-2012|NMS|new|Acquis,Acquis|mandat,mandat|


Rules

File Column delimiter is |
ID column will not have multipe values seperated by commas.
Columns which have values separated by commas, should be made into a New row.
Columns which do not have values , no need make them into New row.
Last column Orequired is to be concatenated with appropriate Os value in order as highlited below.


Output file format requested


Code:
ID|Col_Name|Value
03|Name|Prog
03|Level|3
03|Zip|90001
03|Zip|90002
03|Zip|90003
03|Zip|90004-90006
03|Zip|90007
03|Zip|90008
03|MAD|881
03|MAD|866
03|MAD|862-839
03|MAD|828
03|BAND|99-99
03|BAND|88-88
03|BAND|11-19
03|AType|REG
03|DCode|MOQA
03|DCode|MODD
03|DCode|9999
03|DCode|8229
03|DCode|8228
03|DCode|8227
03|DCode|8226
03|UserRole|Confffers
03|EOption|Leased
03|Os|FREE-TO-GO-2011
03|Os|FREE-TO-GO-2012
03|Orig|NMS
03|Cus|new
03|OType|Acquis
03|OType|Acquis
03|ORequired|FREE-TO-GO-2011/mandat
03|ORequired|FREE-TO-GO-2012/mandat

Last edited by Scott; 05-20-2011 at 07:16 PM.. Reason: Use code tags, please...
# 11  
Old 05-21-2011
Code:
gawk 'BEGIN{FS=OFS="|";print "ID","Col_Name","Value"}!(NR-1){split($0,l,FS);for(i in l){if($i=="Os")x=i;if($i=="ORequired")y=i};next}{for(o=1;++o<NF;){if(o==x){X=$o};if(o==y)split(X,Y,",");split($o,_,",");n=0;for(i in _)n++;for(O=0;O++<n;){print $1,l[o],((Y[O])?Y[O]"/":Z)_[O]}}}' file

 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Programming

Need sql query to string split and normalize data

Hello gurus, I have data in one of the oracle tables as as below: Column 1 Column 2 1 NY,NJ,CA 2 US,UK, 3 AS,EU,NA fyi, Column 2 above has data delimited with a comma as shown. I need a sql query the produce the below output in two columns... (5 Replies)
Discussion started by: calredd
5 Replies

2. Shell Programming and Scripting

Getting data from a flat file based on condition

Hi, I have a flaty file from which i am fetching few columns in tablular form as below code. Now i want to fetch the column 6 and 7 in below code only if it either of them is non zero.However below startement awk -F, '$6==0 && $7==0{exit 1}' ${IFILE} is not working..Not sure where is the... (36 Replies)
Discussion started by: Vivekit82
36 Replies

3. Shell Programming and Scripting

Data is available or not in a flat file generated by Oracle

Hello, please help me an the below issue. i need to check whether data is available or not in a flat file generated by oracle (sometimes sql didn't any records) to overcome this. without opening flat file. Thanks....... (1 Reply)
Discussion started by: mahesh1987
1 Replies

4. Shell Programming and Scripting

Reading XML data in a FLAT FILE

I have a requirement to read the xml file and split the files into two diffrent files in Unix shell script. Could anyone please help me out with this requirement. Sample file --------------- 0,<?xml version="1.0" encoding="UTF-8" standalone="yes"?> <Information... (3 Replies)
Discussion started by: kmanivan82
3 Replies

5. Shell Programming and Scripting

To read a flat file containing XML data

I have a file something like this:aaaa.xml content of the file is 0,<?xml version="1.0" encoding="UTF-8" standalone="yes"?> <storeInformation xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <s> <BRANCH_NO>3061</BRANCH_NO> <BRANCH_NAME>GREEN EXPRESS</BRANCH_NAME> ... (4 Replies)
Discussion started by: kmanivan82
4 Replies

6. Shell Programming and Scripting

Extracting data into flat file thru unix

Hi, I need to extract a oracle staging table to a flat file thru unix batch process.We are expecting more than 4million records in the table.I know I can do it using "UTL_FILE" .But,since "UTL_FILE" takes a lot of time I am looking for better options.Can any body suggest some better options? ... (3 Replies)
Discussion started by: Beena
3 Replies

7. Shell Programming and Scripting

Load data from a flat file to oracle.

I have a flat file with records like Header 123 James Williams Finance2000 124 Pete Pete HR 1500 125 PatrickHeather Engg 3000 Footer The structure is: Eno:4 characters Name:8 characters Surname : 9 characters Dept:7 characters Sal:4characters These are sample... (1 Reply)
Discussion started by: Shivdatta
1 Replies

8. Shell Programming and Scripting

Load data to flat file from table.

Hi all, I need to know how to copy data from a table say ABC to a flat file say XYZ.dat in unix, Please leave ur comments and the fastest way to do so, I need to load the table records into flat file. Regards Ann (4 Replies)
Discussion started by: Haque123
4 Replies

9. Shell Programming and Scripting

inserting data into a table from a flat file

Hi, I want to insert data into a table from a flat file, the file is having around 25 columns and some 10,000 records. The columns values are seperated by a space. Thanks (1 Reply)
Discussion started by: ss_ss
1 Replies

10. Shell Programming and Scripting

Help with Data Positioning from Columns in a flat file.

Hi All, I have used this forum many times to solve my many scripting problems. This time, I would like to seek some answers to a problem that I've been head scratching quite a bit on. My Example: I am converting a 2000-byte file into a 300-byte file this file has no delimiters and hardly any... (3 Replies)
Discussion started by: oott1
3 Replies
Login or Register to Ask a Question