Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Split into columns based on the parameter and use & as delimiter

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Split into columns based on the parameter and use & as delimiter
# 1  
Old 12-12-2010
Question Split into columns based on the parameter and use & as delimiter
Here is my source, i have million lines like this on a file.
Code:
disp0201.php?poc=4060&roc=1&ps=R&ooc=13&mjv=6&mov=5&rel=5&bod=155&oxi=2&omj=5&ozn=1&dav=20&cd=&daz=& drc=&mo=&sid=&lang=EN&loc=JPN

I want to split this into columns in order to load in database, anything starts with"&mjv=6" as first column &rel=5 as second column,&oxi=2 as third column like wise...for entire param list. But all values in parameters may change. So I guess i can use wildcard, but i need to use "&" as delimiter.

Please help me on this
EM
Last edited by elamurugu; 12-12-2010 at 01:13 PM..
# 2  
Old 12-12-2010
Post desired output for that line.
# 3  
Old 12-12-2010
My Source fields are aligned improperly back & forth

Code:
poc=4060&mjv=6&mov=5&roc=1&ps=R&ooc=13&rel=5&bod=155&oxi=2&omj=5&ozn=1&dav=20&cd=&daz=&drc=&mo=&sid=&lang=ja&loc=JPN
poc=42160&mjv=7&mov=5&roc=1&ps=B&rel=5&bod=155&ooc=19&oxi=5&omj=5&ozn=1&dav=20&cd=&daz=&drc=&loc=USA&mo=&sid=&lang=EN
poc=4040&mjv=6&mov=7&roc=1&ps=R&ooc=73&rel=5&oxi=2&omj=5&ozn=3&dav=20&bod=126&cd=&daz=&drc=&mo=&sid=&loc=GBR&lang=EN


In order to do some analysis over the data I need to organize and load in database. My desired output should be like this


Code:
poc=4060	&ooc=13	&mjv=6	&omj=5	&mov=5	&ps=R	&rel=5	&bod=155	&oxi=2	&ozn=1	lang=ja	&loc=JPN
poc=42160	&ooc=19	&mjv=7	&omj=5	&mov=5	&ps=B	&rel=5	&bod=155	&oxi=5	&ozn=1	lang=EN	&loc=USA	
poc=4040	&ooc=73	&mjv=6	&omj=5	&mov=7	&ps=R	&rel=5	&bod=126	&oxi=2	&ozn=3	lang=EN	&loc=GBR

# 4  
Old 12-12-2010
bash code:
  1. #!/bin/bash
  2. while IFS='&' read -a L
  3. do
  4.    for i in ${!L[@]}; do
  5.       eval "${L[$i]}"
  6.    done
  7.    for V in poc ooc mjv omj mov ps rel bod oxi ozn lang loc
  8.    do
  9.       echo -en "$V=${!V}\t"
  10.    done
  11.    echo
  12. done <infile
Code:
$ cat infile
poc=4060&mjv=6&mov=5&roc=1&ps=R&ooc=13&rel=5&bod=155&oxi=2&omj=5&ozn=1&dav=20&cd=&daz=&drc=&mo=&sid=&lang=ja&loc=JPN
poc=42160&mjv=7&mov=5&roc=1&ps=B&rel=5&bod=155&ooc=19&oxi=5&omj=5&ozn=1&dav=20&cd=&daz=&drc=&loc=USA&mo=&sid=&lang=EN
poc=4040&mjv=6&mov=7&roc=1&ps=R&ooc=73&rel=5&oxi=2&omj=5&ozn=3&dav=20&bod=126&cd=&daz=&drc=&mo=&sid=&loc=GBR&lang=EN

output
Code:
poc=4060	ooc=13	mjv=6	omj=5	mov=5	ps=R	rel=5	bod=155	oxi=2	ozn=1	lang=ja	loc=JPN
poc=42160	ooc=19	mjv=7	omj=5	mov=5	ps=B	rel=5	bod=155	oxi=5	ozn=1	lang=EN	loc=USA
poc=4040	ooc=73	mjv=6	omj=5	mov=7	ps=R	rel=5	bod=126	oxi=2	ozn=3	lang=EN	loc=GBR

This User Gave Thanks to frans For This Post:
elamurugu
# 5  
Old 12-12-2010
Wow...Simply amazing...If you could explain the syntax that would be great!
# 6  
Old 12-12-2010
bash code:
  1. while IFS='&' read -a L
  2. # set the field separator to '&' and read the line as an array
  3. # each elemnt of te array corresponds to an axpression between '&'
  4. # L[0]="poc=4060", L[1]="mjv=6", L[2]="mov=5" and so on
  5. # The line is plitted into an array with '&' as separator.
  6. do
  7.    for i in ${!L[@]}; do # for each index in the array
  8.       eval "${L[$i]}" # "evaluate", which means execute the command in the string
  9.    done
  10.    for V in poc ooc mjv omj mov ps rel bod oxi ozn lang loc
  11.    do
  12.       echo -en "$V=${!V}\t" # ${!<variable>} is indirect reference
  13.    done
  14.    echo
  15. done <infile
Code:
for i in ${!L[@]}; do
   eval "${L[$i]}"string
done

could also be done like
Code:
for E in ${L[@]}; do   # for each element of the array
   eval "$E"
done

as we don't need the indexes of the array
Last edited by frans; 12-12-2010 at 05:46 PM..
# 7  
Old 12-12-2010
--removed--
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Split files based on row delimiter count

I have a huge file (around 4-5 GB containing 20 million rows) which has text like: <EOFD>11<EOFD>22<EORD>2<EOFD>2222<EOFD>3333<EORD>3<EOFD>44<EOFD>55<EORD>66<EOFD>888<EOFD>9999<EORD> Actually above is an extracted file from a Sql Server with each field delimited by <EOFD> and each row ends... (8 Replies)
Discussion started by: amvip
8 Replies

2. UNIX for Dummies Questions & Answers

Split 1 column into numerous columns based on patterns

Hi, I have a text file 'Item_List.txt' containing only 1 column. This column lists different products, each separated by the same generic string header "NEW PRODUCT, VERSION 1.1". After this the name of the product is given, then a delimiter string "PRODUCT FIELD", and then the name of the... (11 Replies)
Discussion started by: mmab
11 Replies

3. Shell Programming and Scripting

awk script to split file into multiple files based on many columns

So I have a space delimited file that I'd like to split into multiple files based on multiple column values. This is what my data looks like 1bc9A02 1 10 1000 FTDLNLVQALRQFLWSFRLPGEAQKIDRMMEAFAQRYCQCNNGVFQSTDTCYVLSFAIIMLNTSLHNPNVKDKPTVERFIAMNRGINDGGDLPEELLRNLYESIKNEPFKIPELEHHHHHH 1ku1A02 1 10... (9 Replies)
Discussion started by: viored
9 Replies

4. Shell Programming and Scripting

Passing parameter to script, and split the parameter

i am passing input parameter 'one_two' to the script , the script output should display the result as below one_1two one_2two one_3two if then echo " Usage : <$0> <DATABASE> " exit 0 else for DB in 1 2 3 do DBname=`$DATABASE | awk -F "_" '{print $1_${DB}_$2}` done fi (5 Replies)
Discussion started by: only4satish
5 Replies

5. Shell Programming and Scripting

getting the last parameter through awk & split

Hi, I would like to know how to extract and get the value of the last element in an array after split. Let's say i have an input of aaa mykeyword_111 abc/def/ghi/mno bbb none ttt.aaa.ccc ccc mykeyword_222 ppp/qqq/rrr/ss I expect the output to be like this way. aaa 111 mno ccc... (4 Replies)
Discussion started by: rei125
4 Replies

6. Shell Programming and Scripting

split record based on delimiter

Hi, My inputfile contains field separaer is ^. 12^inms^ 13^fakdks^ssk^s3 23^avsd^ 13^fakdks^ssk^a4 I wanted to print only 2 delimiter occurence i.e 12^inms^ 23^avsd^ (4 Replies)
Discussion started by: Jairaj
4 Replies

7. Shell Programming and Scripting

split the file based on the 2nd column passing as a parameter

I am unable to spit the file based on the 2nd column passing as a parameter with awk command. Source file: “100”,”customer information”,”10000” “200”,”customer information”,”50000” “300”,”product information”,”40000” script: the command is not allowing to pass the parameters with the awk... (7 Replies)
Discussion started by: number10
7 Replies

8. Shell Programming and Scripting

PHP read large string & split in multidimensional arrays & assign fieldnames & write into MYSQL

Hi, I hope the title does not scare people to look into this thread but it describes roughly what I'm trying to do. I need a solution in PHP. I'm a programming beginner, so it might be that the approach to solve this, might be easier to solve with an other approach of someone else, so if you... (0 Replies)
Discussion started by: lowmaster
0 Replies

9. Shell Programming and Scripting

Substring based on delimiter, finding last delimiter

Hi, I have a string like ABC.123.XYZ-A1-B2-P1-C4. I want to delimit the string based on "-" and then get result as only two strings. One with string till last hyphen and other with value after last hyphen... For this case, it would be something like first string as "ABC.123.XYZ-A1-B2-P1" and... (6 Replies)
Discussion started by: gupt_ash
6 Replies

10. Shell Programming and Scripting

Remove lines, Sorted with Time based columns using AWK & SORT

Hi having a file as follows MediaErr.log 84 Server1 Policy1 Schedule1 master1 05/08/2008 02:12:16 84 Server1 Policy1 Schedule1 master1 05/08/2008 02:22:47 84 Server1 Policy1 Schedule1 master1 05/08/2008 03:41:26 84 Server1 Policy1 ... (1 Reply)
Discussion started by: karthikn7974
1 Replies
Login or Register to Ask a Question