Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

mapping of values in shell scripting

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting mapping of values in shell scripting
# 1  
Old 09-19-2011
mapping of values in shell scripting
sample content of file1:
Code:
SSTY1         2145228348       652011011715140100000002419005432092074                 008801726143662             VDZX01                                  MIO2                                           008801726143662
SSRTY         2145228349       652011011715163400000000095006543493559                 030729961474195             RTYUI9                                  MMO2                                           030729961474195
       SSI09  2145228354       652011011715124900000003539000447438811497              08860700789                                     IIUYT4                  IMO2                                       041088860700789
SSRTY         2145228355       652011011715152600000001176008860004296                 041289796822734             KIUTY6                                  MMO2                                           041289796822734
SSRTY  SSRTY  2145228357       652011011715154000000001033009711806239                 027279810510494             ITYIP4              HUIPO5              MMR2MMO2                                       027279810510494
       SSRTY  2145228363       652011011715161500000000286001137000707                 025409450590072                                 LRTER4                  MMO2                                       025409450590072
SSRTY         2145228366       652011011715155800000000456001171220000                 030518679540203             OOUTY6                                  FMO2                                           030518679540203
       SSRTY  2145228367       652011011715142700000002162009833314457                 030099569768624                                 NPAXM2                  MMO2                                       030099569768624
SSRTY  SSRTY  2145228368       652011011715160000000000436009811279923                 031129311382325             RTYUI9              VYVXG5              MUR2MUO2                                       031129311382325
SSRTY  SSRTY  2145228368       652011011715160000000000436009811279923                 031129311382325             IUERW3
          VYVXG5              MUR2MUO2                                       031129311382325

Code:
cat file1 | awk -f $HOME_NEW/abc.awk | sed -e 's/[\t ]//g;/^$/d;' | sed 's/,,/, ,/g' | sed 's/,,/, ,/g' > $CSVFileName

abc.awk
Code:
{
if(substr($0,87,1)=="T" || substr($0,87,1)=="O" || substr($0,87,1)=="M" || substr($0,87,1)=="Y" || substr($0,87,1)=="T" || substr($0,87,1)=="C")
{print sprintf("%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s",substr($0,34,8),substr($0,42,6), "",substr($0,60,28),substr($0,88,28),substr($0,50,4)*3600+substr($0,54,2)*60+substr($0,56,2)+substr($0,58,2)/100,"","",substr ($0,1,7),substr($0,87,20),"","","","I","","","","")}
else if(substr($0,123,1)=="T" || substr($0,123,1)=="O" || substr($0,123,1)=="M" || substr($0,123,1)=="Y" || substr($0,123,1)=="T" || substr($0,123,1)=="C")
{print sprintf("%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s",substr($0,34,8),substr($0,42,6), "",substr($0,60,28),substr($0,88,28),substr($0,50,4)*3600+substr($0,54,2)*60+substr($0,56,2)+substr($0,58,2)/100,"","",substr ($0,8,7),substr($0,123,20),"","","","O","","","","")}
}

the above command was working successfully.and the out put generated was:
output
Code:
20110117,151634, ,09899493559,030729961474195,9.5, , ,SSRTY,RTYUI9, , , ,I, , , ,
20110117,151249, ,00447438811497,08860700789,233.9, , ,SSI09,ITYIP4, , , ,O, , , ,
20110117,151540, ,09711806239,027279810510494,63.3, , ,SSRTY,ITYIP4, , , ,I, , , ,
20110117,151615, ,01137000707,025409450590072,28.6, , ,SSRTY,LRTER4, , , ,O, , , ,
20110117,151600, ,09811279923,031129311382325,43.6, , ,SSRTY,RTYUI9, , , ,I, , , ,
20110117,151600, ,09811279923,031129311382325,43.6, , ,SSRTY,IUERW3, , , ,I, , , ,

the above command was working successfully.
**************************************************************************************************** *************************
But the problem is now My requirement is to search 9th and 10th field of the input file file1 and then compare with another file that has 9th and 10th fields and collect the Continent name from this file.... then put it in the output file. I am able to do it but the issue is there are around 10,000 records in that file.For less records (around 50)...it is working file
for eg.
Code:
id1           id2           Continent name
-----           ------          --------------
SSRTY  RTYUI9  RIGHT1
SSI09  ITYIP4  AUSTRALIA
SSRTY  ITYIP4  ASIA
SSRTY  LRTER4  NORTHUS
SSRTY  RTYUI6  KOREA
SSRTY  IUERW3  ANTARCTICA
.
.
10,000 records are there
.
.

My output should be :
output
Code:
20110117,151634, ,09899493559,030729961474195,9.5, , ,SSRTY,RTYUI9, , , ,I, , , ,RIGHT1,
20110117,151249, ,00447438811497,08860700789,233.9, , ,SSI09,ITYIP4, , , ,O, , , , ,
20110117,151540, ,09711806239,027279810510494,63.3, , ,SSRTY,ITYIP4, , , ,I, , , ,ASIA,
20110117,151615, ,01137000707,025409450590072,28.6, , ,SSRTY,LRTER4, , , ,O, , , ,NORTHUS,
20110117,151600, ,09811279923,031129311382325,43.6, , ,SSRTY,RTYUI9, , , ,I, , , ,RIGHT1,
20110117,151600, ,09811279923,031129311382325,43.6, , ,SSRTY,IUERW3, , , ,I, , , ,ANTARCTICA,

I was using the following approach,, but it was giving me below error:
Code:
cat $filename | awk -f $HOME_NEW/abc.awk | sed -e 's/[\t ]//g;/^$/d;' |  awk -F"," -f $HOME_NEW/AWK_GENERATE_SCRIPT.awk | awk 'NR > 1 { gsub( / ,$/, "," ); print }' | sed 's/,,/, ,/g' | sed 's/,,/, ,/g' > $CSVFileName

AWK_GENERATE_SCRIPT.awk:
Code:
{
if ( $9 =="SSRTY" && $10 == "RTYUI9") CONTINENT_NAME="RIGHT1";
else if ( $9 =="SSI09" && $10 == "ITYIP4") CONTINENT_NAME="AUSTRALIA";
else if ( $9 =="SSRTY" && $10 == "ITYIP4") CONTINENT_NAME="ASIA";
else if ( $9 =="SSRTY" && $10 == "LRTER4") CONTINENT_NAME="NORTHUS";
.
.
.
.
10,000 DIFFERENT ID1 AND ID2
.
.
else CONTINENT_NAME="";
print $0 CONTINENT_NAME,",";
}

eRROR:
Code:
+  YACC Stack Overflow The source line is 49.
 The error context is
                else if ( $9 =="IIYTE" && $10 == >>>  "FRTYH4" <<< ) CONTINENT="NORTH KOREA";
        awk: 0602-541 There are 19 missing ) characters.
++

Please help..... if you can solve it..
Last edited by vsachan; 09-19-2011 at 01:42 PM.. Reason: Use code tags, please...
# 2  
Old 09-19-2011
You can create a sed file with all 10,000 combinations as:
Code:
s/.*,SSRTY,RTYUI9,.*/&RIGHT1,/
s/.*,SSI09,ITYIP4,.*/&AUSTRALIA,/
s/.*,SSRTY,ITYIP4,.*/&ASIA,/
s/.*,SSRTY,LRTER4,.*/&NORTHUS,/

Then run as:
Code:
sed -f sed_file Inp_File

If you encounter any problem, keep breaking the sed file until you can manage to run it, ie 2 files of 5,000 records, then 4 files of 2,500, 10 files of 1,000, etc.
This User Gave Thanks to Shell_Life For This Post:
vsachan
# 3  
Old 09-20-2011
Thx for giving the solution Shell_Life.

But the performance is very poor.I am using below script . Please check if I can improve the performance.

script.sh:
--------
Code:
#!/bin/sh
set -x
file_name=$1;
sed -f 111.txt $file_name > temp.txt
sed -f 222.txt temp.txt > temp1.txt
sed -f 333.txt temp1.txt > temp2.txt
sed -f 444.txt temp2.txt > temp3.txt
sed -f 555.txt temp3.txt > temp4.txt
sed -f 666.txt temp4.txt > temp5.txt
sed -f 777.txt temp5.txt > temp6.txt
sed -f 888.txt temp6.txt > temp7.txt
sed -f 999.txt temp7.txt > temp8.txt
sed -f AAA.txt temp8.txt > temp9.txt
sed -f BBB.txt temp9.txt > temp10.txt
sed -f CCC.txt temp10.txt > temp11.txt
 
rm -f temp.txt temp1.txt temp2.txt temp3.txt temp4.txt temp5.txt temp6.txt temp7.txt temp8.txt temp9.txt temp10.txt
 
mv -f temp11.txt $file_name

*******************************************************

where 111.txt,222.txt,333.txt contains the substitue commands.Each file contains 500 substitution commands. and I am passing the file name as 1st parameter to the script. This file contains around 40,000 Records.

One more concern---I have to put a space ,then a comma if there is no substitution in the record.Please check the third line of the output of the following 2 records.

for eg
if I am not able to find a match, then the output should be :
Code:
20110117,151615, ,01137000707,025409450590072,28.6, , ,SSRTY,LRTER4, , , ,O, , , ,NORTHUS,
20110117,151600, ,09811279923,031129311382325,43.6, , ,SSRTY,RTYUI9, , , ,I, , , ,RIGHT1,
20110117,151600, ,09811279923,031129311382325,43.6, , ,SSRTY,IUERW3, , , ,I, , , ,,

But currently the output is:
Code:
20110117,151615, ,01137000707,025409450590072,28.6, , ,SSRTY,LRTER4, , , ,O, , , ,NORTHUS,
20110117,151600, ,09811279923,031129311382325,43.6, , ,SSRTY,RTYUI9, , , ,I, , , ,RIGHT1,
20110117,151600, ,09811279923,031129311382325,43.6, , ,SSRTY,IUERW3, , , ,I, , , ,

Please help how to improve the performance.
Last edited by Franklin52; 09-20-2011 at 04:37 AM.. Reason: Please use code tags, thank you
# 4  
Old 09-20-2011
Here is an entire different approach:

First create a file with all 10,000 combinations as follows:
Code:
,,,,,,,,SSRTY,RTYUI9,RIGHT1
,,,,,,,,SSI09,ITYIP4,AUSTRALIA
,,,,,,,,SSRTY,ITYIP4,ASIA
,,,,,,,,SSRTY,LRTER4,NORTHUS

Note that there are 8 (eight) commas before the first data.

Then run the following script:
Code:
#!/usr/bin/ksh
sort -A -t',' -k9,10 -k1 Original_File Combination_File > Mixed_File
IFS=','
while read mF1 mF2 mF3 mF4 mF5 mF6 mF7 mF8 mF9 mF10 mF11 mF12 mF13 mF14 mF15 mF16 mF17; do
  if [[ "${mF1}" = "" ]]; then
    mSave9=${mF9}
    mSave10=${mF10}
    mSave11=${mF11}
    continue
  fi
  if [[ "${mF9}" = "${mSave9}" && "${mF10}" = "${mSave10}" ]]; then
    echo "${mF1},${mF2},${mF3},${mF4},${mF5},${mF6},${mF7},${mF8},${mF9},${mF10},${mF11},${mF12},${mF13},${mF14},${mF15},${mF16},${mF17},${mSave11},"
  else
    echo "${mF1},${mF2},${mF3},${mF4},${mF5},${mF6},${mF7},${mF8},${mF9},${mF10},${mF11},${mF12},${mF13},${mF14},${mF15},${mF16},${mF17},,"
  fi
done < Mixed_File

This User Gave Thanks to Shell_Life For This Post:
vsachan
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to calculate avg values of csv file using shell scripting .?

hi all i have a reporting work and i want it to be automated using shell scripting kindly let me know how can i make that possibe . eg data are :... (2 Replies)
Discussion started by: Avinash shaw
2 Replies

2. Shell Programming and Scripting

Regex in Shell Scripting to pick values

Hi After lot of trial and error I am really bowled out with the requirement in hand and honestly you are my last hope Here is what I want to achieve Values *IF *VALUE MS_SQL_Statistics_Summary.Client_Count_Percent_Used *GT 70.00 *AND *VALUE... (20 Replies)
Discussion started by: radioactive9
20 Replies

3. UNIX for Dummies Questions & Answers

How to use square values in Shell Scripting?

:wall: Hi I am a newbie with Shell Scripting who stuck while creating a shell script for Pythagoras theorem.I need to know how to add the squares for the value in shell scripting(for eg: b2 =a2 +c2). Thanks VR (4 Replies)
Discussion started by: VoraciousReader
4 Replies

4. UNIX for Dummies Questions & Answers

How to compare to values returned from sql in shell scripting?

hey i am using this code to connect to sql , store the value in variable and then compare it with another variable after some time by executing the same query but the desired result is not coming #!/bin/bash val=$(sqlplus -s rte/rted2@rel76d2 <<ENDOFSQL set heading off set feedback off... (11 Replies)
Discussion started by: ramsavi
11 Replies

5. Shell Programming and Scripting

Assigning array values using awk in shell scripting

hi My script as below #!/bin/ksh for i in `seq 1 7` do a=$(awk '{print $i}' /home/rama/expenese.txt) done for i in `seq 1 7` do echo "${a}" done content of expense.txt is as below 5032 210179 3110 132813874 53488966 11459221 5300794 I want output as... (6 Replies)
Discussion started by: Ramakrishna V
6 Replies

6. Programming

Help with mapping two tables and filling the null values

Hi All , I have two tables. I will provide sample data in the tables below. table1: dept_id dept_name rep_id admin_lastname 10 dept of int.medicine Paul 10 dept of int.medicine Frank 20 dept of chemistry Young 20 dept of chemistry Bill 30 school of denistry kaufmann 40... (3 Replies)
Discussion started by: megha2525
3 Replies

7. Shell Programming and Scripting

Need help to change XML values with shell scripting for Network Simulation

Hello, I don't have experience in this scripting and I need some help to read a value from an XML file and change it with a random number to use in simulator for different network scenarios. </Description><sim_comm_rounds>35</sim_comm_rounds><num_clusters>1</num_clusters><Clocking> I want to... (5 Replies)
Discussion started by: erhanasd
5 Replies

8. Shell Programming and Scripting

[BASH] mapping of values from file line into variables

Hello, I've been struggling with this for some time but can't find a way to do it and I haven't found any other similar thread. I'd like to get the 'fields' in a line from a file into variables in just one command. The file contains data with the next structure:... (4 Replies)
Discussion started by: semaler
4 Replies

9. UNIX for Dummies Questions & Answers

Concatenating arrays cell values in shell scripting

Hi All, I want to concatenate the array cell values and form a string.. Is it possible? for ex. I have an array word_array contains d u m b and after concatenating the string shld be 'dumb' thanks (2 Replies)
Discussion started by: mathur
2 Replies

10. Programming

Xlib - Mapping Pixel Values to RGB Colors

Hi everyone... I'm working with XLib and I want to find out the pixel value of a particular point on screen and then map it to RGB values. I used XGetImage and XGetPixel to get the pixel value, but how do I get the RGB values of the pixel? I went through a few XLib manuals, there seems to be no... (1 Reply)
Discussion started by: thebin
1 Replies
Login or Register to Ask a Question