Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

How can I modify my script to include or substitute missing data?

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting How can I modify my script to include or substitute missing data?
# 1  
Old 06-18-2013
Sun How can I modify my script to include or substitute missing data?
Let me start off by saying I am a self taught sometimes scripter so what you will see below won't be pretty.

I have created a script to parse through a file with a large amount of data and simply pull out what I need. In doing this I create several files and then paste them together in order to create a .txt file that can be imported into Excel for formatting. When I created the script (with a lot of help from the members of this forum) I "pounded" out some stats that were not required. I now need to pull the stats (the ones that are "pounded" out) but the first issue is there are several times that the true result of the stat is 0 and part of the original input file has erronious data that shows as a 0 so as you can see the script removes any report of a 0 as the result. The second issue is when the 0 is removed the file made for that certain "peg count" doesn't match line for line with the other pegcounts and it scews the data.

This first bit of code (this is only one 15 minute report, the actual file has 96 15 minute reports to encompass the entire days worth of data) is the file I am parsing, where you see the output of
Code:
<r p="12">0</r>

that is the start of the eronious data and where you see the
Code:
<r p="12">2803</r>

is where the data starts to be correct.:
Code:
A20130617.0000-0600-0015-0600_SubNetwork=DMME1,ManagedElement=ces-1
                        <measType p="12">VS.NbrSuccessAttachRequests</measType>
                        <measType p="13">VS.NbrFailedAttachRequestsNotSysRelated_sum</measType>
                        <measType p="14">VS.AttAttachRequests</measType>
                        <measType p="15">VS.NbrAttachFailureSysRelated_sum</measType>
                        <measType p="16">VS.NbrFailedAttachRequests_IllegalME</measType>
                        <measType p="17">VS.NbrFailedAttachRequests_NoCellsInTA</measType>
                        <measType p="18">VS.NbrFailedAttachRequests_Other</measType>
                        <measType p="19">VS.NbrSuccessServiceRequests</measType>
                        <measType p="20">VS.NbrFailedServiceRequestsNotSysRelated_sum</measType>
                        <measType p="21">VS.NbrFailedServiceRequests_Other</measType>
                                <r p="12">0</r>
                                <r p="13">0</r>
                                <r p="14">0</r>
                                <r p="15">0</r>
                                <r p="16">0</r>
                                <r p="17">0</r>
                                <r p="18">0</r>
                                <r p="19">0</r>
                                <r p="20">0</r>
                                <r p="21">0</r>
                                <r p="12">2803</r>
                                <r p="13">73</r>
                                <r p="14">2925</r>
                                <r p="15">6</r>
                                <r p="16">0</r>
                                <r p="17">19</r>
                                <r p="18">6</r>
                                <r p="19">168255</r>
                                <r p="20">1414</r>
                                <r p="21">119</r>
                        <measType p="12">VS.MaxNbrOfConnectedUEperPLMN</measType>
                        <measType p="13">VS.AveNbrOfRegIdleUEperPLMN</measType>
                        <measType p="14">VS.MaxNbrOfRegIdleUEperPLMN</measType>
                        <measType p="15">VS.AveNbrOfHomeSubscribersPerPLMN</measType>
                        <measType p="16">VS.MaxNbrOfHomeSubscribersPerPLMN</measType>
                        <measType p="17">VS.AveNbrOfVisitingSubscribersPerPLMN</measType>
                        <measType p="18">VS.MaxNbrOfVisitingSubscribersPerPLMN</measType>
                                <r p="12">2127</r>
                                <r p="13">8364</r>
                                <r p="14">8494</r>
                                <r p="15">10374</r>
                                <r p="16">10413</r>
                                <r p="17">0</r>
                                <r p="18">0</r>
                        <measType p="12">VS.Total_S11gtpEcho</measType>
                        <measType p="13">VS.NbrMissed_S11gtpEcho</measType>
                        <measType p="14">VS.Total_S10gtpEcho</measType>
                        <measType p="15">VS.NbrMissed_S10gtpEcho</measType>
                        <measType p="16">VS.TotalReqSent_S11gtpc</measType>
                        <measType p="17">VS.NbrTO_S11gtpc</measType>
                        <measType p="18">VS.TotalReqSent_S10gtpc</measType>
                        <measType p="19">VS.NbrTO_S10gtpc</measType>
                        <measType p="20">VS.TotalReqRcvdS11</measType>
                        <measType p="21">VS.TotalReqRcvdS10</measType>
                                <r p="12">0</r>
                                <r p="13">0</r>
                                <r p="14">0</r>
                                <r p="15">0</r>
                                <r p="16">0</r>
                                <r p="17">0</r>
                                <r p="18">0</r>
                                <r p="19">0</r>
                                <r p="20">0</r>
                                <r p="21">0</r>
                                <r p="12">39</r>
                                <r p="13">1</r>
                                <r p="14">15</r>
                                <r p="15">0</r>
                                <r p="16">457386</r>
                                <r p="17">1</r>
                                <r p="18">1</r>
                                <r p="19">0</r>
                                <r p="20">80499</r>
                                <r p="21">0</r>
                        <measType p="12">VS.restartTask</measType>
                        <measType p="13">VS.reInitServiceSelf</measType>
                        <measType p="14">VS.reInitServiceManual</measType>
                        <measType p="15">VS.memAllocFail</measType>
                        <measType p="16">VS.exceptionService</measType>
                                <r p="12">0</r>
                                <r p="13">0</r>
                                <r p="14">0</r>
                                <r p="15">0</r>
                                <r p="16">0</r>
                                <r p="12">0</r>
                                <r p="13">0</r>
                                <r p="14">0</r>
                                <r p="15">0</r>
                                <r p="16">0</r>

This bit of code is the script I use to do the parsing***Please note that for the purpose of this post I corrected the cgrep commands to include the correct amount of lines***: .
Code:
cat DMME_cmds_post
rm DEN_samcomplete.txt samfile2
Yesterday=$(perl -e '@y=localtime(time()-86400);printf "%04d%02d%02d",$y[5]+1900
,$y[4]+1,$y[3];')
cat DEN_samfile | cgrep +30 SubNetwork=DMME1,ManagedElement >> samfile2
sleep 5
echo "starting samp12"
cat samfile2 | cgrep +20 NbrSuccessAttachRequests | egrep -v '>0<' >> samp12c
cat samp12c | cgrep +10 NbrSuccessAttachRequests | grep 'r p="12"'  >> samp12.txt
sleep 5
echo "starting samp13"
cat samfile2 | cgrep +20 NbrFailedAttachRequestsNotSysRelated_sum | egrep -v '>0<' >> samp13c
cat samp13c | cgrep +10 NbrFailedAttachRequestsNotSysRelated_sum | grep 'r p="13"' >> samp13.txt
sleep 5
echo "starting samp14"
cat samfile2 | cgrep +20 AttAttachRequests | egrep -v '>0<' >> samp14c
cat samp14c | cgrep +10 AttAttachRequests | grep 'r p="14"'  >> samp14.txt
sleep 5
echo "starting samp15"
cat samfile2 | cgrep +20 NbrAttachFailureSysRelated_sum | egrep -v '>0<' >> samp15c
cat samp15c | cgrep +10 NbrAttachFailureSysRelated_sum | grep 'r p="15"' >> samp15.txt
#sleep 5
#echo "starting samp16"
#cat samfile2 | cgrep +20 NbrFailedAttachRequests_IllegalME | egrep -v '>0<' >> samp16c
#cat samp16c | cgrep +10 NbrFailedAttachRequests_IllegalME | grep 'r p="16"'  >> samp16.txt
#sleep 5
#echo "starting samp17"
#cat samfile2 | cgrep +20 NbrFailedAttachRequests_NoCellsInTA | egrep -v '>0<' >> samp17c
#cat samp17c | cgrep +10 NbrFailedAttachRequests_NoCellsInTA | grep 'r p="17"' >> samp17.txt
sleep 5
echo "starting samp18"
cat samfile2 | cgrep +20 NbrFailedAttachRequests_Other | egrep -v '>0<' >> samp18c
cat samp18c | cgrep +10 NbrFailedAttachRequests_Other | grep 'r p="18"'  >> samp18.txt
sleep 5
echo "starting samp19"
cat samfile2 | cgrep +20 NbrSuccessServiceRequests | egrep -v '>0<' >> samp19c
cat samp19c | cgrep +10 NbrSuccessServiceRequests | grep 'r p="19"' >> samp19.txt
sleep 5
echo "starting samp20"
cat samfile2 | cgrep +20 NbrFailedServiceRequestsNotSysRelated_sum | egrep -v '>0<' >> samp20c
cat samp20c | cgrep +10 NbrFailedServiceRequestsNotSysRelated_sum | grep 'r p="20"'  >> samp20.txt
sleep 5
echo "starting samp21"
cat samfile2 | cgrep +20 NbrFailedServiceRequests_Other | egrep -v '>0<' >> samp21c
cat samp21c | cgrep +10 NbrFailedServiceRequests_Other | grep 'r p="21"' >> samp21.txt
 
touch samp12a
touch samp13a
touch samp14a
touch samp15a
touch samp16a
touch samp17a
touch samp18a
touch samp19a
touch samp20a
touch samp21a
echo "parsing the files"
sed 's/<r p="12">//g' samp12.txt > samp12a
sed 's/<\/\r>//g' samp12a > samp12b.txt
cat samp12b.txt | tr -d " \t" > samp12.txt
sed 's/<r p="13">//g' samp13.txt > samp13a
sed 's/<\/\r>//g' samp13a > samp13b.txt
cat samp13b.txt | tr -d " \t" > samp13.txt
sed 's/<r p="14">//g' samp14.txt > samp14a
sed 's/<\/\r>//g' samp14a > samp14b.txt
cat samp14b.txt | tr -d " \t" > samp14.txt
sed 's/<r p="15">//g' samp15.txt > samp15a
sed 's/<\/\r>//g' samp15a > samp15b.txt
cat samp15b.txt | tr -d " \t" > samp15.txt
#sed 's/<r p="16">//g' samp16.txt > samp16a
#sed 's/<\/\r>//g' samp16a > samp16b.txt
#cat samp16b.txt | tr -d " \t" > samp16.txt
#sed 's/<r p="17">//g' samp17.txt > samp17a
#sed 's/<\/\r>//g' samp17a > samp17b.txt
#cat samp17b.txt | tr -d " \t" > samp17.txt
sed 's/<r p="18">//g' samp18.txt > samp18a
sed 's/<\/\r>//g' samp18a > samp18b.txt
cat samp18b.txt | tr -d " \t" > samp18.txt
sed 's/<r p="19">//g' samp19.txt > samp19a
sed 's/<\/\r>//g' samp19a > samp19b.txt
cat samp19b.txt | tr -d " \t" > samp19.txt
sed 's/<r p="20">//g' samp20.txt > samp20a
sed 's/<\/\r>//g' samp20a > samp20b.txt
cat samp20b.txt | tr -d " \t" > samp20.txt
sed 's/<r p="21">//g' samp21.txt > samp21a
sed 's/<\/\r>//g' samp21a > samp21b.txt
cat samp21b.txt | tr -d " \t" > samp21.txt
echo "removing work files"
rm samp12a samp12c
rm samp13a samp13c
rm samp14a samp14c
rm samp15a samp15c
#rm samp16a samp16c
#rm samp17a samp17c
rm samp18a samp18c
rm samp19a samp19c
rm samp20a samp20c
rm samp21a samp21c
echo "putting all the files into just 1 file."
paste samp12.txt samp13.txt samp14.txt samp15.txt samp18.txt samp19.txt samp20.txt samp21.txt > DEN_samcomplete.txt
mv DEN_samfile /home/fsanchez/output/oldsamfiles/`echo $Yesterday`_DEN_samfile
uuencode DEN_samcomplete.txt DEN_samcomplete.txt | mailx -r "myemail@email.com" -s "LTE Stats" myemail@myemail.com
rm ./samp*

This bit of code is what I get if I run the file on a complete days worth of data, each column is associated with a "Peg Count" (p=12 - 21). Peg count 16 was all 0's so it is blank which I would like it to show 0. Peg count 17 had only 1 15 minute report with a 0 so it scews the data to look like I am missing the last report which I am actually missing a report from some time at the beginning of the file.
Code:
2803    73      2925    6               19      6       168255  1414    119
2615    84      2735    5               13      9       169562  1379    130
2545    94      2680    6               23      11      162351  1348    101
2535    101     2690    7               17      11      159639  1423    97
2352    71      2465    5               22      6       156197  1337    93
2352    67      2445    3               22      4       154050  1324    80
2296    64      2404    9               18      16      148157  1368    81
2235    67      2345    9               21      9       145067  1351    60
2270    78      2390    7               21      12      142891  1382    65
2208    72      2317    4               12      5       140745  1378    50
2247    71      2353    6               14      12      139418  1396    51
2335    55      2426    5               7       8       137486  1480    60
2357    56      2454    4               7       5       137306  1452    42
2291    65      2410    8               11      10      135976  1436    68
2214    49      2316    6               4       7       135250  1468    80
2322    46      2404    9               3       10      134332  1510    48
2356    69      2479    7               8       14      133548  1509    40
2328    54      2446    11              6       12      132675  1489    36
2457    67      2580    6               2       8       134563  1553    44
2464    53      2556    2               4       3       133210  1556    75
2531    83      2659    7               4       12      134425  1544    57
2529    87      2671    4               8       10      134976  1469    53
2659    93      2807    9               1       13      136561  1566    76
2567    92      2713    10              9       14      137131  1478    91
2687    143     2906    13              8       21      138799  1534    81
2681    114     2855    8               9       11      140083  1512    84
2869    138     3075    8               4       13      141841  1560    84
2921    129     3124    11              7       19      144602  1567    112
2996    133     3182    5               14      8       147394  1559    95
2973    130     3154    6               21      7       149753  1566    86
3053    137     3252    8               17      12      153467  1591    92
3162    164     3396    9               5       12      154118  1668    98
3238    160     3478    9               19      11      156333  1517    88
3246    169     3491    4               19      7       157871  1575    101
3148    196     3408    13              15      22      162255  1511    120
3299    177     3564    13              9       19      163731  1565    132
3143    185     3421    12              17      21      167464  1395    139
3108    178     3356    11              25      19      167298  1379    118
3023    184     3280    13              25      18      170372  1327    114
3210    195     3479    10              24      12      169593  1399    125
3232    214     3553    12              29      18      170630  1339    132
3340    215     3644    15              12      29      174012  1403    148
3264    188     3548    14              17      21      175383  1344    146
3296    214     3586    8               24      12      176957  1342    139
3407    224     3723    11              23      22      176368  1382    173
3368    188     3649    13              10      21      175547  1382    174
3371    212     3693    19              7       38      179640  1340    148
3502    193     3783    11              10      20      183378  1393    154
3530    215     3831    17              11      28      181700  1349    156
3536    194     3823    13              13      20      181513  1311    151
3555    194     3844    11              12      16      181908  1363    190
3404    214     3724    18              24      22      184206  1340    166
3579    248     3910    8               13      16      185521  1343    171
3568    197     3857    11              23      18      185978  1330    133
3497    212     3792    13              11      15      186207  1266    148
3646    214     3957    11              11      15      184140  1346    187
3704    206     4005    13              12      19      185284  1279    133
3526    199     3838    20              17      22      188028  1261    143
3706    204     4012    21              12      26      186988  1361    158
3626    211     3921    11              12      20      186614  1325    178
3795    244     4136    21              12      39      188584  1386    177
3653    221     3967    16              16      30      188447  1337    164
3781    231     4121    21              17      30      188792  1386    178
3751    208     4052    18              17      28      191729  1296    163
3784    229     4105    10              22      20      190018  1357    160
3597    179     3876    12              25      20      191238  1315    179
3606    206     3906    14              15      18      192253  1310    133
3653    205     3962    23              17      31      192431  1385    188
3708    192     3998    20              19      24      192258  1344    159
3595    192     3887    15              35      19      192045  1305    156
3541    220     3861    22              35      30      192387  1289    143
3575    215     3896    19              40      25      192800  1285    138
3708    340     4145    14              40      25      193867  1354    160
3540    513     4147    17              29      24      193792  1369    189
3610    213     3924    19              52      25      191956  1362    164
3613    238     3935    10              42      22      192883  1355    167
3585    209     3884    14              39      19      193032  1370    140
3465    214     3778    14              34      27      191302  1329    164
3324    190     3606    12              30      16      190908  1365    165
3448    180     3697    8               36      13      189965  1320    163
3463    202     3730    13              40      14      191013  1321    150
3397    198     3696    15              22      26      192505  1342    177
3311    146     3542    10              30      15      191154  1350    161
3272    132     3464    11              25      20      189987  1319    147
3380    142     3607    13              25      21      190706  1357    154
3222    142     3440    8               27      19      191109  1378    174
3185    130     3386    14              34      17      190889  1365    168
3207    131     3423    12              27      21      191843  1299    124
3107    141     3313    13              24      20      190705  1326    161
3179    126     3395    12              24      16      187994  1341    132
3051    121     3254    12              29      22      185939  1374    163
2961    123     3155    8               31      9       184144  1297    119
2949    137     3155    10              16      20      184098  1481    191
1919    74      2036    4                       5       123669  902     103

Now for the fun part, the erronious data that I mentioned at the begining isn't always there and I have NO idea when it will show. I hope I have explained this without confusing everyone.
Last edited by fsanchez; 06-18-2013 at 02:26 PM.. Reason: Needed to fix the script file
# 2  
Old 06-18-2013
your issue may be with your data ... confirm what are in samp12c, samp12.txt, samp12a, samp12b.txt and samp12.txt as soon as you get the erroneous data and then also check the file containing the raw data for anything that seems out of place ...
# 3  
Old 06-18-2013
Quote:
Originally Posted by Just Ice
your issue may be with your data ... confirm what are in samp12c, samp12.txt, samp12a, samp12b.txt and samp12.txt as soon as you get the erroneous data and then also check the file containing the raw data for anything that seems out of place ...
My apologies!!! I just realized there was an error in the version of the script file that I posted. I have edited the script file code to be correct now.
Last edited by fsanchez; 06-18-2013 at 02:24 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Programming

Makefile missing include path Although the path exists and defined

i have make file which i try to make them generic but it keeps to compline it missing include directory this is the makefile : CXX=g++ CPPFAGS= -Wall -O0 -g -std=c++14 INCLUDES = -I/home/vagrant/libuv/include -Isrc LIBS_DIRS = -L/home/vagrant/libuv/build LDFLAGS=... (7 Replies)
Discussion started by: umen
7 Replies

2. Shell Programming and Scripting

Modify blocks of text by printing missing columns

Hi Experts, I have a problem where I want to print missing columns (3,4) within a block of text. Each block is separated by "###". Some rows have missing column 3 and 4 which should be same as the previous value in column 3 and 4. The file is space delimited. For example: INPUT ###... (5 Replies)
Discussion started by: mira
5 Replies

3. UNIX for Dummies Questions & Answers

Modify Column Data using Shell Script

HI Guys, Input :- P081 wr1 12p0d5: 22.8 P081 wr1 12p2d18: 23.1 P149 wr1 1pxcud6/port_0_dev_7: 20.4 P149 wr1 1pxcud4/port_1_dev_10: 22.4 OutputP081 wr1 120 22.8 P081 wr1 122 23.1 P149 wr1 10 20.4 P149 wr1 11 22.4 In in First two line delete p and after d untill : In Last two line... (4 Replies)
Discussion started by: pareshkp
4 Replies

4. UNIX for Advanced & Expert Users

Checking missing data's sequence (shell script | UNIX command)

Dear All members, i have some trouble here, i want to ask your help. The case is: I have some data, it's like: -ABCD1234 -ABCD1235 -ABCD1237 -BCDE1111 -BCDE1112 -BCDE1114 there is some missing data's sequence (the format is: ABCD = name 1234 = sequence). I want to print the... (2 Replies)
Discussion started by: septian.tri
2 Replies

5. UNIX for Dummies Questions & Answers

How to modify the script to include cat function

I have the following script that greps lines containing "AT" from data files data1.hsq through data1000.hsq, then cuts their second column and puts in data files called perm1 through perm1000. I want to modify the script so that instead of putting the data in separate data files perm1 through... (2 Replies)
Discussion started by: evelibertine
2 Replies

6. Shell Programming and Scripting

Substitute partial data only within a xml tag

Hello, I have huge xml files and I need to replace only part of the data within a particular xml tag. This doesnt seem to be as simple as it sounds. I have searched everywhere and couldnt find any solution. Ex: In the below case I would like "def" to be replaced by "xyz" only when found in... (8 Replies)
Discussion started by: roshanjain2
8 Replies

7. Shell Programming and Scripting

using awk to substitute data in a column delimited text file

using awk to substitute data in a column delimited text file hello i would like to use awk to do the following calculation from the following snippet. input file C;2390 ;CV BOUILLOTTE 2L 2FACES NERVUREES ;1.00 ;3552612239004;13417 ;25 ;50 ; 12;50000 ; ; ... (3 Replies)
Discussion started by: iindie
3 Replies

8. Shell Programming and Scripting

Read data from .csv file through shell script & modify

I need to read data from a file called "test.csv" through shell script where the file contains values like name,price,descriptor etc. There are rows where descriptor (& in some rows name) are written as string & other characters like "car_+" OR "bike*" etc where it should contains strings like... (3 Replies)
Discussion started by: raj100
3 Replies

9. Shell Programming and Scripting

Sed question to substitute data in \2

Hi All, Here is what I'm trying to do with sed: Input File: somechangeVariable1=Something I would like to change somechangeVariable2=Something else I would like to change ... Output File: somechangeVariable1=Something I would like to different somechangeVariable2=Something else I would... (6 Replies)
Discussion started by: Peace_Dude1
6 Replies

10. Shell Programming and Scripting

how to include the missing column in the original file using awk

Hi Experts, The content of the raw file: date,nomsgsent,nomsgnotdeliver,nomsgdelay 201003251000,1000,1,2 201003251000,900,0,0 201003251000,1450,0,0 201003251000,1230,0,0 However, sometimes, the column will missing in the raw files: e.g. date,nomsgsent,nomsgdelay... (8 Replies)
Discussion started by: natalie23
8 Replies
Login or Register to Ask a Question