Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Use sed/awk to do like copy and paste

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Use sed/awk to do like copy and paste
# 1  
Old 05-19-2016
Use sed/awk to do like copy and paste
I have rrd file which is have the gaps and I want to fill it out with some value , I've got 10 NaN record and I try to populate data from 10 records be for NaN to change instead of NaN Smilie


Code:
                        <!-- 2016-05-19 14:10:00 CST / 1463638200 --> <row><v>8.2147341249e+06</v><v>1.1121553155e+07</v></row>
                        <!-- 2016-05-19 14:15:00 CST / 1463638500 --> <row><v>8.2746054049e+06</v><v>1.1065925833e+07</v></row>
                        <!-- 2016-05-19 14:20:00 CST / 1463638800 --> <row><v>8.5873130347e+06</v><v>1.1330819417e+07</v></row>
                        <!-- 2016-05-19 14:25:00 CST / 1463639100 --> <row><v>8.7400510244e+06</v><v>1.1466312671e+07</v></row>
                        <!-- 2016-05-19 14:30:00 CST / 1463639400 --> <row><v>8.8934198478e+06</v><v>1.1612750075e+07</v></row>
                        <!-- 2016-05-19 14:35:00 CST / 1463639700 --> <row><v>8.6970836405e+06</v><v>1.1451382750e+07</v></row>
                        <!-- 2016-05-19 14:40:00 CST / 1463640000 --> <row><v>8.6519721561e+06</v><v>1.1490998652e+07</v></row>
                        <!-- 2016-05-19 14:45:00 CST / 1463640300 --> <row><v>8.7016410558e+06</v><v>1.1758066110e+07</v></row>
                        <!-- 2016-05-19 14:50:00 CST / 1463640600 --> <row><v>8.7026601272e+06</v><v>1.1808612970e+07</v></row>
                        <!-- 2016-05-19 14:55:00 CST / 1463640900 --> <row><v>8.8925026101e+06</v><v>1.2043713450e+07</v></row>
                        <!-- 2016-05-19 15:00:00 CST / 1463641200 --> <row><v>8.7455789370e+06</v><v>1.2420369632e+07</v></row>
                        <!-- 2016-05-19 15:05:00 CST / 1463641500 --> <row><v>8.6264080619e+06</v><v>1.2602318595e+07</v></row>
                        <!-- 2016-05-19 15:10:00 CST / 1463641800 --> <row><v>8.8378430034e+06</v><v>1.3041342755e+07</v></row>
                        <!-- 2016-05-19 15:15:00 CST / 1463642100 --> <row><v>NaN</v><v>NaN</v></row>
                        <!-- 2016-05-19 15:20:00 CST / 1463642400 --> <row><v>NaN</v><v>NaN</v></row>
                        <!-- 2016-05-19 15:25:00 CST / 1463642700 --> <row><v>NaN</v><v>NaN</v></row>
                        <!-- 2016-05-19 15:30:00 CST / 1463643000 --> <row><v>NaN</v><v>NaN</v></row>
                        <!-- 2016-05-19 15:35:00 CST / 1463643300 --> <row><v>NaN</v><v>NaN</v></row>
                        <!-- 2016-05-19 15:40:00 CST / 1463643600 --> <row><v>NaN</v><v>NaN</v></row>
                        <!-- 2016-05-19 15:45:00 CST / 1463643900 --> <row><v>NaN</v><v>NaN</v></row>
                        <!-- 2016-05-19 15:50:00 CST / 1463644200 --> <row><v>NaN</v><v>NaN</v></row>
                        <!-- 2016-05-19 15:55:00 CST / 1463644500 --> <row><v>NaN</v><v>NaN</v></row>
                        <!-- 2016-05-19 16:00:00 CST / 1463644800 --> <row><v>NaN</v><v>NaN</v></row>
                        <!-- 2016-05-19 16:05:00 CST / 1463645100 --> <row><v>9.4998580882e+06</v><v>1.3563528239e+07</v></row>
                        <!-- 2016-05-19 16:10:00 CST / 1463645400 --> <row><v>9.3834388158e+06</v><v>1.3154323934e+07</v></row>


expected result shoule be with out NaN , actually the reson behind this is need to get rid of the gaps when plot the graph..
Code:
                        <!-- 2016-05-19 13:50:00 CST / 1463637000 --> <row><v>7.8650237427e+06</v><v>1.1633105231e+07</v></row>
                        <!-- 2016-05-19 13:55:00 CST / 1463637300 --> <row><v>7.7744480110e+06</v><v>1.1049584241e+07</v></row>
                        <!-- 2016-05-19 14:00:00 CST / 1463637600 --> <row><v>8.0235789005e+06</v><v>1.1089589128e+07</v></row>
                        <!-- 2016-05-19 14:05:00 CST / 1463637900 --> <row><v>8.1035284930e+06</v><v>1.1177274228e+07</v></row>
                        <!-- 2016-05-19 14:10:00 CST / 1463638200 --> <row><v>8.2147341249e+06</v><v>1.1121553155e+07</v></row>
                        <!-- 2016-05-19 14:15:00 CST / 1463638500 --> <row><v>8.2746054049e+06</v><v>1.1065925833e+07</v></row>
                        <!-- 2016-05-19 14:20:00 CST / 1463638800 --> <row><v>8.5873130347e+06</v><v>1.1330819417e+07</v></row>
                        <!-- 2016-05-19 14:25:00 CST / 1463639100 --> <row><v>8.7400510244e+06</v><v>1.1466312671e+07</v></row>
                        <!-- 2016-05-19 14:30:00 CST / 1463639400 --> <row><v>8.8934198478e+06</v><v>1.1612750075e+07</v></row>
                        <!-- 2016-05-19 14:35:00 CST / 1463639700 --> <row><v>8.6970836405e+06</v><v>1.1451382750e+07</v></row>
                        <!-- 2016-05-19 14:40:00 CST / 1463640000 --> <row><v>8.6519721561e+06</v><v>1.1490998652e+07</v></row>
                        <!-- 2016-05-19 14:45:00 CST / 1463640300 --> <row><v>8.7016410558e+06</v><v>1.1758066110e+07</v></row>
                        <!-- 2016-05-19 14:50:00 CST / 1463640600 --> <row><v>8.7026601272e+06</v><v>1.1808612970e+07</v></row>
                        <!-- 2016-05-19 14:55:00 CST / 1463640900 --> <row><v>8.8925026101e+06</v><v>1.2043713450e+07</v></row>
                        <!-- 2016-05-19 15:00:00 CST / 1463641200 --> <row><v>8.7455789370e+06</v><v>1.2420369632e+07</v></row>
                        <!-- 2016-05-19 15:05:00 CST / 1463641500 --> <row><v>8.6264080619e+06</v><v>1.2602318595e+07</v></row>
                        <!-- 2016-05-19 15:10:00 CST / 1463641800 --> <row><v>8.8378430034e+06</v><v>1.3041342755e+07</v></row>
                        <!-- 2016-05-19 15:15:00 CST / 1463642100 --> <row><v>8.7400510244e+06</v><v>1.1466312671e+07</v></row> --
                        <!-- 2016-05-19 15:20:00 CST / 1463642400 --> <row><v>8.8934198478e+06</v><v>1.1612750075e+07</v></row> --
                        <!-- 2016-05-19 15:25:00 CST / 1463642700 --> <row><v>8.6970836405e+06</v><v>1.1451382750e+07</v></row> --
                        <!-- 2016-05-19 15:30:00 CST / 1463643000 --> <row><v>8.6519721561e+06</v><v>1.1490998652e+07</v></row> --
                        <!-- 2016-05-19 15:35:00 CST / 1463643300 --> <row><v>8.7016410558e+06</v><v>1.1758066110e+07</v></row> --
                        <!-- 2016-05-19 15:40:00 CST / 1463643600 --> <row><v>8.7026601272e+06</v><v>1.1808612970e+07</v></row> --
                        <!-- 2016-05-19 15:45:00 CST / 1463643900 --> <row><v>8.8925026101e+06</v><v>1.2043713450e+07</v></row> --
                        <!-- 2016-05-19 15:50:00 CST / 1463644200 --> <row><v>8.7455789370e+06</v><v>1.2420369632e+07</v></row> --
                        <!-- 2016-05-19 15:55:00 CST / 1463644500 --> <row><v>8.6264080619e+06</v><v>1.2602318595e+07</v></row> -- 
                        <!-- 2016-05-19 16:00:00 CST / 1463644800 --> <row><v>8.8378430034e+06</v><v>1.3041342755e+07</v></row> --
                        <!-- 2016-05-19 16:05:00 CST / 1463645100 --> <row><v>9.4998580882e+06</v><v>1.3563528239e+07</v></row> 
                        <!-- 2016-05-19 16:10:00 CST / 1463645400 --> <row><v>9.3834388158e+06</v><v>1.3154323934e+07</v></row>

# 2  
Old 05-19-2016
Parsing XML is not trivial. Because of frequent requests, I've got an awk script which works in some common situations however.

Contents of xml.awk:
Code:
BEGIN {
        FS=">"; OFS=">";
        RS="<"; ORS="<"
}

# These should be special variables for match() but aren't.
function rbefore(STR)   { return(substr(STR, N, RSTART-1)); }# before match
function rmid(STR)      { return(substr(STR, RSTART, 1)); }  # First char match
function rall(STR)      { return(substr(STR, RSTART, RLENGTH)); }# Entire match
function rafter(STR)    { return(substr(STR, RSTART+RLENGTH)); }# after match

function aquote(OUT, A, PFIX, TA) { # Turns Q SUBSEP R into A[PFIX":"Q]=R
        if(OUT)
        {
                if(PFIX) PFIX=PFIX":"
                split(OUT, TA, SUBSEP);
                A[toupper(PFIX) toupper(TA[1])]=TA[2];
        }

        return("");
}

# Intended to be less stupid about quoted text in XML/HTML.
# Splits a='b' c='d' e='f' into A[PFIX":"a]=b, A[PFIX":"c]=d, etc.
function qsplit(STR, A, PFIX, X, OUT) {
        while(STR && match(STR, /([ \n\t]+)|[\x27\x22=]/))
        {
                OUT = OUT rbefore(STR);
                RMID=rmid(STR);

                if((RMID == "'") || (RMID == "\""))     # Quote characters
                {
                        if(!Q)          Q=RMID;         # Begin quote section
                        else if(Q == RMID)      Q="";   # End quote section
                        else                    OUT = OUT RMID; # Quoted quote
                } else if(RMID == "=") {
                        if(Q)   OUT=OUT RMID; else OUT=OUT SUBSEP;
                } else if((RMID=="\r")||(RMID=="\n")||(RMID=="\t")||(RMID==" ")) {
                        if(Q)   OUT = OUT rall(STR); # Literal quoted whitespace
                        else    OUT = aquote(OUT, A, PFIX); # Unquoted WS, next block
                }
                STR=rafter(STR); # Strip off the text we've processed already.
        }

        aquote(OUT STR, A, PFIX); # Process any text we haven't already.
}


{ SPEC=0 ; TAG="" }

NR==1 {
        if(ORS == RS) print;
        next } # The first "line" is blank when RS=<

/^[!?]/ { SPEC=1    }   # XML specification junk

# Handle open-tags
match($1, /^[^\/ \r\n\t>]+/) {
        TAG=substr(toupper($1), RSTART, RLENGTH);
        if((!SPEC) && !($1 ~ /\/$/))
        {
                TAGS=TAG "%" TAGS;
                DEP++;
                LTAGS=TAGS
        }

        for(X in ARGS) delete ARGS[X];

        qsplit(rafter($1), ARGS, "", "", "");
}

# Handle close-tags
(!SPEC) && /^[\/]/ {
        sub(/^\//, "", $1);
        LTAGS=TAGS
#        sub("^.*" toupper($1) "%", "", TAGS);
        sub("^" toupper($1) "%", "", TAGS);
        $1="/"$1
        DEP=split(TAGS, TA, "%")-1;
        if(DEP < 0) DEP=0;
}

How to use it in this case:
Code:
$ awk -f xml.awk -e 'BEGIN { P[0]=0; P[1]=0; ORS="" }
                /^row>/ { D=0 }
                /^v>/ {
                        if($2 == "NaN") $2=P[D];
                        else            P[D]=$2;
                        D++;
                }
                { print "<" $0; }' datafile

<!-- 2016-05-19 14:10:00 CST / 1463638200 --> <row><v>8.2147341249e+06</v><v>1.1121553155e+07</v></row>
                        <!-- 2016-05-19 14:15:00 CST / 1463638500 --> <row><v>8.2746054049e+06</v><v>1.1065925833e+07</v></row>
                        <!-- 2016-05-19 14:20:00 CST / 1463638800 --> <row><v>8.5873130347e+06</v><v>1.1330819417e+07</v></row>
                        <!-- 2016-05-19 14:25:00 CST / 1463639100 --> <row><v>8.7400510244e+06</v><v>1.1466312671e+07</v></row>
                        <!-- 2016-05-19 14:30:00 CST / 1463639400 --> <row><v>8.8934198478e+06</v><v>1.1612750075e+07</v></row>
                        <!-- 2016-05-19 14:35:00 CST / 1463639700 --> <row><v>8.6970836405e+06</v><v>1.1451382750e+07</v></row>
                        <!-- 2016-05-19 14:40:00 CST / 1463640000 --> <row><v>8.6519721561e+06</v><v>1.1490998652e+07</v></row>
                        <!-- 2016-05-19 14:45:00 CST / 1463640300 --> <row><v>8.7016410558e+06</v><v>1.1758066110e+07</v></row>
                        <!-- 2016-05-19 14:50:00 CST / 1463640600 --> <row><v>8.7026601272e+06</v><v>1.1808612970e+07</v></row>
                        <!-- 2016-05-19 14:55:00 CST / 1463640900 --> <row><v>8.8925026101e+06</v><v>1.2043713450e+07</v></row>
                        <!-- 2016-05-19 15:00:00 CST / 1463641200 --> <row><v>8.7455789370e+06</v><v>1.2420369632e+07</v></row>
                        <!-- 2016-05-19 15:05:00 CST / 1463641500 --> <row><v>8.6264080619e+06</v><v>1.2602318595e+07</v></row>
                        <!-- 2016-05-19 15:10:00 CST / 1463641800 --> <row><v>8.8378430034e+06</v><v>1.3041342755e+07</v></row>
                        <!-- 2016-05-19 15:15:00 CST / 1463642100 --> <row><v>8.8378430034e+06</v><v>1.3041342755e+07</v></row>
                        <!-- 2016-05-19 15:20:00 CST / 1463642400 --> <row><v>8.8378430034e+06</v><v>1.3041342755e+07</v></row>
                        <!-- 2016-05-19 15:25:00 CST / 1463642700 --> <row><v>8.8378430034e+06</v><v>1.3041342755e+07</v></row>
                        <!-- 2016-05-19 15:30:00 CST / 1463643000 --> <row><v>8.8378430034e+06</v><v>1.3041342755e+07</v></row>
                        <!-- 2016-05-19 15:35:00 CST / 1463643300 --> <row><v>8.8378430034e+06</v><v>1.3041342755e+07</v></row>
                        <!-- 2016-05-19 15:40:00 CST / 1463643600 --> <row><v>8.8378430034e+06</v><v>1.3041342755e+07</v></row>
                        <!-- 2016-05-19 15:45:00 CST / 1463643900 --> <row><v>8.8378430034e+06</v><v>1.3041342755e+07</v></row>
                        <!-- 2016-05-19 15:50:00 CST / 1463644200 --> <row><v>8.8378430034e+06</v><v>1.3041342755e+07</v></row>
                        <!-- 2016-05-19 15:55:00 CST / 1463644500 --> <row><v>8.8378430034e+06</v><v>1.3041342755e+07</v></row>
                        <!-- 2016-05-19 16:00:00 CST / 1463644800 --> <row><v>8.8378430034e+06</v><v>1.3041342755e+07</v></row>
                        <!-- 2016-05-19 16:05:00 CST / 1463645100 --> <row><v>9.4998580882e+06</v><v>1.3563528239e+07</v></row>
                        <!-- 2016-05-19 16:10:00 CST / 1463645400 --> <row><v>9.3834388158e+06</v><v>1.3154323934e+07</v></row>

$

This User Gave Thanks to Corona688 For This Post:
boobytrap
# 3  
Old 05-19-2016
I have this after try to simulate

Code:
awk: xml.awk:3: fatal: cannot open file `-e' for reading (No such file or directory)

# 4  
Old 05-19-2016
Append these lines to the end of xml.awk:

Code:
BEGIN { P[0]=0; P[1]=0; ORS="" }
                /^row>/ { D=0 }
                /^v>/ {
                        if($2 == "NaN") $2=P[D];
                        else            P[D]=$2;
                        D++;
                }
                { print "<" $0; }

...then just run awk -f xml.awk datafile
This User Gave Thanks to Corona688 For This Post:
boobytrap
# 5  
Old 05-19-2016
I just realized that my awk bundle with Ubuntu 12.04 GNU Awk 3.1.8 after compile with new version GNU Awk 4.1.3, API: 1.1 your script work like a charm.
This User Gave Thanks to boobytrap For This Post:
Corona688
# 6  
Old 05-19-2016
Sorry about that, ubuntu uses mawk by default these days as it's faster.

It should work with nearly any version of GNU awk. It even works with busybox's minimal awk. But mawk lacks -e, annoyingly.

You can work around that by just adding the lines to the file as noted above.
# 7  
Old 05-19-2016
Is it possible to filter only from 2016-05-19 15:15:00 - 2016-05-19 15:55:00 CST ? , haven't look in to script and understand how it really work yet .

Follow your suggestion all NaN in rrd dump file will change to some value.

Code:
                        <!-- 2016-05-19 15:15:00 CST / 1463642100 --> <row><v>NaN</v><v>NaN</v></row>
                        <!-- 2016-05-19 15:20:00 CST / 1463642400 --> <row><v>NaN</v><v>NaN</v></row>
                        <!-- 2016-05-19 15:25:00 CST / 1463642700 --> <row><v>NaN</v><v>NaN</v></row>
                        <!-- 2016-05-19 15:30:00 CST / 1463643000 --> <row><v>NaN</v><v>NaN</v></row>
                        <!-- 2016-05-19 15:35:00 CST / 1463643300 --> <row><v>NaN</v><v>NaN</v></row>
                        <!-- 2016-05-19 15:40:00 CST / 1463643600 --> <row><v>NaN</v><v>NaN</v></row>
                        <!-- 2016-05-19 15:45:00 CST / 1463643900 --> <row><v>NaN</v><v>NaN</v></row>
                        <!-- 2016-05-19 15:50:00 CST / 1463644200 --> <row><v>NaN</v><v>NaN</v></row>
                        <!-- 2016-05-19 15:55:00 CST / 1463644500 --> <row><v>NaN</v><v>NaN</v></row>
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Using SED to copy/paste with slashes and tabs.

I have: 2012/01_January/Kite/foldername/otherfoldername/placeholderBlue I want to end up with: /foldername/otherfoldername/2012/01_January/Kite/Blue Basically take everything before the first tab and put it in the place of the consistently named word placeholder and add a slash in place... (2 Replies)
Discussion started by: crowman
2 Replies

2. Shell Programming and Scripting

sed help - search/copy from one file and search/paste to another

I am a newbie and would like some help with the following - Trying to search fileA for a string similar to - AS11000022010 30.4 31.7 43.7 53.8 60.5 71.1 75.2 74.7 66.9 56.6 42.7 32.5 53.3 I then want to replace that string with a string from fileB - ... (5 Replies)
Discussion started by: ncwxpanther
5 Replies

3. Shell Programming and Scripting

Using a combination of sort/cut/grep/awk/join/paste/sed

I have a file and need to only select users that have a shell of “/bin/bash” in the line using awk or sed please help (4 Replies)
Discussion started by: boyboy1212
4 Replies

4. Shell Programming and Scripting

Copy and Paste to a new document

Hello, I am quite new to shell scripting so don't know all the tools available. What I'm trying to do is open a file optimal.txt search for objectiveValue and copy the number in quotes next to it. e.g. ... solutionName="incumbent" solutionIndex="-1" objectiveValue="13246" ... (6 Replies)
Discussion started by: StephanR
6 Replies

5. UNIX for Dummies Questions & Answers

vi copy/paste problem

I'm having a problem copy/pasting from a txt file in windows to vi. What happens is I copy a chunk of text, go to the putty terminal, go into insert mode, and right click, and it will stop pasting at a random point and freeze up. Nothing I do gets out of it. This only happens on my account... (1 Reply)
Discussion started by: solidarity
1 Replies

6. Shell Programming and Scripting

sed copy paste

Hello, I have this path and file: /dir/dir/dir/dir/dir/dir/dir/dir/dir/THIS_SPOT/fle.txt I want to end up with: /dir/dir/dir/dir/dir/dir/dir/dir/dir/THIS_SPOT/fle.txtTHIS_SPOT Take the dir after the 10th slash, add a tab at the end and paste the dir it copied. Thanks (4 Replies)
Discussion started by: crowman
4 Replies

7. Shell Programming and Scripting

copy/paste with awk

Hi everybody, I have two XML files. I am working on a script that could copy and paste the contents of the first xml file to the desired location in the second xml file. Here is my first XML file. This is the second XML file. Finaly, I wnat to obtain something like that : ... (2 Replies)
Discussion started by: lsaas
2 Replies

8. Shell Programming and Scripting

awk/grep copy and paste and insert in between lines.

Hi all, I'm a unix newb andI'm trying to write a script that can copy some text paste it in a certian place and then add a number. It's not really clear but I'll show an example. what the file looks like right now: Linux 2.6.24-24-generic (abc) 07/15/09 23:25:01 CPU ... (6 Replies)
Discussion started by: the1hand3r
6 Replies

9. Shell Programming and Scripting

how to replace paste utility with sed/awk

hi I have two file that I would like to paste line by line. I use to use paste for this. But my current platform will not have paste utility. Could anyone please suggest how I it can be done using sed,awk or even simple bourne shell scripts? Thanks Sabina (1 Reply)
Discussion started by: ssayeed
1 Replies

10. UNIX for Dummies Questions & Answers

awk or sed or paste

i have a file and i need the text to line up currently the file looks like so job scheduled complete 12 12:00 wendsday 13 1:00 wednsday its a text file but not sure how to manipulate the file for it to line up (3 Replies)
Discussion started by: leprichaun
3 Replies
Login or Register to Ask a Question