Sponsored Content
Top Forums UNIX for Beginners Questions & Answers Multiline html tag parse shell script Post 303044115 by stomp on Friday 14th of February 2020 04:51:23 AM
Old 02-14-2020
Hi,

here's a suggestion using pup, a html-parser written in go:

Code:
pup 'div p text{}' < data.html

# Output:

        text1
        

        text2
        

        text3

Explanation: Get all p-Elements with div-elements as parents and output the text data of it.

To get rid of the empty lines, I suggest a small sed command afterwards:

Code:
pup 'div p text{}' < data.html | sed '/^\s*$/d'

# Output
        text1
        text2
        text3

Another short demonstration of pup which I shortly used to get the numbers of cases for the coronovirus out of a complex website and into variables(for generating this graph: coronavirus statistics)) with only one combined command:

Code:
 read n n n n infected deceased recovered < <(wget -O- -q https://www.worldometers.info/coronavirus/  \
       | pup 'div[id="maincounter-wrap"]' | pup 'h1,span text{}' | xargs echo)

Pup is found here: pup on Github

As all GO binaries, it's statically linked and quite large in size(4 MB). Precompiled Binaries are available on github(link above).

Last edited by stomp; 02-14-2020 at 08:32 AM..
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

How do I extract text only from html file without HTML tag

I have a html file called myfile. If I simply put "cat myfile.html" in UNIX, it shows all the html tags like <a href=r/26><img src="http://www>. But I want to extract only text part. Same problem happens in "type" command in MS-DOS. I know you can do it by opening it in Internet Explorer,... (4 Replies)
Discussion started by: los111
4 Replies

2. Shell Programming and Scripting

how to use html tag in shell scripting

Hai friends I have a small doubt.. how can we use html tag in shell scripting code : echo "<html>" echo "<body>" echo " welcome to peace world " echo "</body>" echo "</html>" output displayed like this: <html> <body> welcome to peace world </body> </html> (5 Replies)
Discussion started by: jrex1983
5 Replies

3. UNIX for Advanced & Expert Users

shell script to parse html file

hi all, i have a html file something similar to this. <tr class="evenrow"> <td class="data">added</td><td class="data">xyz@abc.com</td> <td class="data">filename.sql</td><td class="modifications-data">08/25/2009 07:58:40</td><td class="data">Added TK prof script</td> </tr> <tr... (1 Reply)
Discussion started by: sais
1 Replies

4. Shell Programming and Scripting

Parse HTML tag parameters and text

Hi! I have a bunch of HTML files, which I want to parse to CSV files. Every page has a table in it, and I need to parse each row into a csv record. With awk and sed, I managed to put every table row in separate lines. So my file looks like this: <TR> .... </TR> <TR> .... </TR> ...One... (1 Reply)
Discussion started by: senszey
1 Replies

5. Shell Programming and Scripting

Script to delete HTML tag

Guys, I have a little script that I got of the internet and that I use in Squid to block ads. I used that script with linux but now i have moved my servers to freebsd. I have a step learning curve there but it is fun: Back to the script issue. The script used to work i with linux but... (15 Replies)
Discussion started by: zongo
15 Replies

6. Shell Programming and Scripting

awk Script to parse a XML tag

I have an XML tag like this: <property name="agent" value="/var/tmp/root/eclipse" /> Is there way using awk that i can get the value from the above tag. So the output should be: /var/tmp/root/eclipse Help will be appreciated. Regards, Adi (6 Replies)
Discussion started by: asirohi
6 Replies

7. Shell Programming and Scripting

Search for a html tag and print the entire tag

I want to print from <fruits> to </fruits> tag which have <fruit> as mango. Also i want both <fruits> and </fruits> in output. Please help eg. <fruits> <fruit id="111">mango<fruit> . another 20 lines . </fruits> (3 Replies)
Discussion started by: Ashik409
3 Replies

8. Shell Programming and Scripting

Using shell command need to parse multiple nested tag value of a XML file

I have this XML file - <gp> <mms>1110012</mms> <tg>988</tg> <mm>LongTime</mm> <lv> <lkid>StartEle=ONE, Desti = Motion</lkid> <kk>12</kk> </lv> <lv> <lkid>StartEle=ONE, Source = Velocity</lkid> <kk>2</kk> </lv> <lv> ... (3 Replies)
Discussion started by: NeedASolution
3 Replies

9. Shell Programming and Scripting

XML Parse between to tag with upper tag

Hi Guys Here is my Input : <?xml version="1.0" encoding="UTF-8"?> <xn:MeContext id="01736"> <xn:VsDataContainer id="01736"> <xn:attributes> <xn:vsDataType>vsDataMeContext</xn:vsDataType> ... (12 Replies)
Discussion started by: pareshkp
12 Replies

10. Shell Programming and Scripting

How to remove html tag which has multiple lines in SHELL?

I want to clean a html file. I try to remove the script part in the html and remove the rest of tags and empty lines. The code I try to use is the following: sed '/<script/,/<\/script>/d' webpage.html | sed -e 's/<*>//g' | sed '/^\s*$/d' > output.txt However, in this method, I can not... (10 Replies)
Discussion started by: YuhuiFeng
10 Replies
HXNORMALIZE(1)							  HTML-XML-utils						    HXNORMALIZE(1)

NAME
hxnormalize - pretty-print an HTML file SYNOPSIS
hxnormalize [ -x ] [ -e ] [ -d ] [ -s ] [ -L ] [ -i indent ] [ -l line-length ] [ -c commentmagic ] [ file-or-URL ] DESCRIPTION
The hxnormalize command pretty-prints an HTML file, and also tries to fix small errors. The output is the same HTML, but with a maximum line length and with optional indentation to indicate the nesting level of each line. OPTIONS
The following options are supported: -x Use XML conventions: empty elements are written with a slash at the end: <IMG />. Implies -e. -e Always insert endtags, even if HTML does not require them (for example: </p> and </li>). -d Omit the DOCTYPE from the output. -i indent Set the number of spaces to indent each nesting level. Default is 2. Not all elements cause an indent. In general, elements that can occur in a block environment are started on a new line and cause an indent, but inline elements, such as EM and SPAN do not cause an indent. -l line-length Sets the maximum length of lines. hxnormalize will wrap lines so that all lines are as long as possible, but no longer than this length. Default is 72. Words that are longer than the line length will not be broken, and will extend past this length. A content of the STYLE, SCRIPT and PRE elements will not be line-wrapped. -s Omit <span> tags that don't have any attributes. -L Remove redundant "lang" and "xml:lang" attributes. (I.e., those whose value is the same as the language inherited from the parent element.) -c commentmagic Comments are normally placed right after the preceding text. That is usually correct for short comments, but some comments are meant to be on a separate line. commentmagic is a string and when that string occurs inside a comment, hxnormalize will output an empty line before that comment. E.g. -c "====" can be used to put all comments that contain "====" on a separate line, pre- ceded by an empty line. By default, no comments are treated that way. OPERANDS
The following operand is supported: file-or-URL The name or URL of an HTML file. If absent, standard input is read instead. EXIT STATUS
The following exit values are returned: 0 Successful completion. > 0 An error occurred in the parsing of the HTML file. hxnormalize will try to correct the error and produce output anyway. ENVIRONMENT
To use a proxy to retrieve remote files, set the environment variables http_proxy and ftp_proxy. E.g., http_proxy="http://localhost:8080/" BUGS
The error recovery for incorrect HTML is primitive. hxnormalize will not omit an endtag if the white space after it could possibly be significant. E.g., it will not remove the first </p> from "<div><p>text</p> <p>text</p></div>". hxnormalize can currently only retrieve remote files over HTTP. It doesn't handle password-protected files, nor files whose content depends on HTTP "cookies." SEE ALSO
asc2xml(1), xml2asc(1), UTF-8 (RFC 2279) 6.x 10 Jul 2011 HXNORMALIZE(1)
All times are GMT -4. The time now is 01:36 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy