Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Multiline html tag parse shell script
Top Forums UNIX for Beginners Questions & Answers Multiline html tag parse shell script Post 303040234 by stomp on Friday 25th of October 2019 08:09:45 AM
Old 10-25-2019
Since HTML is very similar to XML, you may use an xml tool to parse your file.

Since your HTML-File is not fully standards compliant, the parser complains about it and the file has either be adapted by hand to be compliant or to be preprocessed prior to the parsing. The <br> is the problematic element. Compliant would be <br/> with a slash within the tag.

So you can do it with an xmlparser like xmlstarlet in three steps:

Code:
sed 's|<br>|<br/>|gi' data.html      |
  xmlstarlet sel -t -v '//body/div/p'   |
  sed -e '/^\s*$/d' -e 's/^\s*//'

1. Make the html file compliant by replacing the br-Tags
2. Get the wanted HTML-Element with xmlstarlet
3. suppress unwanted empty lines and leading whitespace in data / xmlstarlet output
Last edited by stomp; 10-25-2019 at 09:31 AM..
These 2 Users Gave Thanks to stomp For This Post:
drl SorcRR
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

How do I extract text only from html file without HTML tag

I have a html file called myfile. If I simply put "cat myfile.html" in UNIX, it shows all the html tags like <a href=r/26><img src="http://www>. But I want to extract only text part. Same problem happens in "type" command in MS-DOS. I know you can do it by opening it in Internet Explorer,... (4 Replies)
Discussion started by: los111
4 Replies

2. Shell Programming and Scripting

how to use html tag in shell scripting

Hai friends I have a small doubt.. how can we use html tag in shell scripting code : echo "<html>" echo "<body>" echo " welcome to peace world " echo "</body>" echo "</html>" output displayed like this: <html> <body> welcome to peace world </body> </html> (5 Replies)
Discussion started by: jrex1983
5 Replies

3. UNIX for Advanced & Expert Users

shell script to parse html file

hi all, i have a html file something similar to this. <tr class="evenrow"> <td class="data">added</td><td class="data">xyz@abc.com</td> <td class="data">filename.sql</td><td class="modifications-data">08/25/2009 07:58:40</td><td class="data">Added TK prof script</td> </tr> <tr... (1 Reply)
Discussion started by: sais
1 Replies

4. Shell Programming and Scripting

Parse HTML tag parameters and text

Hi! I have a bunch of HTML files, which I want to parse to CSV files. Every page has a table in it, and I need to parse each row into a csv record. With awk and sed, I managed to put every table row in separate lines. So my file looks like this: <TR> .... </TR> <TR> .... </TR> ...One... (1 Reply)
Discussion started by: senszey
1 Replies

5. Shell Programming and Scripting

Script to delete HTML tag

Guys, I have a little script that I got of the internet and that I use in Squid to block ads. I used that script with linux but now i have moved my servers to freebsd. I have a step learning curve there but it is fun: Back to the script issue. The script used to work i with linux but... (15 Replies)
Discussion started by: zongo
15 Replies

6. Shell Programming and Scripting

awk Script to parse a XML tag

I have an XML tag like this: <property name="agent" value="/var/tmp/root/eclipse" /> Is there way using awk that i can get the value from the above tag. So the output should be: /var/tmp/root/eclipse Help will be appreciated. Regards, Adi (6 Replies)
Discussion started by: asirohi
6 Replies

7. Shell Programming and Scripting

Search for a html tag and print the entire tag

I want to print from <fruits> to </fruits> tag which have <fruit> as mango. Also i want both <fruits> and </fruits> in output. Please help eg. <fruits> <fruit id="111">mango<fruit> . another 20 lines . </fruits> (3 Replies)
Discussion started by: Ashik409
3 Replies

8. Shell Programming and Scripting

Using shell command need to parse multiple nested tag value of a XML file

I have this XML file - <gp> <mms>1110012</mms> <tg>988</tg> <mm>LongTime</mm> <lv> <lkid>StartEle=ONE, Desti = Motion</lkid> <kk>12</kk> </lv> <lv> <lkid>StartEle=ONE, Source = Velocity</lkid> <kk>2</kk> </lv> <lv> ... (3 Replies)
Discussion started by: NeedASolution
3 Replies

9. Shell Programming and Scripting

XML Parse between to tag with upper tag

Hi Guys Here is my Input : <?xml version="1.0" encoding="UTF-8"?> <xn:MeContext id="01736"> <xn:VsDataContainer id="01736"> <xn:attributes> <xn:vsDataType>vsDataMeContext</xn:vsDataType> ... (12 Replies)
Discussion started by: pareshkp
12 Replies

10. Shell Programming and Scripting

How to remove html tag which has multiple lines in SHELL?

I want to clean a html file. I try to remove the script part in the html and remove the rest of tags and empty lines. The code I try to use is the following: sed '/<script/,/<\/script>/d' webpage.html | sed -e 's/<*>//g' | sed '/^\s*$/d' > output.txt However, in this method, I can not... (10 Replies)
Discussion started by: YuhuiFeng
10 Replies

LEARN ABOUT REDHAT

html::tagset

Tagset(3)						User Contributed Perl Documentation						 Tagset(3)

NAME

       HTML::Tagset - data tables useful in parsing HTML

SYNOPSIS

	 use HTML::Tagset;
	 # Then use any of the items in the HTML::Tagset package
	 #  as need arises

DESCRIPTION

       This module contains several data tables useful in various kinds of HTML parsing operations.

       Note that all tag names used are lowercase.

       In the following documentation, a "hashset" is a hash being used as a set -- the hash conveys that its keys are there, and the actual val-
       ues associated with the keys are not significant.  (But what values are there, are always true.)

       hashset %HTML::Tagset::emptyElement
	   This hashset has as values the tag-names (GIs) of elements that cannot have content.  (For example, "base", "br", "hr".)  So
	   $HTML::Tagset::emptyElement{'hr'} exists and is true.  $HTML::Tagset::emptyElement{'dl'} does not exist, and so is not true.

       hashset %HTML::Tagset::optionalEndTag
	   This hashset lists tag-names for elements that can have content, but whose end-tags are generally, "safely", omissible.  Example:
	   $HTML::Tagset::emptyElement{'li'} exists and is true.

       hash %HTML::Tagset::linkElements
	   Values in this hash are tagnames for elements that might contain links, and the value for each is a reference to an array of the names
	   of attributes whose values can be links.

       hash %HTML::Tagset::boolean_attr
	   This hash (not hashset) lists what attributes of what elements can be printed without showing the value (for example, the "noshade"
	   attribute of "hr" elements).  For elements with only one such attribute, its value is simply that attribute name.  For elements with
	   many such attributes, the value is a reference to a hashset containing all such attributes.

       hashset %HTML::Tagset::isPhraseMarkup
	   This hashset contains all phrasal-level elements.

       hashset %HTML::Tagset::is_Possible_Strict_P_Content
	   This hashset contains all phrasal-level elements that be content of a P element, for a strict model of HTML.

       hashset %HTML::Tagset::isHeadElement
	   This hashset contains all elements that elements that should be present only in the 'head' element of an HTML document.

       hashset %HTML::Tagset::isList
	   This hashset contains all elements that can contain "li" elements.

       hashset %HTML::Tagset::isTableElement
	   This hashset contains all elements that are to be found only in/under a "table" element.

       hashset %HTML::Tagset::isFormElement
	   This hashset contains all elements that are to be found only in/under a "form" element.

       hashset %HTML::Tagset::isBodyMarkup
	   This hashset contains all elements that are to be found only in/under the "body" element of an HTML document.

       hashset %HTML::Tagset::isHeadOrBodyElement
	   This hashset includes all elements that I notice can fall either in the head or in the body.

       hashset %HTML::Tagset::isKnown
	   This hashset lists all known HTML elements.

       hashset %HTML::Tagset::canTighten
	   This hashset lists elements that might have ignorable whitespace as children or siblings.

       array @HTML::Tagset::p_closure_barriers
	   This array has a meaning that I have only seen a need for in "HTML::TreeBuilder", but I include it here on the off chance that someone
	   might find it of use:

	   When we see a "<p>" token, we go lookup up the lineage for a p element we might have to minimize.  At first sight, we might say that if
	   there's a p anywhere in the lineage of this new p, it should be closed.  But that's wrong.  Consider this document:

	     <html>
	       <head>
		 <title>foo</title>
	       </head>
	       <body>
		 <p>foo
		   <table>
		     <tr>
		       <td>
			  foo
			  <p>bar
		       </td>
		     </tr>
		   </table>
		 </p>
	       </body>
	     </html>

	   The second p is quite legally inside a much higher p.

	   My formalization of the reason why this is legal, but this:

	     <p>foo<p>bar</p></p>

	   isn't, is that something about the table constitutes a "barrier" to the application of the rule about what p must minimize.

	   So @HTML::Tagset::p_closure_barriers is the list of all such barrier-tags.

       hashset %isCDATA_Parent
	   This hashset includes all elements whose content is CDATA.

CAVEATS

       You may find it useful to alter the behavior of modules (like "HTML::Element" or "HTML::TreeBuilder") that use "HTML::Tagset"'s data tables
       by altering the data tables themselves.	You are welcome to try, but be careful; and be aware that different modules may or may react dif-
       ferently to the data tables being changed.

       Note that it may be inappropriate to use these tables for producing HTML -- for example, %isHeadOrBodyElement lists the tagnames for all
       elements that can appear either in the head or in the body, such as "script".  That doesn't mean that I am saying your code that produces
       HTML should feel free to put script elements in either place!  If you are producing programs that spit out HTML, you should be intimately
       familiar with the DTDs for HTML or XHTML (available at "http://www.w3.org/"), and you should slavishly obey them, not the data tables in
       this document.

SEE ALSO

       HTML::Element, HTML::TreeBuilder, HTML::LinkExtor

COPYRIGHT

       Copyright 1995-2000 Gisle Aas; copyright 2000 Sean M. Burke.

       This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

AUTHOR

       Current maintainer: Sean M. Burke, <sburke@cpan.org>

       Most of the code/data in this module was adapted from code written by Gisle Aas <gisle@aas.no> for "HTML::Element", "HTML::TreeBuilder",
       and "HTML::LinkExtor".

perl v5.8.0							    2000-10-20								 Tagset(3)
All times are GMT -4. The time now is 07:02 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy