Sponsored Content
Top Forums UNIX for Beginners Questions & Answers Multiline html tag parse shell script Post 303040234 by stomp on Friday 25th of October 2019 08:09:45 AM
Old 10-25-2019
Since HTML is very similar to XML, you may use an xml tool to parse your file.

Since your HTML-File is not fully standards compliant, the parser complains about it and the file has either be adapted by hand to be compliant or to be preprocessed prior to the parsing. The <br> is the problematic element. Compliant would be <br/> with a slash within the tag.

So you can do it with an xmlparser like xmlstarlet in three steps:

Code:
sed 's|<br>|<br/>|gi' data.html      |
  xmlstarlet sel -t -v '//body/div/p'   |
  sed -e '/^\s*$/d' -e 's/^\s*//'

1. Make the html file compliant by replacing the br-Tags
2. Get the wanted HTML-Element with xmlstarlet
3. suppress unwanted empty lines and leading whitespace in data / xmlstarlet output

Last edited by stomp; 10-25-2019 at 09:31 AM..
These 2 Users Gave Thanks to stomp For This Post:
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

How do I extract text only from html file without HTML tag

I have a html file called myfile. If I simply put "cat myfile.html" in UNIX, it shows all the html tags like <a href=r/26><img src="http://www>. But I want to extract only text part. Same problem happens in "type" command in MS-DOS. I know you can do it by opening it in Internet Explorer,... (4 Replies)
Discussion started by: los111
4 Replies

2. Shell Programming and Scripting

how to use html tag in shell scripting

Hai friends I have a small doubt.. how can we use html tag in shell scripting code : echo "<html>" echo "<body>" echo " welcome to peace world " echo "</body>" echo "</html>" output displayed like this: <html> <body> welcome to peace world </body> </html> (5 Replies)
Discussion started by: jrex1983
5 Replies

3. UNIX for Advanced & Expert Users

shell script to parse html file

hi all, i have a html file something similar to this. <tr class="evenrow"> <td class="data">added</td><td class="data">xyz@abc.com</td> <td class="data">filename.sql</td><td class="modifications-data">08/25/2009 07:58:40</td><td class="data">Added TK prof script</td> </tr> <tr... (1 Reply)
Discussion started by: sais
1 Replies

4. Shell Programming and Scripting

Parse HTML tag parameters and text

Hi! I have a bunch of HTML files, which I want to parse to CSV files. Every page has a table in it, and I need to parse each row into a csv record. With awk and sed, I managed to put every table row in separate lines. So my file looks like this: <TR> .... </TR> <TR> .... </TR> ...One... (1 Reply)
Discussion started by: senszey
1 Replies

5. Shell Programming and Scripting

Script to delete HTML tag

Guys, I have a little script that I got of the internet and that I use in Squid to block ads. I used that script with linux but now i have moved my servers to freebsd. I have a step learning curve there but it is fun: Back to the script issue. The script used to work i with linux but... (15 Replies)
Discussion started by: zongo
15 Replies

6. Shell Programming and Scripting

awk Script to parse a XML tag

I have an XML tag like this: <property name="agent" value="/var/tmp/root/eclipse" /> Is there way using awk that i can get the value from the above tag. So the output should be: /var/tmp/root/eclipse Help will be appreciated. Regards, Adi (6 Replies)
Discussion started by: asirohi
6 Replies

7. Shell Programming and Scripting

Search for a html tag and print the entire tag

I want to print from <fruits> to </fruits> tag which have <fruit> as mango. Also i want both <fruits> and </fruits> in output. Please help eg. <fruits> <fruit id="111">mango<fruit> . another 20 lines . </fruits> (3 Replies)
Discussion started by: Ashik409
3 Replies

8. Shell Programming and Scripting

Using shell command need to parse multiple nested tag value of a XML file

I have this XML file - <gp> <mms>1110012</mms> <tg>988</tg> <mm>LongTime</mm> <lv> <lkid>StartEle=ONE, Desti = Motion</lkid> <kk>12</kk> </lv> <lv> <lkid>StartEle=ONE, Source = Velocity</lkid> <kk>2</kk> </lv> <lv> ... (3 Replies)
Discussion started by: NeedASolution
3 Replies

9. Shell Programming and Scripting

XML Parse between to tag with upper tag

Hi Guys Here is my Input : <?xml version="1.0" encoding="UTF-8"?> <xn:MeContext id="01736"> <xn:VsDataContainer id="01736"> <xn:attributes> <xn:vsDataType>vsDataMeContext</xn:vsDataType> ... (12 Replies)
Discussion started by: pareshkp
12 Replies

10. Shell Programming and Scripting

How to remove html tag which has multiple lines in SHELL?

I want to clean a html file. I try to remove the script part in the html and remove the rest of tags and empty lines. The code I try to use is the following: sed '/<script/,/<\/script>/d' webpage.html | sed -e 's/<*>//g' | sed '/^\s*$/d' > output.txt However, in this method, I can not... (10 Replies)
Discussion started by: YuhuiFeng
10 Replies
MKDoc::XML::TreeBuilder(3pm)				User Contributed Perl Documentation			      MKDoc::XML::TreeBuilder(3pm)

NAME
MKDoc::XML::TreeBuilder - Builds a parsed tree from XML data SYNOPSIS
my @top_nodes = MKDoc::XML::TreeBuilder->process_data ($some_xml); SUMMARY
MKDoc::XML::TreeBuilder uses MKDoc::XML::Tokenizer to turn XML data into a parsed tree. Basically it smells like an XML parser, looks like an XML parser, and awfully overlaps with XML parsers. But it's not an XML parser. XML parsers are required to die if the XML data is not well formed. MKDoc::XML::TreeBuilder doesn't give a rip: it'll parse whatever as long as it's good enough for it to parse. XML parsers expand entities. MKDoc::XML::TreeBuilder doesn't. At least not yet. XML parsers generally support namespaces. MKDoc::XML::TreeBuilder doesn't - and probably won't. DISCLAIMER
This module does low level XML manipulation. It will somehow parse even broken XML and try to do something with it. Do not use it unless you know what you're doing. API
my @top_nodes = MKDoc::XML::Tokenizer->process_data ($some_xml); Returns all the top nodes of the $some_xml parsed tree. Although the XML spec says that there can be only one top element in an XML file, you have to take two things into account: 1. Pseudo-elements such as XML declarations, processing instructions, and comments. 2. MKDoc::XML::TreeBuilder is not an XML parser, it's not its job to care about the XML specification, so having multiple top elements is just fine. my $tokens = MKDoc::XML::Tokenizer->process_data ('/some/file.xml'); Same as MKDoc::XML::TreeBuilder->process_data ($some_xml), except that it reads $some_xml from '/some/file.xml'. Returned parsed tree - data structure I have tried to make MKDoc::XML::TreeBuilder look enormously like HTML::TreeBuilder. So most of this section is stolen and slightly adapted from the HTML::Element man page. START PLAGIARISM HERE It may occur to you to wonder what exactly a "tree" is, and how it's represented in memory. Consider this HTML document: <html lang='en-US'> <head> <title>Stuff</title> <meta name='author' content='Jojo' /> </head> <body> <h1>I like potatoes!</h1> </body> </html> Building a syntax tree out of it makes a tree-structure in memory that could be diagrammed as: html (lang='en-US') / / / head body / / / title meta h1 | (name='author', | "Stuff" content='Jojo') "I like potatoes" This is the traditional way to diagram a tree, with the "root" at the top, and it's this kind of diagram that people have in mind when they say, for example, that "the meta element is under the head element instead of under the body element". (The same is also said with "inside" instead of "under" -- the use of "inside" makes more sense when you're looking at the HTML source.) Another way to represent the above tree is with indenting: html (attributes: lang='en-US') head title "Stuff" meta (attributes: name='author' content='Jojo') body h1 "I like potatoes" Incidentally, diagramming with indenting works much better for very large trees, and is easier for a program to generate. The $tree->dump method uses indentation just that way. However you diagram the tree, it's stored the same in memory -- it's a network of objects, each of which has attributes like so: element #1: _tag: 'html' _parent: none _content: [element #2, element #5] lang: 'en-US' element #2: _tag: 'head' _parent: element #1 _content: [element #3, element #4] element #3: _tag: 'title' _parent: element #2 _content: [text segment "Stuff"] element #4 _tag: 'meta' _parent: element #2 _content: none name: author content: Jojo element #5 _tag: 'body' _parent: element #1 _content: [element #6] element #6 _tag: 'h1' _parent: element #5 _content: [text segment "I like potatoes"] The "treeness" of the tree-structure that these elements comprise is not an aspect of any particular object, but is emergent from the relatedness attributes (_parent and _content) of these element-objects and from how you use them to get from element to element. STOP PLAGIARISM HERE This is pretty much the kind of data structure MKDoc::XML::TreeBuilder returns. More information on different nodes and their type is available in MKDoc::XML::Token. NOTES
Did I mention that MKDoc::XML::TreeBuilder is NOT an XML parser? AUTHOR
Copyright 2003 - MKDoc Holdings Ltd. Author: Jean-Michel Hiver This module is free software and is distributed under the same license as Perl itself. Use it at your own risk. SEE ALSO
MKDoc::XML::Token MKDoc::XML::Tokenizer perl v5.10.1 2004-10-06 MKDoc::XML::TreeBuilder(3pm)
All times are GMT -4. The time now is 07:01 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy