Extract the tables from html


Login or Register to Reply

 
Thread Tools Search this Thread
# 8  
Old 5 Days Ago
Quote:
Originally Posted by deepti01
i am just learning to do it . ...
Doesn't really answer Neo's question. Please explicitly tell us HOW and WHERE you learn to do it. What is your motivation to do so?



I see you incorporated but commented out my recent proposal to extract the second table from the HTML file. What keeps you from adapting it to your needs?

And, why don't you go for doing all of your task in one single script, e.g. awk?

RavinderSingh13 asked you to post "samples of input and expected output". If done, you could get hints on how to do it optimally.
# 9  
Old 5 Days Ago
Hi,

The end use if this task is to dump the data into the database through Informatica.
So for that I am converting the HTML files to .csv file.

Sample input : i have already placed the html code . that is my sample file (.html)

Sample output should be like:

Code:
No    TABLE NAME    Exported Rows    Imported Rows    Diff Rows
1    NAME_DATA    24760        Not exist on Imp
2    WHITE_LIST_MIG    12912        Not exist on Imp
1    ADDRESS_DATA    13753    13753    0
2    ADDRESS_NAME_LINK    68715    68715    0
3    AGREEMENT    0    0    0
4    AGREEMENT_RESOURCE    29979    29979    0
5    AGR_RES_HISTORY    0    0    0
 6    AR1_ACCOUNT    12912    12912    0





Moderator's Comments:
Mod Comment Please use CODE tags as required by forum rules!

Last edited by RudiC; 5 Days Ago at 05:10 AM.. Reason: Added CODE tags.
# 10  
Old 5 Days Ago
Please note that awk is NOT an adequate tool (nor is shell) to analyse / extract html data; there are way more apt tools to do this.
Please note as well that none of your files is valid html.
And, "csv" means "comma separated values", so I used commas for the extracted datas' separator.

Howsoever, for this - and exactly this - simple case, try


Code:
awk '
/<TABLE/        {P = 1
                }
/<\/TABLE/      {P = 0
                }
!P              {next
                }
/<[Tt][Rr]>/    {printf ORS
                 DL = ""
                }
/<td/           {gsub (/<[^>]*>/, "")
                 gsub (/^ *| *$/, "")
                 printf "%s%s", DL, $0
                 DL = ","
                }
' file

No,ExpSchemae,ExportTables,ImpSchema,ImportTables,Diff
1,FVT4,54,PRDCUSTO,54,
1,FVT4,56,AllImpSchema,54,2
No,TABLENAME,ExportedRows,ImportedRows,DiffRows
1,NAME_DATA,24760,,NotexistonImp
2,WHITE_LIST_MIG,12912,,NotexistonImp
No,TABLENAME,ExportedRows,ImportedRows,DiffRows
1,ADDRESS_DATA,13753,13753,0
2,ADDRESS_NAME_LINK,68715,68715,0
3,AGREEMENT,0,0,0
4,AGREEMENT_RESOURCE,29979,29979,0
5,AGR_RES_HISTORY,0,0,0
6,AR1_ACCOUNT,12912,12912,0
.
.
.


Last edited by RudiC; 5 Days Ago at 06:12 AM..
This User Gave Thanks to RudiC For This Post:
deepti01 (5 Days Ago)
# 11  
Old 4 Days Ago
Hi.

There are some codes that understand HTML, such as lynx, elinks. For example, here is elinks extracting all the stuff in the sample HTML, and then, step-by-step, tables are refined, with sed, grep, etc., into a CSV file:
Code:
#!/usr/bin/env bash

# @(#) s1       Demonstrate table extraction from HTML.

# Utility functions: print-as-echo, print-line-with-visual-space, debug.
# export PATH="/usr/local/bin:/usr/bin:/bin"
LC_ALL=C ; LANG=C ; export LC_ALL LANG
pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }
em() { pe "$*" >&2 ; }
db() { ( printf " db, ";for _i;do printf "%s" "$_i";done;printf "\n" ) >&2 ; }
db() { : ; }
C=$HOME/bin/context && [ -f $C ] && $C specimen elinks grep dixf

FILE=${1-data1}

# Utility functions: print-as-echo, print-line-with-visual-space.
pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }

pl " Input data file $FILE:"
specimen 5:5:5 $FILE

pl " Results:"
elinks -dump $FILE |
tee t1 |
grep -v '^  *[|+-]-' |
tee t2 |
grep '^  *[|]' |
tee t3 |
sed -e 's/^  *[|]//' -e 's/[|]$//' |
tee t4 |
sed 's/  *[|]/,/g' |
tee t5 |
specimen 5:5:5

pl " Samples of intermediate output:"
head -5 t[1-5]

pl " Some details for codes:"
dixf elinks specimen grep

exit 0

producing:
Code:
$ ./s1

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 3.16.0-7-amd64, x86_64
Distribution        : Debian 8.11 (jessie) 
bash GNU bash 4.3.30
specimen (local) 1.17
ELinks 0.12pre6
grep (GNU grep) 2.20
dixf (local) 1.57

-----
 Input data file data1:
Edges: 5:5:5 of 295 lines in file "data1"
<html>
<body>
<b><br>Running Date: </b>11-JAN-2019 03:07</br>
<h2> Schema mapping and info    </h2>
<BR><TABLE  width="100%" class="x1h" cellpadding="1" cellspacing="0" border="5">
   ---
<b><td class="x3w" bgcolor="#BDD7EE">15 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">AR1_JGL_CONTROL  </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD></TR>
   ---
<b><td class="x3w" bgcolor="#BDD7EE">BL9_PROVINCIAL_PCP  </TD>
<b><td class="x3w" bgcolor="#BDD7EE">18259 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">18259 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD></TR>
<tr>

-----
 Results:
Edges: 5:5:5 of 46 lines in file "-"
No,Exp Schema e,Export Tables,Imp Schema,Import Tables,Diff
1,FVT4,54,PRDCUSTO,54,    
1,FVT4,56,All Imp Schema,54,2   
No,TABLE NAME,Exported Rows,Imported Rows,Diff Rows        
1,NAME_DATA,24760,,Not exist on Imp 
   ---
15,AR1_JGL_CONTROL,0,0,0             
16,AR1_PAYMENT,8439,8439,0             
17,AR1_PAYMENT_DETAILS,8439,8439,0             
18,AR1_PAY_CHANNEL,12912,12912,0             
19,AR1_PROOF_AND_BALANCE,4,4,0             
   ---
35,BL1_PAY_CHANNEL,12912,12912,0             
36,BL1_RC_RATES,30928,30928,0             
37,BL1_TAX,0,0,0             
38,BL1_TAX_ITEM,0,0,0             
39,BL9_PROVINCIAL_PCP,18259,18259,0             

-----
 Samples of intermediate output:
==> t1 <==
   Running Date: 11-JAN-2019 03:07

Schema mapping and info

   +------------------------------------------------------------------------+

==> t2 <==
   Running Date: 11-JAN-2019 03:07

Schema mapping and info

   |No |Exp Schema e   |Export Tables  |Imp Schema     |Import Tables  |Diff|

==> t3 <==
   |No |Exp Schema e   |Export Tables  |Imp Schema     |Import Tables  |Diff|
   |1  |FVT4           |54             |PRDCUSTO       |54             |    |
   |1  |FVT4           |56             |All Imp Schema |54             |2   |
   |No  |TABLE NAME      |Exported Rows   |Imported Rows  |Diff Rows        |
   |1   |NAME_DATA       |24760           |               |Not exist on Imp |

==> t4 <==
No |Exp Schema e   |Export Tables  |Imp Schema     |Import Tables  |Diff
1  |FVT4           |54             |PRDCUSTO       |54             |    
1  |FVT4           |56             |All Imp Schema |54             |2   
No  |TABLE NAME      |Exported Rows   |Imported Rows  |Diff Rows        
1   |NAME_DATA       |24760           |               |Not exist on Imp 

==> t5 <==
No,Exp Schema e,Export Tables,Imp Schema,Import Tables,Diff
1,FVT4,54,PRDCUSTO,54,    
1,FVT4,56,All Imp Schema,54,2   
No,TABLE NAME,Exported Rows,Imported Rows,Diff Rows        
1,NAME_DATA,24760,,Not exist on Imp 

-----
 Some details for codes:

elinks  lynx-like alternative character mode WWW browser (man)
Path    : /usr/bin/elinks
Version : 0.12pre6
Type    : ELF 64-bit LSB shared object, x86-64, version 1 ( ...)
Repo    : Debian 8.11 (jessie) 
Home    : http://elinks.cz/ (pm)

specimen        Extract, display sections, sample, provide taste of file. (what)
Path    : ~/bin/specimen
Version : 1.17
Length  : 334 lines
Type    : Perl script, ASCII text executable
Shebang : #!/usr/bin/env perl
Modules : (for perl codes)
 warnings       1.23
 strict 1.08
 English        1.09
 Carp   1.3301
 Data::Dumper   2.151_01
 Getopt::Long   2.42
 feature        1.36_01
 Regexp::Common 2013031301
 Algorithms::Numerical::Sample  1.1

grep    print lines matching a pattern (man)
Path    : /bin/grep
Version : 2.20
Type    : ELF 64-bit LSB shared object, x86-64, version 1 ( ...)
Repo    : Debian 8.11 (jessie) 
Home    : http://www.gnu.org/software/grep/ (pm)

The intermediate sample show the transformations that are done, and could, of course, probably be collected into an awk or perl script.

Best wishes ... cheers, drl
This User Gave Thanks to drl For This Post:
deepti01 (4 Days Ago)
Login or Register to Reply

|
Thread Tools Search this Thread
Search this Thread:
Advanced Search

More UNIX and Linux Forum Topics You Might Find Helpful
Unable to send attachment with html tables in UNIX shell script Harsha Vardhan HP-UX 2 02-28-2018 02:34 PM
Splitting csv into 3 tables in html file archana25 Shell Programming and Scripting 7 07-30-2017 11:02 PM
Awk/sed HTML extract p1ne Shell Programming and Scripting 8 08-01-2016 10:33 AM
Extract table from an HTML file koutroul UNIX for Dummies Questions & Answers 4 04-11-2014 01:35 PM
Extract/Parse information from html (website) TehOne Shell Programming and Scripting 5 05-02-2012 03:12 AM
extract fields from a downloaded html file gubbu Shell Programming and Scripting 1 04-16-2012 01:55 AM
awk to create two HTML Tables dynamax Shell Programming and Scripting 2 07-08-2011 10:40 PM
extract data with awk from html files sbobotex Shell Programming and Scripting 6 12-20-2010 10:39 AM
How to extract url from html page? 14th Shell Programming and Scripting 36 10-18-2010 02:12 AM
Extract data from DB2 tables and FTP it to outside company's firewall priyanka3006 AIX 1 07-19-2010 10:27 AM
SED to extract HTML text data, not quite right! lagagnon Shell Programming and Scripting 2 01-31-2010 12:14 AM
Extract URLs from HTML code using sed L0rd Shell Programming and Scripting 13 11-30-2009 11:35 PM
sed to extract HTML content stargazerr UNIX for Advanced & Expert Users 2 03-21-2009 03:31 PM
extract data from html tables Streetrcr UNIX for Dummies Questions & Answers 8 03-20-2008 06:14 AM
How do I extract text only from html file without HTML tag los111 UNIX for Dummies Questions & Answers 4 11-28-2007 03:40 AM