Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Parse multiple html files in directory

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Parse multiple html files in directory
# 1  
Old 12-16-2014
Parse multiple html files in directory
I have downloaded source code for 97 files using:

Code:
 wget -x -i link.txt

then run a rename loop:

Code:
 for file in *
do
   mv $file $file.txt
done

to keep the html tags but make the file a text that can be parsed.

In each of the 97 txt files the gene # is variable, but the gene is associated or should have a corresponding OMIM #. They are all in,
Code:
 C:\Users\cmccabe\Desktop\list\geneticslab.emory.edu.txt\tests_txt

Is there a way to search the source code for these gene names and OMIM #’s?

For example, in the attached file there are 26 genes:

Output (tab-delimited)
Code:
A            B
Gene     OMIM
AKT1	164730
ALK	105590
APC	611731

The gene names seem to be after
Code:
 target = '_blank'>AKT1</a>

and the OMIM # seem to be
Code:
 style = 'margin-bottom:10px;'><a href =

I think
Code:
 sed

can parse html but I am not familiar enough to know how to code it for multiple files in a directory.

Thank you to all for the help Smilie.
CM080.txt (17.2 KB) 
Last edited by Scrutinizer; 12-16-2014 at 03:16 PM.. Reason: Extra code tags
# 2  
Old 12-16-2014
You don't need to rename the files to .txt to parse them with sed. Try:
Code:
awk     '/^<h2 id="genes"/      {getline
                                 for (i=1; i<=NF; i++)
                                        {n1=gsub ("[, ]*<a href = .http://www.omim.org/entry/", "", $i)
                                         n2=gsub (". target = ._blank.", "", $i)
                                         n3=gsub ("</a", "", $(i+1))
                                         if (n1 + n2) print $(i+1) "\t" $i
                                        }
                                 exit
                                }
        ' FS=">" /tmp/CM080
AKT1    164730
ALK     105590
APC     611731
BRAF    164757
CDH1    192090
CTNNB1  116806
EGFR    131550
ERBB2   164870
FBXW7   606278
FGFR2   176943
FOXL2   605597
GNAQ    600998
GNAS    139320
KIT     164920
KRAS    190070
MAP2K1  176872
MET     164860
MSH6    600678
NRAS    164790
PDGFRA  173490
PIK3CA  171834
PTEN    601728
SMAD4   600993
SRC     190090
STK11   602216
TP53    191170

# 3  
Old 12-16-2014
Can a loop be used to parse each of the 97 files in
Code:
 C:\Users\cmccabe\Desktop\list\geneticslab.emory.edu.txt\tests

keeping the original file and the newly created parsed.txt? Thank you Smilie.

So, CM080 (original) - CM080parsed.txt

Also, if I run the command to write an output file it takes a very long time
Code:
 awk     '/^<h2 id="genes"/      {getline
                                 for (i=1; i<=NF; i++)
                                        {n1=gsub ("[, ]*<a href = .http://www.omim.org/entry/", "", $i)
                                         n2=gsub (". target = ._blank.", "", $i)
                                         n3=gsub ("</a", "", $(i+1))
                                         if (n1 + n2) print $(i+1) "\t" $i
                                        }
                                 exit
                                }
        ' FS=">" CM080 > output.txt

, but if there is no output it runs quickly.
Code:
 awk     '/^<h2 id="genes"/      {getline
                                 for (i=1; i<=NF; i++)
                                        {n1=gsub ("[, ]*<a href = .http://www.omim.org/entry/", "", $i)
                                         n2=gsub (". target = ._blank.", "", $i)
                                         n3=gsub ("</a", "", $(i+1))
                                         if (n1 + n2) print $(i+1) "\t" $i
                                        }
                                 exit
                                }
        ' FS=">" CM080

Thank you Smilie.
# 4  
Old 12-17-2014
There's no reason it should run slower when printing to a file, and it doesn't if I try. For your multiple files, try
Code:
awk     '/^<h2 id="genes"/      {getline
                                 for (i=1; i<=NF; i++)
                                        {n1=gsub (".*omim.org/entry/", "", $i)
                                         n2=gsub (". target = ._blank.>", "\t", $i)
                                         n3=gsub ("</a>.*", "", $i)
                                         print $i  >  FILENAME".parsed"
                                        }
                                }
        ' FS="," /tmp/CM*

after having adapted the input files' path. It requires that ALL input files have the same structure as CM080 does.
# 5  
Old 12-17-2014
Code:
 awk     '/^<h2 id="genes"/      {getline
>                                  for (i=1; i<=NF; i++)
>                                         {n1=gsub (".*omim.org/entry/", "", $i)
>                                          n2=gsub (". target = ._blank.>", "\t", $i)
>                                          n3=gsub ("</a>.*", "", $i)
>                                          print $i  >  FILENAME".parsed"
>                                         }
>                                 }
>         ' FS="," C:\Users\cmccabe\Desktop\list\geneticslab.emory.edu.txt\parse
awk: fatal: cannot open file `C:UserscmccabeDesktoplistgeneticslab.emory.edu.txtparse' for reading (No such file or directory)

I am using cygwin on a windows machine. Should I try Ubuntu? Thank you Smilie.
# 6  
Old 12-17-2014
Looks like it suppressed all "\" in the input file path. Try to escape or quote.
You did not specify any wildcard, so it would work on file "parse" only. Is this what you want? Or is "parse" a directory?
# 7  
Old 12-17-2014
I put all 97 of the files to be parsed in a directory:

Code:
 C:\Users\cmccabe\Desktop\list\geneticslab.emory.edu.txt\parse

So maybe that should be
Code:
 "C:\Users\cmccabe\Desktop\list\geneticslab.emory.edu.txt\parse
"*

Not sure how to escape.


Thank you Smilie.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Merge Multiple html files into one

Hi all I have written some code to write my output in html. As i have multiple servers, need to generate single html file. but my code is generating html file for each server. I have merged the files using below code. cat /home/*_FinalData.html > /home/MergedFinalData.html But how to... (1 Reply)
Discussion started by: Snehasish
1 Replies

2. Shell Programming and Scripting

awk Parse And Create Multiple Files Based on Field Value

Hello: I am working parsing a large input file which will be broken down into multiples based on the second field in the file, in this case: STORE. The idea is to create each file with the corresponding store number, for example: Report_$STORENUM_$DATETIMESTAMP , and obtaining the... (7 Replies)
Discussion started by: ec012
7 Replies

3. Shell Programming and Scripting

Parse html

I downloaded source code using: wget -qO- http://fulgentdiagnostics.com/test/clinical-exome/ | cat > flugentsource.txt Now I am trying to use sed to parse it to confirm a gene count. Basically, output (flugent.txt) all the gene names with a total count after them I'm not all that... (5 Replies)
Discussion started by: cmccabe
5 Replies

4. UNIX for Advanced & Expert Users

Mutt for html body and multiple html & pdf attachments

Hi all: Been racking my brain on this for the last couple of days and what has been most frustrating is that this is the last piece I need to complete a project. There are numerous posts discussing mutt in this forum and others but I have been unable to find similar issues. Running with... (1 Reply)
Discussion started by: raggmopp
1 Replies

5. Shell Programming and Scripting

Read multiple files, parse data and append to a file

Hi..Can anyone suggest a simple way of achieving this. I have several files which ends with extension .vcf . I will give example with two files In the below files, we are interested in File 1: 38 107 C 3 T 6 C/T 38 241 C 4 T 5 C/T 38 247 T 4 C 5 T/C 38 259 T 3 C 6 T/C... (8 Replies)
Discussion started by: empyrean
8 Replies

6. Shell Programming and Scripting

Parse files in directory and compare with another file

I have two files File 1 in reading directory is of following format Read 1 A T Read 3 T C Read 5 G T Read 7 A G Read 10 A G Read 12 C G File 2 in directory contains Read 5 A G Read 6 T C Read 7 G A Read 8 G A Read 20 A T File2 contains (1 Reply)
Discussion started by: empyrean
1 Replies

7. Shell Programming and Scripting

Need script to remove millions of tmp files in /html/cache/ directory

Hello, I just saw that on my vps (centOS) my oscommerce with a seo script has created millions of tmp files inside the /html/cache/ directory. I would need to remove all those files (millions), I tried via shell but the vps loads goes to very high and it hangs, is there some way to do a... (7 Replies)
Discussion started by: andymc1
7 Replies

8. Shell Programming and Scripting

sed to parse html

Hello, I have a html file like this : <html> ... ... ... <table> ....... ...... </table> <table name = "hi"> ...... ..... ... </table> <h1> Welcome </h1> ....... ...... </html> (11 Replies)
Discussion started by: prasanna1157
11 Replies

9. Shell Programming and Scripting

to parse a directory and its subdirectories and find owner name of files

hi all, i need to capture all the files in a directory and its subdirectories that have owner name different than the root owner. for one file it is " stat -c %U filename " but i need to search for each and every file and record it. thanks in advance (14 Replies)
Discussion started by: vyasa
14 Replies

10. Shell Programming and Scripting

Multiple edits to a bunch of html files

I'm trying to upgrade a whole bunch of pages on my site to a new design. I thought one way of doing it would be to enclose the content in special comment tags and then use some form of script to wrap the new html around it. Like this: <!-- content start --> <h1>Blah blah blah</h1> yada yada... (9 Replies)
Discussion started by: dheian
9 Replies
Login or Register to Ask a Question