Visit Our UNIX and Linux User Community


Script to test bookmarks


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Script to test bookmarks
# 1  
Old 12-20-2014
Script to test bookmarks

Does anyone have a script to test for invalid bookmarks? I have a ton of them and I'd bet that a good percentage could be dropped.

Tnx
# 2  
Old 12-20-2014
Guessing you are talking of web browser bookmarks, what format/structure are they? And, how would you define "invalid"?
# 3  
Old 12-20-2014
@ RudiC

Hello. Invalid would mean 'broken links'; sites that no longer exist or are unreachable, etc.

I can extract the http links from html files (firefox) with sed easily enough, producing a list to loop through. I'm just not sure whether to use wget or ??? to attempt to reach the sites and get the error msgs. I want to do it in the console, not the browser. I thought I'd ask around for a script or advice from someone who might have already done this.

Thanks for the reply.
# 4  
Old 12-21-2014
Hmm. The biggest problem I see is how many sites redirect bad links to valid pages. Try https://www.unix.com/qwerty/ui/op for an example. It's one thing to catch a 404 -- another to check that the contents were what you wanted.

You can weed out pages that redirect, at least.

Code:
$ wget --max-redirect=0 --spider burningsmell.org/asdf burningsmell.org/index.html burningsmell.org/index.php -nv -O /dev/null
0 redirections exceeded.
0 redirections exceeded.
2014-12-20 23:57:32 URL: http://burningsmell.org/index.php 200 OK

$

Use -i filename to read a list of URL's to check from file.
# 5  
Old 12-22-2014
Thanks for that Corona688

wget does the deed nicely. I found that there are some false positives. Some websites just dont appreciate robots, so they return 'broken link' when the site is actually alive and reachable. But for my intentions that's no big deal.

What I have so far...
Code:
#!/bin/bash

SORTED_LIST=./urls.lst
BOOKMARKS=./bookmarks.html
toDEL=./del.lst
> $toDEL

while read XX
do
    log=$(wget --tries=1 -T 5 --spider -nv $XX 2>&1)
    if [[ $(echo $log | grep -c "broken link") -gt 0 ]]; then
        echo "$log"
        echo "$XX" >> $toDEL
        # sed -n "s|$XX|&|p" $BOOKMARKS
        sed -i "s|^.*$XX.*$||g" $BOOKMARKS
    fi
done < $SORTED_LIST

Found 370 broken links out of 4,000+ bookmarks.

Previous Thread | Next Thread
Test Your Knowledge in Computers #705
Difficulty: Medium
Confirmation bias is the tendency to search for, interpret, favor, and recall information in a way that affirms what we currently believe or think we already know.
True or False?

6 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Script to parse bookmarks file

I am using Internet Explorer v10 at work and regularly need to import my personal Firefox bookmarks over. Long story short, I have found the import falling over on any bookmark elements which are over 256 characters. The bookmark file contains bookmarks of this format: <DT><A... (4 Replies)
Discussion started by: ozgadgetguy
4 Replies

2. UNIX and Linux Applications

Creating pdf with bookmarks using ps2pdf ??

Hello to all, I have a *.ps file, which needs to be converted to a pdf file. The file is huge & i would like it to have bookmarked according to main titles (already provided). What is the option used in ps2pdf command, that will generate bookmarks from a *.ps file ? Kindly help. Thanks. ... (0 Replies)
Discussion started by: frozensmilz
0 Replies

3. Shell Programming and Scripting

How to check weather a string is like test* or test* ot *test* in if condition

How to check weather a string is like test* or test* ot *test* in if condition (5 Replies)
Discussion started by: johnjerome
5 Replies

4. Shell Programming and Scripting

Test on string containing spacewhile test 1 -eq 1 do read a $a if test $a = quitC then break fi d

This is the code: while test 1 -eq 1 do read a $a if test $a = stop then break fi done I read a command on every loop an execute it. I check if the string equals the word stop to end the loop,but it say that I gave too many arguments to test. For example echo hello. Now the... (1 Reply)
Discussion started by: Max89
1 Replies

5. UNIX for Dummies Questions & Answers

Moving bookmarks from XP to Ubuntu

This is probably one of the dumber questions I've posted on ANY Linux forum, but I'm in a fog and can't answer it myself. My attempts at searching the forums go no where. I have Win XP and Ubuntu dual booting (via WUBI install). I want to use the Ubuntu side more but I've socked away a LOT of... (1 Reply)
Discussion started by: BillinDetroit
1 Replies

6. UNIX for Dummies Questions & Answers

Push out Firefox bookmarks to multiple users

Ok, so I want to push out Firefox bookmarks to all the student accounts on my Mac server, but we all know how fun-loving Firefox profiles are, with their random czg6hi72.default profile names. My question is how would I send the bookmarks.html file out to /Students/<username>/Library/Application... (5 Replies)
Discussion started by: yodomino6
5 Replies

Featured Tech Videos