Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

URL extraction from JSON file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting URL extraction from JSON file
# 1  
Old 03-06-2013
URL extraction from JSON file
I'm trying to export URLs from within a JSON file which in turn resulted from export of Mozilla-Firefox bookmarks. Its single line file with below given values from awk

Code:
$ awk 'END { print NR }' bookmarks.json
1
$ awk 'END { print NF }' bookmarks.json
2706
$ awk -F, 'END { print NF }' bookmarks.json
4754

using sed, it gives me only 1st occurrence and rest is missed.

Code:
$ sed  's/.*"\(http:.*\)"/\1/' bookmarks.json
http://www.oracle.com/us/products/servers-storage/servers/blades/index.html","charset":"UTF-8}]}]}
$

extract from json file is something like below:

Code:
{"title":"","id":1,"dateAdded":1331548812311000,"lastModified":1331549028262000,"type":"text/x-moz-place-container","root":"placesRoot","children":[{"title":
"Bookmarks Menu","id":2,"parent":1,"dateAdded":1331548812311000,"lastModified":1342096853234000,"type":"text/x-moz-place-container","root":"bookmarksMenuFold
er","children":[{"title":"Recent Tags","id":925,"parent":2,"annos":[{"name":"Places/SmartBookmark","flags":0,"expires":4,"mimeType":null,"type":3,"value":"Re
centTags"}],"type":"text/x-moz-place","uri":"place:sort=14&type=6&maxResults=10&queryType=1"},{"index":1,"title":"Recently Bookmarked","id":924,"parent":2,"a
nnos":[{"name":"Places/SmartBookmark","flags":0,"expires":4,"mimeType":null,"type":3,"value":"RecentlyBookmarked"}],"type":"text/x-moz-place","uri":"place:fo
lder=BOOKMARKS_MENU&folder=UNFILED_BOOKMARKS&folder=TOOLBAR&sort=12&excludeQueries=1&maxResults=10&queryType=1"},{"index":2,"title":"","id":26,"parent":2,"da
teAdded":1243009025055489,"lastModified":1331549044829000,"annos":[{"name":"placesInternal/GUID","flags":0,"expires":4,"mimeType":null,"type":3,"value":"{445
36f3f-1d99-4e6d-8b77-d5e89c334d2d}2"}],"type":"text/x-moz-place-separator"},{"index":3,"title":"Get Bookmark Add-ons","id":27,"parent":2,"dateAdded":12430090
25055489,"lastModified":1331549044829000,"annos":[{"name":"placesInternal/GUID","flags":0,"expires":4,"mimeType":null,"type":3,"value":"{44536f3f-1d99-4e6d-8
b77-d5e89c334d2d}3"}],"type":"text/x-moz-place","uri":"https://en-us.add-ons.mozilla.com/en-US/firefox/bookmarks/"},{"index":4,"title":"","id":28,"parent":2,
"dateAdded":1243009025055489,"lastModified":1331549044829000,"annos":[{"name":"placesInternal/GUID","flags":0,"expires":4,"mimeType":null,"type":3,"value":"{
44536f3f-1d99-4e6d-8b77-d5e89c334d2d}4"}],"type":"text/x-moz-place-separator"},{"index":5,"title":"Mozilla Firefox","id":29,"parent":2,"dateAdded":1243009025
055489,"lastModified":1331549044845000,"annos":[{"name":"placesInternal/GUID","flags":0,"expires":4,"mimeType":null,"type":3,"value":"{44536f3f-1d99-4e6d-8b7
7-d5e89c334d2d}5"}],"type":"text/x-moz-place-container","children":[{"title":"Help and Tutorials","id":30,"parent":29,"dateAdded":1243009025055489,"lastModif
ied":1331549044845000,"annos":[{"name":"placesInternal/GUID","flags":0,"expires":4,"mimeType":null,"type":3,"value":"{44536f3f-1d99-4e6d-8b77-d5e89c334d2d}6"
}],"type":"text/x-moz-place","uri":"http://en-us.www.mozilla.com/en-US/firefox/help/"},{"index":1,"title":"Customize Firefox","id":31,"parent":29,"dateAdded"
:1243009025055489,"lastModified":1331549044845000,"annos":[{"name":"placesInternal/GUID","flags":0,"expires":4,"mimeType":null,"type":3,"value":"{44536f3f-1d
99-4e6d-8b77-d5e89c334d2d}7"}],"type":"text/x-moz-place","uri":"http://en-us.www.mozilla.com/en-US/firefox/customize/"},

Regards,
# 2  
Old 03-06-2013
Code:
awk -F'"' '{ for(i=1; i<=NF; i++) { if($i ~ /^http/) print $i } } ' bookmarks.json

This User Gave Thanks to Yoda For This Post:
busyboy
# 3  
Old 03-06-2013
Could this work:
Code:
 grep -o 'http:[^"]*' file

This User Gave Thanks to hanson44 For This Post:
busyboy
# 4  
Old 03-06-2013
both are perfect..Thanks buddies
# 5  
Old 03-06-2013
Quote:
Originally Posted by hanson44
Could this work:
Code:
 grep -o 'http:[^"]*' file

Yes, will work on GNU grep

Also require minor modification to include secure http:
Code:
grep -o 'http[s]*:[^"]*'

# 6  
Old 03-06-2013
Code:
awk -F'"' '{ for(i=1; i<=NF; i++) { if($i ~ /^http/|| $i ~ /https/ ) print $i } } ' bookmarks

For http and https both
# 7  
Old 03-06-2013
Quote:
Originally Posted by busyboy
Code:
awk -F'"' '{ for(i=1; i<=NF; i++) { if($i ~ /^http/|| $i ~ /https/ ) print $i } } ' bookmarks

For http and https both
if($i ~ /^http/) regexp should cover both http & https
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Parse property from json file

Hello All, Greetings.. I have a json file that I need to pase its URLs and other values. The match should start with "notifications" and output URLs and settings values. I tried with python or awk but hardly could get URLs only. Or whole URLs from full json file. Could not match... (2 Replies)
Discussion started by: 7adi
2 Replies

2. UNIX for Beginners Questions & Answers

Convert String to an Array using shell scripting in JSON file.

This is the sample json I have pasted here. I want all the IP address strings to be converted into an array. For example "10.38.32.202" has to be converted to everywhere in the JSON. There are multiple IPs in a JSON I am pasting one sample object from the JSON. But the IPs already in an Array... (11 Replies)
Discussion started by: vinshas1
11 Replies

3. How to Post in the The UNIX and Linux Forums

Read a json file passed as cmd line argument

usage: myscript.sh config.json config.json: { "HOST":"abc", "DB_NM":"xyz", "USR_NM":"asd", "PWD":"xxx", ......... ......... ......... ........ } myscript.sh: (2 Replies)
Discussion started by: RGRT
2 Replies

4. Shell Programming and Scripting

Parsing and Editing a json file with bash script

I am trying to automate editing of a json file using bash script. The file I initially receive is { "appMap": { "URL1": { "name": "a" }, "URL2": { "name": "b" }, "URL3": { "name": "c" }, } WHat I would like to do is replace... (5 Replies)
Discussion started by: Junaid Subhani
5 Replies

5. Shell Programming and Scripting

How to download Images and Json file from server(godaddy) to Local machine (Ubuntu 14.04).?

Hi Guys, Just entering the Linux word, So I need help to write a script on my local machine(Ubuntu 14.04) that continuously check the particular folder(contains images) and a json file on the server and download whenever new images are added to that folder and whenever there is a change in the... (10 Replies)
Discussion started by: g4v1n
10 Replies

6. Shell Programming and Scripting

Reading URL using Mechanize and dump all the contents of the URL to a file

Hello, Am very new to perl , please help me here !! I need help in reading a URL from command line using PERL:: Mechanize and needs all the contents from the URL to get into a file. below is the script which i have written so far , #!/usr/bin/perl use LWP::UserAgent; use... (2 Replies)
Discussion started by: scott_cog
2 Replies

7. Shell Programming and Scripting

How to define a variable in a BASH script by using a JSON file online?

Hello, I would like to modify an existing script of mine that uses a manually defined "MCVERSION" variable and make it define that variable instead based on this JSON file stored online: https://s3.amazonaws.com/Minecraft.Download/versions/versions.json Within that JSON, I 'm looking for... (4 Replies)
Discussion started by: nbsparks
4 Replies

8. UNIX for Dummies Questions & Answers

Awk: print all URL addresses between iframe tags without repeating an already printed URL

Here is what I have so far: find . -name "*php*" -or -name "*htm*" | xargs grep -i iframe | awk -F'"' '/<iframe*/{gsub(/.\*iframe>/,"\"");print $2}' Here is an example content of a PHP or HTM(HTML) file: <iframe src="http://ADDRESS_1/?click=5BBB08\" width=1 height=1... (18 Replies)
Discussion started by: striker4o
18 Replies

9. Shell Programming and Scripting

url calling and parameter passing to url in script

Hi all, I need to write a unix script in which need to call a url. Then need to pass parameters to that url. please help. Regards, gander_ss (1 Reply)
Discussion started by: gander_ss
1 Replies

10. UNIX for Advanced & Expert Users

url calling and parameter passing to url in script

Hi all, I need to write a unix script in which need to call a url. Then need to pass parameters to that url. please help. Regards, gander_ss (1 Reply)
Discussion started by: gander_ss
1 Replies
Login or Register to Ask a Question