Help with shell script "wget"


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Help with shell script "wget"
# 1  
Old 04-04-2009
Help with shell script "wget"

I am trying to gather a list of files (products) every 30 days that have been added to a website. First it will delete the files that are already in the directory from 30 days ago. Then it will it gather the updated files (products) from the webstie and it will place them in a directory. Lastly it will tar the files that are in the directory. If anyone knows how to get this done your assistance would be appreciated.
# 2  
Old 04-04-2009
Quote:
Originally Posted by jhampt
I am trying to gather a list of files (products) every 30 days that have been added to a website. First it will delete the files that are already in the directory from 30 days ago. Then it will it gather the updated files (products) from the webstie and it will place them in a directory. Lastly it will tar the files that are in the directory. If anyone knows how to get this done your assistance would be appreciated.
The requirement is not clear..However, u can try:

Quote:
- What is the content of webpage? Paste a sample source of website
- For deleting files, you can go for
find dir_name "*" -type f -mtime+30 -exec rm {} \;
- From where the source files will be available?
- you can do a tar by
ls -1 | xargs tar cvf file.tar -
# 3  
Old 04-06-2009
The content is on our intranet and I am unable to disclose the specifics. It is bascially a website were analyst post their findings. I need to pull these posting every 30 days into a directory and tar them. Here is what i have come up with so far:

#! /bin/sh
#script.sh
[ -z $1 ] && " Defaulting to $ (date +%F)" && DATES=$ (date +%F) || DATE="1"
touch -d "1" /tmp/OLD
touch -r /tmp/OLD -F 2592000 /tmp/NEW
# /tmp/NEW 30 days newer than /tmp/OLD
find . -name * -newer /tmp/OLD ! -newer /tmp/NEW ! -size 0
rm -f /tmp /OLD /tmp/NEW/NEW

I have to figure out how to incorporate either a get or wget to pull the updates.
I hope this makes it a little clearer. Thanks for the initial response.
# 4  
Old 04-06-2009
Another solution is to use:
grep "/products/2009" after the get command but how would I only grep for a specific timeframe.... ie 30 days
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Bash script - Print an ascii file using specific font "Latin Modern Mono 12" "regular" "9"

Hello. System : opensuse leap 42.3 I have a bash script that build a text file. I would like the last command doing : print_cmd -o page-left=43 -o page-right=22 -o page-top=28 -o page-bottom=43 -o font=LatinModernMono12:regular:9 some_file.txt where : print_cmd ::= some printing... (1 Reply)
Discussion started by: jcdole
1 Replies

2. UNIX for Dummies Questions & Answers

Using "mailx" command to read "to" and "cc" email addreses from input file

How to use "mailx" command to do e-mail reading the input file containing email address, where column 1 has name and column 2 containing “To” e-mail address and column 3 contains “cc” e-mail address to include with same email. Sample input file, email.txt Below is an sample code where... (2 Replies)
Discussion started by: asjaiswal
2 Replies

3. Shell Programming and Scripting

awk command to replace ";" with "|" and ""|" at diferent places in line of file

Hi, I have line in input file as below: 3G_CENTRAL;INDONESIA_(M)_TELKOMSEL;SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL My expected output for line in the file must be : "1-Radon1-cMOC_deg"|"LDIndex"|"3G_CENTRAL|INDONESIA_(M)_TELKOMSEL"|LAST|"SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL" Can someone... (7 Replies)
Discussion started by: shis100
7 Replies

4. UNIX for Advanced & Expert Users

Why is wget copying my directory tree with some files with "@"?

I'm using wget 1.11.4 on Cygwin 1.5.25. I'm trying to recursively download a directory tree, which is the root of a javadoc tree. This is approximately the command line I tried: wget -x -p -r http://<host>/.../apidoc When it finished, it seemed like it downloaded... (0 Replies)
Discussion started by: dkarr
0 Replies

5. Shell Programming and Scripting

Command Character size limit in the "sh" and "bourne" shell

Hi!!.. I would like to know what is maximum character size for a command in the "sh" or "bourne" shell? Thanks in advance.. Roshan. (1 Reply)
Discussion started by: Roshan1286
1 Replies

6. UNIX for Advanced & Expert Users

Command Character size limit in the "sh" and "bourne" shell

Hi!!.. I would like to know what is maximum character size for a command in the "sh" or "bourne" shell? Thanks in advance.. Roshan. (1 Reply)
Discussion started by: Roshan1286
1 Replies

7. UNIX for Dummies Questions & Answers

Command Character size limit in the "sh" and "bourne" shell

Hi!!.. I would like to know what is maximum character size for a command in the "sh" or "bourne" shell? Thanks in advance.. Roshan. (1 Reply)
Discussion started by: Roshan1286
1 Replies

8. AIX

"too big" and "not enough memory" errors in shell script

Hi, This is odd, however here goes. There are several shell scripts that run in our production environment AIX 595 LPAR m/c, which has sufficient memory 14GB (physical memory) and horsepower 5CPUs. However from time to time we get the following errors in these shell scripts. The time when these... (11 Replies)
Discussion started by: jerardfjay
11 Replies

9. UNIX for Dummies Questions & Answers

Explain the line "mn_code=`env|grep "..mn"|awk -F"=" '{print $2}'`"

Hi Friends, Can any of you explain me about the below line of code? mn_code=`env|grep "..mn"|awk -F"=" '{print $2}'` Im not able to understand, what exactly it is doing :confused: Any help would be useful for me. Lokesha (4 Replies)
Discussion started by: Lokesha
4 Replies

10. UNIX for Dummies Questions & Answers

No utpmx entry: you must exec "login" from lowest level "shell"

Hi I have installed solaris 10 on an intel machine. Logged in as root. In CDE, i open terminal session, type login alex (normal user account) and password and i get this message No utpmx entry: you must exec "login" from lowest level "shell" :confused: What i want is: open various... (0 Replies)
Discussion started by: peterpan
0 Replies
Login or Register to Ask a Question