Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

filename in the script as input

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting filename in the script as input
# 1  
Old 11-30-2011
filename in the script as input
Hey guys,

I have a list of urls which I want to download and analyze with bash and lynx, I'm using this codefor each url:

downloader.sh:


Code:
#!/bin/bash
lynx -source http://domain.com/product.php?e=123456 > 123456.html
lynx -dump 123456.html > 1234567.dat
cat 123456.html | grep -A 1 "<title>" >> 123456.dat

I'm storing the HTML and dumped HTML and adding the title to the end of the dump file for further analysis.

and I have a list of webpages IDs like this

IDlist.dat:

Code:
123456
515408
515407
925723
590005
540236
675455
515406
992468
355053
104286
527371
637974
963469

So I don't know how to do that recursively with the downloader.sh for example by writing this command, I want downloader.sh to run for all of above IDs:

Code:
downloader.sh IDlist.dat

tnx
# 2  
Old 11-30-2011
That's a useless use of cat, grep and any other command which reads stdin is quite capable of reading a file without it. Just redirect in the shell.

Code:
while read ID
do
        lynx -source "http://domain.com/product.php?e=${ID}" > /tmp/$$.html
        lynx -dump /tmp/$$.html > ${ID}.dat
        grep -A 1 "<title>" < /tmp/$$.html >> "${ID}.dat"
done < listfile

rm -f /tmp/$$.html

This User Gave Thanks to Corona688 For This Post:
Johanni
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Space in input filename with pipe

Hello, Normally below script works, could you please comment out what could be the reason of failure if there are spaces in input filename: script.sh #!/bin/bash cd /home/hts/.hts/tvh/ file="$1 $2 $3 $4" read -d $'\x04' name < "$file" /usr/bin/ffmpeg -i ""$name"" -vcodec copy -preset... (1 Reply)
Discussion started by: baris35
1 Replies

2. Shell Programming and Scripting

Use while loop to read file and use ${file} for both filename input into awk and as string to print

I have files named with different prefixes. From each I want to extract the first line containing a specific string, and then print that line along with the prefix. I've tried to do this with a while loop, but instead of printing the prefix I print the first line of the file twice. Files:... (3 Replies)
Discussion started by: pathunkathunk
3 Replies

3. Open Source

Splitting files using awk and reading filename value from input data

I have a process that requires me to read data from huge log files and find the most recent entry on a per-user basis. The number of users may fluctuate wildly month to month, so I can't code for it with names or a set number of variables to capture the data, and the files are large so I don't... (7 Replies)
Discussion started by: rbatte1
7 Replies

4. Shell Programming and Scripting

Call a awk script with variable and input filename

HI, MY question is a very simple one: if i want to call an awk script with the input file name and also pass a variable value , then how to do it. #>awk -f my_script.awk -v variable=value my_inputfile.txt I can't do it like this. throws error: awk: my_script.awk:18:... (0 Replies)
Discussion started by: Onkar Banerjee
0 Replies

5. Shell Programming and Scripting

Script to delete files with an input for directories and an input for path/file

Hello, I'm trying to figure out how best to approach this script, and I have very little experience, so I could use all the help I can get. :wall: I regularly need to delete files from many directories. A file with the same name may exist any number of times in different subdirectories.... (3 Replies)
Discussion started by: *ShadowCat*
3 Replies

6. Shell Programming and Scripting

Read input and output redirection filename within a script

Hello everyone, My requirement is that within a script I need to construct the command line exactly that it was invoked with. For example : sh a.sh arg1 arg2 arg3 < input.txt > output.txt Now within a.sh, I construct a file which has these contents " sh a.sh arg1 arg2 arg3 < input.txt >... (8 Replies)
Discussion started by: hedonist12
8 Replies

7. Shell Programming and Scripting

use input filename as an argument to name output file

I know this is a simple matter, but I'm new to this. I have a shell script that calls a sed script from within it. I want the output of the shell script to be based on the input file I pass as an argument to the original script. In other words... ./script.sh file.txt (script.sh calls sed... (2 Replies)
Discussion started by: estebandido
2 Replies

8. Shell Programming and Scripting

Filename from splitting files to have the same filename of the original file with counter value

Hi all, I have a list of xml file. I need to split the files to a different files when see the <ko> tag. The list of filename are B20090908.1100-20090908.1200_CDMA=1,NO=2,SITE=3.xml B20090908.1200-20090908.1300_CDMA=1,NO=2,SITE=3.xml B20090908.1300-20090908.1400_CDMA=1,NO=2,SITE=3.xml ... (3 Replies)
Discussion started by: natalie23
3 Replies

9. Shell Programming and Scripting

Need script to take input from file, match on it in file 2 and input data

All, I am trying to figure out a script to run in windows that will allow me to match on First column in file1 to 8th Column in File2 then Insert file1 column2 to file2 column4 then create a new file. File1: 12345 Sam 12346 Bob 12347 Bill File2:... (1 Reply)
Discussion started by: darkoth
1 Replies

10. Shell Programming and Scripting

AWK set FILENAME via user input

I am trying to write a awk script that prompts user for input to set the FILENAME varable. I can get it set, but I think awk is not doing anything with it. this is what I have so far #!/usr/bin/nawk -f BEGIN { FILENAME = "" printf "Enter name of file to check in : " ... (2 Replies)
Discussion started by: timj123
2 Replies
Login or Register to Ask a Question