Wget, grep, sort, sed in 1 command/script Post: 302996152

Sponsored Content

Top Forums UNIX for Beginners Questions & Answers Wget, grep, sort, sed in 1 command/script Post 302996152 by p1ne on Thursday 20th of April 2017 09:18:20 AM

04-20-2017

Registered User

Code:

wget -q -r -l1 -O - URL/ |
>     grep -hrio "\b[a-z0-9.-]\+@[a-z0-9.-]\+\.[a-z]\{2,4\}\+\b" |
>     sort -u |
>     sed '0~5 a\\' > 6.txt
-k or -r can be used together with -O only if outputting to a regular file.

...and this yields an empty file in its own directory (or grabs all previous wget'd folders in shared directory--see my post above):

Code:

wget -q -r -l1 -O wget.html URL/ |     grep -hrio "\b[a-z0-9.-]\+@[a-z0-9.-]\+\.[a-z]\{2,4\}\+\b" |     sort -u |     sed '0~5 a\\' > 6.txt

So the problem persists as mentioned in my post above, that grep is not getting the wget I'm trying to attach to it. When the URL is downloaded, it creates a folder with many files, so I can see why outputting everything to an .html file (or any single file) is not going to work. Even if it did work, it's not getting piped to the grep statement, because executed in a directory with other previously wget'd URL's, the script is extracting info from all those other folders. So I need the script to access only the wget'd folder in the current command. I can do this by creating a new directory (I know, I'm repeating myself...), but I'd like to do it in the same directory.

p1ne

View Public Profile for p1ne

Find all posts by p1ne

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

sort script/command

ok. i am doing a project where i have hand typed in the titles of nearly 500 DVD titles, each one is on a seperate line. but they arent in any type of alphebetical order, and i need them sorted in that format (A-Z or a-z) ..... i know that the 'sort' command can be used but also know the...

2. Shell Programming and Scripting

Complex find grep or sed command

Haven't worked in bash for ages. did a good bit of shell scripting in regular sh, but have forgotten most of it. I have several thousand php files that now include the following line at the end of the file. There is no LF or CR/LF before it begins, it is just concatenated to the final line of...

3. UNIX for Dummies Questions & Answers

Using grep output as input for sed command

Hi, I would like to know if this is possible, and if so what can i do to make this work. I would like to grep a line X from fileA and then use the output to replace a word Y in fileB. grep "line X" fileA | sed -e 's/Y/X/g' > outfile this statement does not work, as i do not know how to...

4. Shell Programming and Scripting

Help with grep awk sed command

I have a txt file with data abc:def:ghi:jkl:mno pq stu vwx I want to take out abc what should i do? I try awk awk '/abc/ {print $1}' listfile.txt> extendedfile.txt

5. Shell Programming and Scripting

Shell script with wget in ssh command

Hi, I am using a linux with bash. I have a script written which will login to a remote server and from there it runs a "wget" to downlaod a build file from a webserver. Here is the line inside the script: ssh -t -q -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@${a}'wget...

6. Shell Programming and Scripting

Using a combination of sort/cut/grep/awk/join/paste/sed

I have a file and need to only select users that have a shell of “/bin/bash” in the line using awk or sed please help

7. Shell Programming and Scripting

Deleting lines from wget script using find/sed

I am downloading numerous files (600 plus) using a 'wget' script. Some files are not downloading and have zero byte size. I am using the following 'find' command to find the files in my cd which have non-zero byte size after the wget script has been run. find -type f -size +0 -exec basename {}...

8. Shell Programming and Scripting

applescript & grep - sed command

I'm new using Unix commands in applescript. The following script you choose different folders with PDfs, get file count of PDfs on chosen folders, & write the results in text file. set target_folder to choose folder with prompt "Choose target folders containing only PDFs to count files" with...

9. Programming

[ask]SQL command act like sort and grep

for example, I have a text file in random content inside, maybe something like this. 234234 54654 123134 467456 24234234 7867867 23424 568567if I run this command cat "filename.txt" | sort -n | grep "^467456$" -A 1 -B 1the result is 234234 467456 568567is it possible to do this command...

LEARN ABOUT DEBIAN

jigdo-lite

JIGDO-LITE(1)															     JIGDO-LITE(1)

NAME

       jigdo-lite - Download jigdo files using wget

SYNOPSIS

       jigdo-lite [ URL ]

DESCRIPTION

       See jigdo-file(1) for an introduction to Jigsaw Download.

       Given  the  URL	of  a  `.jigdo' file, jigdo-lite downloads the large file (e.g. a CD image) that has been made available through that URL.
       wget(1) is used to download the necessary pieces of administrative data (contained in the `.jigdo' file	and  a	corresponding  `.template'
       file) as well as the many pieces that the large file is made from. The jigdo-file(1) utility is used to reconstruct the large file from the
       pieces.

       `.jigdo' files that contain references to Debian mirrors are treated specially: When such a file is recognized, you are asked to select one
       mirror out of a list of all Debian mirrors.

       If  URL	is not given on the command line, the script prompts for a location to download the `.jigdo' file from. The following command line
       options are recognized:

       -h --help
	      Output short summary of command syntax.

       -v --version
	      Output version number.

       --scan FILES
	      Do not ask for "Files to scan", use this path.

       --noask
	      Do not ask any questions, instead behave as if the user had pressed Return at all prompts. This can be useful  when  running  jigdo-
	      lite from cron jobs or in other non-interactive environments.

SEE ALSO

       jigdo-file(1), jigdo-mirror(1), wget(1) (or `info wget')

       CD images for Debian Linux can be downloaded with jigdo <URL:http://www.debian.org/CD/jigdo-cd/>.

AUTHOR

       Jigsaw  Download  <URL:http://atterer.net/jigdo/>  was written by Richard Atterer <jigdo atterer.net>, to make downloading of CD ROM images
       for the Debian Linux distribution more convenient.

								    19 May 2006 						     JIGDO-LITE(1)