Wget, grep, sort, sed in 1 command/script Post: 302996152

Sponsored Content

Top Forums UNIX for Beginners Questions & Answers Wget, grep, sort, sed in 1 command/script Post 302996152 by p1ne on Thursday 20th of April 2017 09:18:20 AM

04-20-2017

Registered User

Code:

wget -q -r -l1 -O - URL/ |
>     grep -hrio "\b[a-z0-9.-]\+@[a-z0-9.-]\+\.[a-z]\{2,4\}\+\b" |
>     sort -u |
>     sed '0~5 a\\' > 6.txt
-k or -r can be used together with -O only if outputting to a regular file.

...and this yields an empty file in its own directory (or grabs all previous wget'd folders in shared directory--see my post above):

Code:

wget -q -r -l1 -O wget.html URL/ |     grep -hrio "\b[a-z0-9.-]\+@[a-z0-9.-]\+\.[a-z]\{2,4\}\+\b" |     sort -u |     sed '0~5 a\\' > 6.txt

So the problem persists as mentioned in my post above, that grep is not getting the wget I'm trying to attach to it. When the URL is downloaded, it creates a folder with many files, so I can see why outputting everything to an .html file (or any single file) is not going to work. Even if it did work, it's not getting piped to the grep statement, because executed in a directory with other previously wget'd URL's, the script is extracting info from all those other folders. So I need the script to access only the wget'd folder in the current command. I can do this by creating a new directory (I know, I'm repeating myself...), but I'd like to do it in the same directory.

p1ne

View Public Profile for p1ne

Find all posts by p1ne

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

sort script/command

ok. i am doing a project where i have hand typed in the titles of nearly 500 DVD titles, each one is on a seperate line. but they arent in any type of alphebetical order, and i need them sorted in that format (A-Z or a-z) ..... i know that the 'sort' command can be used but also know the...

2. Shell Programming and Scripting

Complex find grep or sed command

Haven't worked in bash for ages. did a good bit of shell scripting in regular sh, but have forgotten most of it. I have several thousand php files that now include the following line at the end of the file. There is no LF or CR/LF before it begins, it is just concatenated to the final line of...

3. UNIX for Dummies Questions & Answers

Using grep output as input for sed command

Hi, I would like to know if this is possible, and if so what can i do to make this work. I would like to grep a line X from fileA and then use the output to replace a word Y in fileB. grep "line X" fileA | sed -e 's/Y/X/g' > outfile this statement does not work, as i do not know how to...

4. Shell Programming and Scripting

Help with grep awk sed command

I have a txt file with data abc:def:ghi:jkl:mno pq stu vwx I want to take out abc what should i do? I try awk awk '/abc/ {print $1}' listfile.txt> extendedfile.txt

5. Shell Programming and Scripting

Shell script with wget in ssh command

Hi, I am using a linux with bash. I have a script written which will login to a remote server and from there it runs a "wget" to downlaod a build file from a webserver. Here is the line inside the script: ssh -t -q -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@${a}'wget...

6. Shell Programming and Scripting

Using a combination of sort/cut/grep/awk/join/paste/sed

I have a file and need to only select users that have a shell of “/bin/bash” in the line using awk or sed please help

7. Shell Programming and Scripting

Deleting lines from wget script using find/sed

I am downloading numerous files (600 plus) using a 'wget' script. Some files are not downloading and have zero byte size. I am using the following 'find' command to find the files in my cd which have non-zero byte size after the wget script has been run. find -type f -size +0 -exec basename {}...

8. Shell Programming and Scripting

applescript & grep - sed command

I'm new using Unix commands in applescript. The following script you choose different folders with PDfs, get file count of PDfs on chosen folders, & write the results in text file. set target_folder to choose folder with prompt "Choose target folders containing only PDFs to count files" with...

9. Programming

[ask]SQL command act like sort and grep

for example, I have a text file in random content inside, maybe something like this. 234234 54654 123134 467456 24234234 7867867 23424 568567if I run this command cat "filename.txt" | sort -n | grep "^467456$" -A 1 -B 1the result is 234234 467456 568567is it possible to do this command...

LEARN ABOUT MINIX

grep

GREP(1) 						      General Commands Manual							   GREP(1)

NAME

       grep - search a file for lines containing a given pattern

SYNOPSIS

       grep [-elnsv] pattern [file] ...

OPTIONS

       -e     -e pattern is the same as pattern

       -c     Print a count of lines matched

       -i     Ignore case

       -l     Print file names, no lines

       -n     Print line numbers

       -s     Status only, no printed output

       -v     Select lines that do not match

EXAMPLES

       grep mouse file	   # Find lines in file containing mouse

       grep [0-9] file	   # Print lines containing a digit

DESCRIPTION

       Grep  searches  one  or	more  files (by default, stdin) and selects out all the lines that match the pattern.  All the regular expressions
       accepted by ed and mined are allowed.  In addition, + can be used instead of * to mean 1 or more occurrences, ? can be used to mean 0 or  1
       occurrences,  and  |  can  be used between two regular expressions to mean either one of them.  Parentheses can be used for grouping.  If a
       match is found, exit status 0 is returned.  If no match is found, exit status 1 is returned.  If an error is detected,  exit  status  2	is
       returned.

SEE ALSO

       cgrep(1), fgrep(1), sed(1), awk(9).

																	   GREP(1)