Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Help with curl piped to awk

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Help with curl piped to awk
# 8  
Old 10-31-2012
oops, there is no output when redirected to a file as earlier..

Best Regards,
Vinu
# 9  
Old 10-31-2012
I don't understand!

You confirmed that you got 83.5 MB as output, you just have to modify the same command by printing first parameter ($1) which is the URL.
# 10  
Old 10-31-2012
Hi,

yes.. the command is working..though not on that earlier url which has now expired and the page does not have the 83.5 mb on it..

tried on another url as in out6.txt.. looks like there is some unwanted data..could you help in filtering these..

Best Regards,
Vinu
out6.txt (338 Bytes) 
# 11  
Old 10-31-2012
Got it. Looks like the best option is to pass the URL to awk to print it beside the extracted file size value:-

Code:
curl -iSA "Mozilla/5.001" http://hipfile.com/1b0kksnik2pn |grep "http://hipfile.com/1b0kksnik2pn" | awk -F"[)(]" ' { print $2 " http://hipfile.com/1b0kksnik2pn" } '

I hope it helps.
This User Gave Thanks to Yoda For This Post:
vinu2k3
# 12  
Old 10-31-2012
Yes!Smilie

that was the best method at least presently..Smilie

Also, Bipinajith .. could you help further in other problematic urls..? in your free time ofcourse ..

just got one moreSmilie as in url2.txt.. where 83.82 MB is to be got from..

Best Regards,
Vinu
url2.txt (988 Bytes) 
# 13  
Old 10-31-2012
Is it really necessary to attach such small files? Can't you just paste the contents between code tags?

My downloads folder already has enough stuff in it Smilie
# 14  
Old 10-31-2012
Fine Scott.. oops!! had forgotten about the 5 posts requirement to post urls..Smilie
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Use cut output as variable piped awk command

Hi, I would like use the output of my cut command as a variable in my following awk command. Here's what I've written. cut -f1 info.txt | awk -v i=xargs -F'' '{if($6 == $i) print $20}' summary.txt Where obviously the 'xargs' doesn't do what I want. How can I pass my cut result to my awk... (3 Replies)
Discussion started by: heyooo
3 Replies

2. Shell Programming and Scripting

awk statement piped inside sed

Hello folks, I have multiple occurrences of the pattern: ).: where is any digit, in various text context but the pattern is unique as this regex. And I need to turn this decimal fraction into an integer (corresponding percent value: the range of 0-100). What I'm doing is: cat... (1 Reply)
Discussion started by: roussine
1 Replies

3. Shell Programming and Scripting

Sending awk variables into curl in a bash script

Hello experts! I have a file1 with the following format (yr,day, month, hour,minute): 201201132435 201202141210 201304132030 201410100110 ... What i want to do is to assign variables and then use them in the curl command to download the text of each event from a web page. What I have... (6 Replies)
Discussion started by: phaethon
6 Replies

4. Shell Programming and Scripting

Output piped to case insensitive awk

I have an encrypted password file, and I've created a simple script to search the password file for a particular record. There are multiple lines per record, so I'm using a record delimiter. #!/bin/bash PATTERN=$1 openssl des3 -d -salt -in ~/docs/pass.des3 | awk '{ FS="\n" ; RS="*" }... (2 Replies)
Discussion started by: 0rac1e
2 Replies

5. Shell Programming and Scripting

Perl: How to check if there is something being piped in

Hi, I am somewhat new to Perl and currently checking it out. I have a problem testing, if there is nothing being piped in to that script. I am reading input from STDIN this way: while( defined($line = <STDIN>) ) { chomp($line); if( $line =~ m/($ARGV)/g ) { ... (7 Replies)
Discussion started by: zaxxon
7 Replies

6. Shell Programming and Scripting

Cat piped output

Hello, How can I efficiently cat piped output with another file? > (awk command) | cat file1 (piped output) Thanks! (11 Replies)
Discussion started by: palex
11 Replies

7. Programming

How to use a debugger a piped program ?

Hi, I have 1 program that writes in to the STDIN of another program as shown below. Both programs contain 4 or 5 lines & would terminate in under a second. $ driver.exe | program.exe How is that I can attach the debugger (gdb) to program.exe ? so that I can step through and see what all... (0 Replies)
Discussion started by: RipClaw
0 Replies

8. Shell Programming and Scripting

Using piped input to gzip

Can anyone tell me why does'nt the following command work? find /gfp1/home/arijit -name "*.sas7bdat" | gzip I am trying to compress all files with extension sas7bdat with gzip. It gives error message gzip: compressed data not written to a terminal. Use -f to force compression.... (5 Replies)
Discussion started by: bs0409
5 Replies

9. Shell Programming and Scripting

KSH variable containing piped commands

Hi, I am trying to execute a bunch of piped command which are stored in a variable, but only one command executed at a time, and rest of the pipes, commands and their arguments are considered as argument to the very first command. Can you please help me in this? bash-2.05$ cat test.sh... (1 Reply)
Discussion started by: prashant.ladha
1 Replies

10. Shell Programming and Scripting

Return value of piped command?

grep $SEARCH_STRING /etc/passwd | cut -d":" -f 1,5 I need to check the $? value of grep in the above. If I place a test for $? after the above piped command, it returns success status of grep piped to cut. How can I get the success status of grep alone? (5 Replies)
Discussion started by: krishmaths
5 Replies
Login or Register to Ask a Question