Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Cygwin script log redirection not working well

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Cygwin script log redirection not working well
# 1  
Old 11-02-2012
Cygwin script log redirection not working well
I have a simple script which will send a curl request and redirect the output to a log file.

Code:
for i in {1..20}
do
 curl google.com -is >>log.log &
 echo "request # $i" >> log.log
done

After it completes the execution, if I run the following command I should see 20 lines because I am printing it in the loop.
Code:
cat log.log|grep request|wc -l

But I see 1 or 2 lines and all my log is messed up. I know that I am sending curl command to the background so the log will be mixed up. But I should see 20 "request" when I grep the log.

The same script works fine in my ubuntu. Also it works fine in my friends cygwin. I uninstalled cygwin and installed it back but no change. I guess it is something with my windows configuration? Anyone know why it is happening?

My messed up log
Code:
request # 1
HTTP/1.1 301HTTP/1.1 301HTTP/1.1 301HTTP/1.1 301 Moved Permanently
Location: http://www.google.com/
Content-Type: text/html; charset=UTF-8
Date: Fri, 02 Nov 2012 14:22:32 GMT
Expires: Sun, 02 Dec 2012 14:22:32 GMT
Cache-Control: public, max-age=2592000
Server: gws
Content-Length: 219
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN

<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.googHTTP/1.1 301 Moved PermaHTTP/1.1 301HTTP/1.1 301HTTP/1.1 301 Moved Permanently
Location: http://www.google.com/
Content-Type: text/html; charset=UTF-8
Date: Fri, 02 Nov 2012 14:22:32 GMT
Expires: Sun, 02 Dec 2012 14:22:32 GMT
Cache-Control: public, max-age=2592000
Server: gws
Content-Length: 219
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN

<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.googHTTP/1.1 301 HTTP/1.1 301 HTTP/1.1 301 HTTP/1.1 301 HTTP/1.1 301 Moved Permanently
Location: http://www.google.com/
Content-Type: text/html; charset=UTF-8
Date: Fri, 02 Nov 2012 14:22:32 GMT
Expires: Sun, 02 Dec 2012 14:22:32 GMT
Cache-Control: public, max-age=2592000
Server: gws
Content-Length: 219
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN

<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">hHTTP/1.1 301 HTTP/1.1 301 HTTP/1.1 301 HTTP/1.1 301 HTTP/1.1 301 HTTP/1.1 301 Moved Permanently
Location: http://www.google.com/
Content-Type: text/html; charset=UTF-8
Date: Fri, 02 Nov 2012 14:22:32 GMT
Expires: Sun, 02 Dec 2012 14:22:32 GMT
Cache-Control: public, max-age=2592000
Server: gws
Content-Length: 219
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN

<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.gHTTP/1.1 301 Moved Permanently
Location: http://www.google.com/
Content-Type: text/html; charset=UTF-8
Date: Fri, 02 Nov 2012 14:22:32 GMT
Expires: Sun, 02 Dec 2012 14:22:32 GMT
Cache-Control: public, max-age=2592000
Server: gws
Content-Length: 219
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN

<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>

# 2  
Old 11-02-2012
You are running curl in the background. It runs independently -- ergo, it prints when it pleases. Whether it will print lines precisely when you want it to is entirely coincidental. It could even print overtop of other lines sometimes but probably not all the time.

This is not a bug. This is just because you have no control over the order they print.

If you must run them in parallel, each instance of curl should write to its own logfile, IMHO. You can merge them after they're finished but not before.
# 3  
Old 11-02-2012
Something like:
Code:
for i in {1..20}
do
 curl google.com -is >log-${i}.log &
done

wait

for i in {1..20}
do
        cat log-${i}.log
        rm log-${i}.log
        echo "request # $i"
done > log.log

# 4  
Old 11-02-2012
I understand that the log will be mixed up as I am running the curl command in the background. But the echo request should be in the log for 20 times right? I tested my script in different computers and they all show request 20 times in the log. It is just not working in my computer on cygwin. If you look at my log.log above, I am fine with the curl response mixed up but I see only one line "request # 1". The remaining echos are missing. As you said it may be coincidental. But I am not sure why it works every time in other computers.

Thanks for the code you sent. I guess I can use it in my machine to fix the issue.
Last edited by heykiran; 11-02-2012 at 12:59 PM.. Reason: Added more info
# 5  
Old 11-14-2012
Quote:
Originally Posted by heykiran
I understand that the log will be mixed up as I am running the curl command in the background. But the echo request should be in the log for 20 times right?
Not necessarily.

Process 1 writes to the file.

Process 2 writes a fraction later, and isn't aware the file has been appended to, overwriting the section the previous one just wrote. Or worse, overwriting part of the section, since one may be longer than the other.

That code works "a lot of the time" doesn't mean it won't occasionally blow up in your face, in ways you can't predict. In short, don't make them compete for the file like that.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

>& redirection not working within csh script

I'm having a strange problem with basic >& output redirection to a simple log file in csh. When I run this particular output redirection on the command line, it works, but then when I run the same output redirection command >& in my c shell script, I get a blank log file. Nothing is output to the... (5 Replies)
Discussion started by: silencio
5 Replies

2. Red Hat

Log Redirection to Remote Host

hey, I need your help. I have a rhel server that runs application, that application writing to her local application log. I want that log will be realtime copying to a another remote rhel server. The thing is that I prohibited touch the application code. Is that possible? ... (5 Replies)
Discussion started by: moshesa
5 Replies

3. UNIX and Linux Applications

Cygwin application not working on my machine.

I have windows XP machine. Today i have tried to install Cygwin, I've downloaded all the packages then ran the setup. It's showed completed. But when i try to open this it's not working. Do i need to do anything extra for this..? Please help... (3 Replies)
Discussion started by: pamu
3 Replies

4. Shell Programming and Scripting

Command output redirection in script not working

I need to count the number of lines in a .txt file and put it in a variable. I am using the following code #!/bin/bash count = $(wc -l "some file.txt" | awk '{print$1}') echo $count It is giving the following error. line3: count: command not foundWhat am I doing wrong here? :confused: (7 Replies)
Discussion started by: haritha.gorijav
7 Replies

5. Shell Programming and Scripting

SED on cygwin not working with Hex or Octal

Hi, I have downloaded a web page that I need to cleanup before passing to xmlstarlet. Using UltraEdit's HEX utility part of my download is as follows: 3C 2F 61 3E 0A 09 0A 09 09 3C 2F 61 3E which in ASCII is </a> </a> I need to locate this string and replace it with just... (7 Replies)
Discussion started by: dazhoop
7 Replies

6. Shell Programming and Scripting

stderr redirection not working

I am running tcsh/csh shell on my machine. lately i have realized my stderr file redirection is not working. Please find the terminal logs as below: >echo b c >>& log >cat log b c >echo $a b c >>& log a: Undefined variable. >cat log b c I have never faced such issues, hence not sure... (5 Replies)
Discussion started by: animesharma
5 Replies

7. Shell Programming and Scripting

opening new instance of cygwin from withing cygwin

I'm using cygwin on win7, What I would like to do is something like this: cygstart cygwin tail -f /foo/test.log | perl -pe 's/error/\e I know I can start a new instance using either of these: mintty -e ... cygstart tail ... But neither of those open in ANSI mode, so I can't do... (0 Replies)
Discussion started by: Validatorian
0 Replies

8. UNIX for Dummies Questions & Answers

Capping output redirection log file length

I am trying to write a script which will output notifications to a logfile, but I would like to cap the logfile to, let's say, 200 lines. Specifically I am using custom firmware, DD-wrt, on my router and I am implementing a script to connect to my work vpn. I have a loop that pings a computer... (2 Replies)
Discussion started by: joemommasfat
2 Replies

9. UNIX for Dummies Questions & Answers

Redirection of output to a log file

Apologies for the trivial nature of this question but I cannot seem to get a simple re direct to a log file to work Step 1 touch log.txt at -f batch.sh now >> log.txt I am trying to get the batch.sh contents into the log file Manny Thanks (8 Replies)
Discussion started by: JohnCrump
8 Replies

10. UNIX for Dummies Questions & Answers

Redirection not working as expected

Portion of my script below : if ; then NUMBEROFFEILDS=`cat ${BASE_SCRIPT_LOC}/standardfilecleanup.lst|grep -w ${db_file_path}|awk -F: '{print NF}'` COUNT=4 while ; do awk_var="$"`echo $COUNT` file_name1=`cat ${BASE_SCRIPT_LOC}/standardfilecleanup.lst|grep -w... (1 Reply)
Discussion started by: findprakash
1 Replies
Login or Register to Ask a Question