UNIX for Beginners Questions & Answers

Hi zajtat,
Yes, the first invocation of combining.executable:

Code:

./match.pl -f 9477885102_R01C02header -g 9477885102_R01C01header -k 1 -l 1 -v '5 6 7' > 0

creates a file named 0 and the second invocation of combining.executable:

Code:

./match.pl -f 9477885102_R02C01header -g 0 -k 1 -l 1 -v '5 6 7' > 1

uses file 0 and creates file 1. Subsequent passes through the loop alternate between creating file 0 and using file 1 on odd numbered invocations (after invocation 1) and creating file 1 and using file 0 on even numbered invocations.

And, on OS X, the script I suggested works with both bash (version: GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin15)) and with ksh producing the output you said you wanted with the three input files you provided as samples in post #1 in this thread (except the order of the 1st two lines in the output was switched) with combining.executable being the script:

Code:

#!/bin/bash
awk '
FNR == NR {
	k[$1, $2, $3] = $0
	next
}
($1, $2, $3) in k {
	print k[$1, $2, $3], $(NF - 2), $(NF - 1), $NF
	next
}
1' "$2" "$1"

which works fine as long as the input files are text files. Note that with the number of files you are combining, the above script might or might not work. The awk utility is only required to work on text files and some of the files this script will be processing will not be text files (due to excessive line lengths).

If your perl script won't accept 0 as a filename, that is your problem. The script I provided (after changing the 1st line to use your shell) meets all of your specifications.

If you would like to give us a clear specification of the allowed filenames for the -g option to your perl script, I might be able to help you make the script I suggested use different temporary filenames.

Also note that bakunin has a typo in his bash invocation second part:-

#! /bin/bash

......shloud be......

#!/bin/bash

......without the whitespace between "!" and "/".

EDIT:
I forgot to add, that he actually mentioned this is his text, but forgot to correct it.

Thank you all for your suggestions and explanations!

Sorry to correct you, but this is NOT a typo and it was indeed intended that way. Here is a bit of UNIX history:

The UNIX kernel has to have some clue about what constitutes an executable file. OK, there is the x-bit in the filemode, but still, the kernel has to know what to do. For binary files there is the so-called "magic number", which is the first 4 bytes of an executable. Look at any first 4 bytes of binary, compiled files in your system and you will see that they are the same.These 4 bytes tell the kernel to use some member of the exec*() family of system calls to actually execute it.

Unfortunately this is not possible with the input files of the many interpreters a system has: all sorts of shells, awk, sed, many editors (ed, ex, ...) which can be scripted, and so on. Historically there was only one shell and the UNIX kernel had a special provision to try invoking the shell with the file as input in case the magic number was not understandable. Today this has been altered to the system default shell and is still in place: you can see it at work when you write a script without any shebang line. In this case the startup process is like: the kernel finds no valid magic number, then loads the system default shell (which is a normal executable binary) via normal exec*(), then feeds it the file in the hope that the shell can make sense of it. If it is indeed a shell script, it does. If not, you will see some error message, which is coming from the shell, not the kernel itself.

But as UNIX evolved and many different shells (and other interpreters of script-like text) were available the developers of UNIX searched for ways to name the specific shell/interpreter for scripts. The UNIX kernel was extended for a new magic number and this four-byte code was:

Code:

#! /

If the kernel finds this magic number it reads the rest of the first line, loads the specified interpreter and - upon succeeding in that - feeds it the file as input. Because the '#' is a comment to the shell it doesn't alter any script at all, the shell will just ignore it like any other comment.

This is the reason why you have to have the shebang in the very first line and why it is not allowed to be indented and why you have to use the absolute path of the shells executable: #! /bin/bash is a legal shebang, #! ../../bin/bash is not.

So, basically, the correct shebang line is:

Code:

#! /path/to/interpreter

Unfortunately most people were not able to reliably put a single space where it belongs and many wrote

Code:

#!/path/to/interpreter

Which was still a normal comment and the kernel would - failing to recognize the magic number - still load the default shell, but maybe not the specified one. Which is why the kernel was extended again to also recognize a special 3-byte magic number, which was the shebang without the space.

This is the situation we have now. Basically it doesn't matter any more if you write the space or not, but, being as old as i am, you have to have something that sets you apart from the youngsters (save for having to get up at three in the morning to pee). Therefore i always write the "originally correct" shebang, and not these newly-invented hacks which are only going back to the seventies.

I hope this helps.

bakunin

Hi bakunin,
Sorry to disagree with you here, but back in the early UNIX days on the PDP-11, the magic number determining the type of executable was 2 bytes (16-bits) not 4 bytes. And, when #! was added to the magic numbers recognized by the kernel, a leading space was not allowed. Since then, some kernels allow one or more leading spaces, some kernels allow a single option (e.g., #!/bin/sh -xv), and some kernels may even invoke a shell to evaluate the entire first line starting from the 3rd character as a shell command with the rest of the file as input (although I am not aware of any of these systems that are still produced today).

I believe that the PWB UNIX Systems I used when I was learning the OS treated:

Code:

#! /bin/sh

as a request to run the sh utility in the bin directory in the <space> directory, but I don't remember if it interpreted it as / /bin/sh or as ./ /bin/sh.

I was going through the post again to look over things and have double-checked that the perl script works fine with files that have numbers as names. There are no bugs in perl script and there has never been such a problem, that's why I've answered a question about the name of the output file that it does not matter. I did try to provide clear information and answer all the questions to the best of my abilities and I apologise that it was not understandable. Thank you all of your time and help once more!

The trace output you showed us in post #11 in this thread indicated that the command:

Code:

./match.pl -f 9477885102_R01C02header -g 9477885102_R01C01header -k 1 -l 1 -v '5 6 7' > 0

succeeded, that the next command:

Code:

./match.pl -f 9477885102_R02C01header -g 0 -k 1 -l 1 -v '5 6 7' > 1

failed producing the diagnostic:

Code:

Must specify file one using -g switch

and, then it appears that the command:

Code:

./match.pl -f 9477885102_R02C02header -g 1 -k 1 -l 1 -v '5 6 7' > 0

succeeds (although we didn't look at enough of the trace output to be sure of that).

You say that your script works with an input file named 0, but you have shown us diagnostic output from the script (as copied above) that clearly shows that it does not. Should we believe what you showed us in post #11 or should we believe what you said in post #20? They are in direct conflict with each other.