Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Mkbootfs writing to stdout in bash script

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Mkbootfs writing to stdout in bash script
# 15  
Old 04-18-2013
Very interesting

splitting the cmd line like you suggested works perfectly, with or without quoting the variables

Now, running my single line $CMD_LINE1 with -xv debug mode gives this:

Code:
#!/bin/bash -xv
CMD_LINE1="/root/Desktop/build_cm10/android_cm-10.1/system/out/host/linux-x86/bin/mkbootfs /root/Desktop/build_cm10/android_cm-10.1/system/out/target/product/n7100/recovery/root > /root/Desktop/build_cm10/android_cm-10.1/system/out/target/product/n7100/ramdisk-recovery.cpio"
+ CMD_LINE1='/root/Desktop/build_cm10/android_cm-10.1/system/out/host/linux-x86/bin/mkbootfs /root/Desktop/build_cm10/android_cm-10.1/system/out/target/product/n7100/recovery/root > /root/Desktop/build_cm10/android_cm-10.1/system/out/target/product/n7100/ramdisk-recovery.cpio'
$CMD_LINE1
+ /root/Desktop/build_cm10/android_cm-10.1/system/out/host/linux-x86/bin/mkbootfs /root/Desktop/build_cm10/android_cm-10.1/system/out/target/product/n7100/recovery/root '>' /root/Desktop/build_cm10/android_cm-10.1/system/out/target/product/n7100/ramdisk-recovery.cpio
error: cannot open directory '>'

So, it seems the > was quoted into ' ' by the shell
I tried to escape the > using \> in my $CMD_LINE1, but it did not fix it

Just want to understand things. Is there any way to escape the > and have the shell stop quoting it?
# 16  
Old 04-18-2013
Shell scripting does not work this way. Once a string is inside a variable, it doesn't get reparsed for quotes, escapes, or substitutions anymore.

You'd have to use eval, which is a very bad idea for many reasons... One, it makes your code needlessly convoluted; adding substitution to your substitution is nesting layers of doublethink that are extremely difficult for anyone to program and understand. Two, it's very insecure and a horrible habit; if someone manages to put `rm -Rf ~/` into one of your variables, the shell will execute that!

As others have said, there's almost no good reasons to do this. It's a typical beginner mistake from not knowing the right kinds of substitution yet. You can kludge all gaps with eval since it can do anything at all and build footbridges out of toothpicks, but it'd be simpler and safer and faster to use the thoroughfares that are already there.

If I understood why you were putting it into a variable, I could probably show you how to do what you wanted without this problem.

And no, I don't mean how to store a complete shell statement in a variable. I mean, what you wanted to accomplish by storing it in a variable.
Last edited by Corona688; 04-18-2013 at 12:49 PM..
This User Gave Thanks to Corona688 For This Post:
Phil3759
# 17  
Old 04-18-2013
Code:
So, it seems the > was quoted into ' ' by the shell
I tried to escape the > using \> in my $CMD_LINE1, but it did not fix it

Just want to understand things. Is there any way to escape 
the > and have the shell stop quoting it?

I don't know the answer. It's a good question you ask. However, I would suggest "just don't do it". And as pointed out by the other poster, there is the complex "eval" command that might work. Anyway, you observed:
Code:
$ cat test.sh
set -x
CMD='date > date.txt'
$CMD

Code:
$ ./test.sh
++ CMD='date > date.txt'
++ date '>' date.txt
date: extra operand `date.txt'
Try `date --help' for more information.

It's good you're curious. I am too. I tried several ways to escape the redirection. The shell always insisted on quoting it, as you observe.

It is quoting the redirection for a reason. I cannot fathom the reason. But someone a lot smarter than me wrote the shell and decided it was needed to quote that redirection.

The shell is a very complex beast. It's trying to do many things in a hostile environment. It's not perfect, has many design compromises, has evolved over many years. You have to work within it's limits, not overtax it.

If it were really helpful in this case to put the entire command line (including redirection) in a variable, and then run the variable, this would be worth trying to figure out. You're right to think perhaps the syntax "should" work in some way. But it seemingly doesn't. Smilie
This User Gave Thanks to hanson44 For This Post:
Phil3759
# 18  
Old 04-18-2013
Thank you both, really

@Corona688: no, I am not going to use eval of course, but like hanson44 said, I am very curious.

Why the command is in a variable?
Well, the script sample is part of a long complex script that automates compiling custom android recovery, extracting stock android recovery images, repacking compiled image with ramdisk from a different previously extracted stock image, do incremental backups of all the output, make and sign flashable zips and tar.md5 files if needed...

Basically, I enter something like:
Code:
philz_repack.sh 4.93.6 i9300

and it processes everything up to the final step

The script handles many different devices. Each device has a different hardware that needs to be handled...

The shell sample I gave was just an example of part of the repacking process for just one device.
In fact, every part of the line is assigned to variables for paths, device_id, kernel cmdline for the device... So, I populated the variables to make it easier to understand.

As I said, I worked around it using combination of cpio and gzip
Now, thanks to hanson44, I have another workaround by splitting that command line in 3 and putting the redirection > outside the variable
However, it would be great if some one can explain why the bash is quoting the redirection symbol '>' when inside a string like hanson44 showed it in his example!
Last edited by Phil3759; 04-18-2013 at 07:09 PM..
# 19  
Old 04-18-2013
I would also like to know. My best guess is that shell quotes the redirection in this context because the redirection symbol can also be used as a literal character, and it was decided the literal meaning should take precedence. They (the creators of shell) had to make a decision, and chose the path most generally useful. They perhaps ruled out putting an entire command line, including redirection, within a variable as being not needed. Look at the following which I think shows the advantage of quoting the redirection:
Code:
$ cat input
111>222>333

Code:
$ cat test.sh
set -x
cut -d ">" -f 2 input
CMD='cut -d > -f 2 input'; $CMD   # works because of quoting
CMD='cut -d ">" -f 2 input'; $CMD # fails, but we have previous way

Code:
$ ./test.sh
++ cut -d '>' -f 2 input
222
++ CMD='cut -d > -f 2 input'
++ cut -d '>' -f 2 input
222
++ CMD='cut -d ">" -f 2 input'
++ cut -d '">"' -f 2 input
cut: the delimiter must be a single character
Try `cut --help' for more information.

This User Gave Thanks to hanson44 For This Post:
Phil3759
# 20  
Old 04-18-2013
Quote:
Originally Posted by hanson44
I would also like to know. My best guess is that shell quotes the redirection in this context because the redirection symbol can also be used as a literal character, and it was decided the literal meaning should take precedence. They (the creators of shell) had to make a decision, and chose the path most generally useful. They perhaps ruled out putting an entire command line, including redirection, within a variable as being not needed. Look at the following which I think shows the advantage of quoting the redirection:
Code:
$ cat input
111>222>333

Code:
$ cat test.sh
set -x
cut -d ">" -f 2 input
CMD='cut -d > -f 2 input'; $CMD   # works because of quoting
CMD='cut -d ">" -f 2 input'; $CMD # fails, but we have previous way

Code:
$ ./test.sh
++ cut -d '>' -f 2 input
222
++ CMD='cut -d > -f 2 input'
++ cut -d '>' -f 2 input
222
++ CMD='cut -d ">" -f 2 input'
++ cut -d '">"' -f 2 input
cut: the delimiter must be a single character
Try `cut --help' for more information.

yes, potential good explanation
It is really the first time I use such a way
In any case it was my initial debugging script. The final script goes the clean ways like you did, by splitting things in a command variable

Despite my issue was fixed since page 1, I leave this thread open if someone can add an input / explanation about the shell quoting redirection inside a variable
# 21  
Old 04-18-2013
The shell does these and only these things to an unquoted $VARIABLE:
  • Splits into arguments on whitespace. (Technically, it splits on any character in the IFS variable, which you can alter at need.)
  • As a side effect, flattens whitespace. A variable containing " a b c" will become the arguments a, b, and c without leading, trailing, or any extra spaces.
  • Expand wildcards like * and ? from each argument into filenames. You can disable this if unwanted via set -f, and re-enable with set +f

It will not parse anything else. Not quotes, not variables, not backticks, not redirection. It simply takes them as literal parts of the string. It was giving mkbootfs the string > as an argument.

Take a look:

Code:
$ echo 'asdf'

asdf

$ VAR="echo 'asdf'"
$ $VAR

'asdf'

$

Imagine that you had to escape the > character every time you actually wanted to print that letter. How many special cases would there be? Should escaping happen before it gets in the string or after? Does the string need an actual \ inside it or not? How many? What if you want an actual backslash? How many different things would you end up needing to escape to get them into a string? There'd be so many weird special cases that BASH code would start looking like Perl.

Now imagine your program reads a string from the keyboard, and someone decides to be clever by typing in `rm -Rf ~/`. Should the shell blindly execute that shell statement next time you use that variable? Keeping your code secure from unintended situations would be near impossible.

For these and other reasons, the Bourne shell is very strict and consistent about when substitution happens. If it's not in the file, it doesn't do it, unless you force it with eval.

eval will blindly substitute this way, and for this reason, must be used with extreme care.
This User Gave Thanks to Corona688 For This Post:
Phil3759
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

[BASH] Performance question - Script to STDOUT

Hello Coders Some time ago i was asking about python and bash performances, and i was told i could post the regarding code, and someone would kindly help to make it faster (if possible). If you have noted, i'm on the way to finalize, finish, stable TUI - Text(ual) User Interface. It is a... (6 Replies)
Discussion started by: sea
6 Replies

2. Shell Programming and Scripting

Writing Hbase and pig scripts in the bash script file

Hi, I have a script file where i'm validatig the input file and storing the validated records on HDFS. I wanted to load data from HDFS to HBASE using pig script. So for that i have created a HBASE table and written pig script to load data from HDFS to HBASE which is working fine. Now i wanted... (0 Replies)
Discussion started by: shree11
0 Replies

3. Shell Programming and Scripting

Writing hive scripts in bash script file

Hi, I wanted to load data from HDFS to HIVE by writing bash script. Description: I have written a bash script to validate the data and loaded validated data from local file system to HDFS. Now in the same bash script i wanted to load the data from HDFS to HIVE. How can i do it ? Also how tyhe... (2 Replies)
Discussion started by: shree11
2 Replies

4. Shell Programming and Scripting

Question about writing a bash script

Hello, I want to write a bash script to delete the content after '#'. However, if '#' appears in a string with "", ignore this. For example, input file: test #delete "test #not delete" Output file: test "test #not delete" Does anyone know how to write this script? Thanks (1 Reply)
Discussion started by: jeffwang66
1 Replies

5. Shell Programming and Scripting

Writing a bash script using host

Im trying to write a script using the host command but its not working properly. I cant understand what Im doing wrong. When I use it at the command prompt, it works fine. But its being used actually in the script, it says its not found: 2 SERVFAIL. Can anyone help me? Here's what I have so far: no... (6 Replies)
Discussion started by: relsha
6 Replies

6. Homework & Coursework Questions

brand new user!.. Lost on BASH script writing

Use and complete the template provided. The entire template must be completed. If you don't, your post may be deleted! 1. The problem statement, all variables and given/known data: I have just gotten into writing bash scripts for a class, part of the assignment is to read and be able to tell... (4 Replies)
Discussion started by: Byrang
4 Replies

7. Shell Programming and Scripting

Help with writing simple bash script

I want to write a bash script to: 1. Send an email from localhost to an external gmail account. (gmail then automatically forwards the message back to a pop account on the same server. 2. Script waits 3 minutes then checks to see if the email arrived, and if not, it sends an email to... (9 Replies)
Discussion started by: sallyanne
9 Replies

8. Shell Programming and Scripting

bash, help with stdout manipulation.

Hey all, Im kind of lost on how to do what I want so I figured I would ask. I want to pipe STDOUT of an app to a log file, but I want to prepend each line of that output with the date and time. Im drawing a complete blank on how to do this?? Any ideas? i.e. output is currently this:... (9 Replies)
Discussion started by: trey85stang
9 Replies

9. Shell Programming and Scripting

Writing Bash script

Could anyone help me to Write a script in BASH Shell to determine the percentage of system disk space you are using. (1 Reply)
Discussion started by: boris
1 Replies

10. Shell Programming and Scripting

Problems writing bash script to unzip files

I'm getting the following errors when I try to write a script to unzip some zip files. When I use the free trial copy of the commerical winzip program, however, they work fine. When I use -l or -t on unzip it indicates no errors. When I use the -o switch interactively from the bash command line it... (1 Reply)
Discussion started by: siegfried
1 Replies
Login or Register to Ask a Question