Sponsored Content
Full Discussion: Looping through input/output
Top Forums UNIX for Beginners Questions & Answers Looping through input/output Post 302979861 by zajtat on Friday 19th of August 2016 11:48:42 PM
Old 08-20-2016
Looping through input/output

Hi,

I've got a directory of about 6000 txt files that look like this:

Code:
a b c d
e f g h
k l m n

I need to execute a command on them to combine them and, in the end, have one big file with all the needed columns taken form all the 6000 files. I've got the "combining" program, but my problem is that once I've combined the first two files that output file should be the input file for adding the third one and so on.
Here is a schematic:

Code:
combining.executable infile1 infile2 > outfile1
combining.executable infile3 outfile1 > outfile2
combining.executable infile4 outfile2 > outfile3
combining.executable infile5 outfile3 > outfile4
etc

I've created a list of all the files than need to be combined (named master-infile) and wrote this loop:

Code:
for i in $(cat master-infile);
do
   for((a=1;a<=6000;i++);
   do
      combining.executable ${i} ${i} > ${a}
   done
done

But the looping variables seem to be all wrong and all sorts of weird combinations of files get combined. I guess I need to get a count of output files? Any ideas?

Any help would be greatly appreciated!

Last edited by vbe; 08-20-2016 at 10:55 AM.. Reason: icode tags => code tags
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Using Output from one command as input to another

This site has been very helpful thus far.. I thank you all in advance for sharing the knowledge. Let me get to it. I am trying to write a very small script to take away from the boredom of doing the same thing over and over. Everynow and again I have to get the hex value of a file using a... (2 Replies)
Discussion started by: BkontheShell718
2 Replies

2. UNIX for Advanced & Expert Users

input/Output settings

How can we view all the input/output settings of unix environment for a session (6 Replies)
Discussion started by: paritoshc
6 Replies

3. Shell Programming and Scripting

input -output file

Hi, I am having an Input file .which is having a list of names. comapring with our database , needs to write the out put in file called output.txt , format should be name--> country--->phone number could you please help me.. thanks in advance (7 Replies)
Discussion started by: hegdeshashi
7 Replies

4. Shell Programming and Scripting

perl: looping through the output of a 'system' command

Hi there could anybody point me in the right direction when it comes to looping through the output of a system command in perl (i.e. df -k) doing a test against each line to see if it matches? for example if i have a df -k output like this and I wanted to grab the lines that matched "sda" or... (3 Replies)
Discussion started by: rethink
3 Replies

5. Shell Programming and Scripting

Dynamic output file generation using a input text file with predefined output format

Hi, I have two files , one file with data file with attributes that need to be sent to another file to generate a predefined format. Example: File.txt AP|{SSHA}VEEg42CNCghUnGhCVg== APVG3|{SSHA}XK|"password" AP3|{SSHA}XK|"This is test" .... etc --------- test.sh has... (1 Reply)
Discussion started by: hudson03051nh
1 Replies

6. Solaris

SVM Solaris 8 Problem. Metastat output looping

Hi friends, I'm newbie to SVM. Just wanna try installed it on one of our server (to do mirroring for disk0 and disk1) but i think im lost until now. :( the steps i've taken is as below:- 1.prtvtoc /dev/rdsk/c1t0d0s2 | fmthard -s - /dev/rdsk/c1t1d0s2 2.metadb -a -c 3 -f c1t0d0s7... (3 Replies)
Discussion started by: kronenose
3 Replies

7. Shell Programming and Scripting

Looping through for user input

Legends, I want to remain in the script until user passes the correct name. I had tried the below code; but it didn't work out. Please help echo "\nPlease enter the source system: \c" while read SYSTEM_NAME do if ]; then echo "\nMaking $SYSTEM_NAME as source system for particular... (5 Replies)
Discussion started by: sdosanjh
5 Replies

8. Shell Programming and Scripting

looping and saving output of each line separately

I have been trying this program for a long time. I am trying to read a file named "odon" line by line; read the first line, send it to do a command saved in a file "perm", once the first line has finished going through the content of the file perm, the result is saved with the number of the line.... (17 Replies)
Discussion started by: iconig
17 Replies

9. UNIX for Dummies Questions & Answers

Looping through the contents of array for output name

Hi all, I am trying to loop through the string contents of an array, to add it during the saving of the output files. I am trying this code to print each column and save it to unique file name, but it doesn't work. Thanks for any help. fnam=(japan usa uk) alldata.dat contained sample data... (1 Reply)
Discussion started by: ida1215
1 Replies

10. Shell Programming and Scripting

Looping over output of 'ls'

Hi, I have some output from 'ls' command and I want to loop over the output in a bash script. What would be a good way to go about it? For example, if the output of the ls command gives me 'prefix1 prefix2 prefix3', how can I set a loop that will iterate over these? many thanks! (5 Replies)
Discussion started by: pc2001
5 Replies
gpgwrap(1)						      General Commands Manual							gpgwrap(1)

NAME
gpgwrap - a small wrapper for gpg SYNOPSIS
gpgwrap -V gpgwrap -P [-v] [-i] [-a] [-p <file>] gpgwrap -F [-v] [-i] [-a] [-c] [-p <file>] [-o <name>] [--] <file> [<file> ... ] gpgwrap [-v] [-i] [-a] [-p <file>] [-o <name>] [--] gpg [gpg options] DESCRIPTION
The GNU Privacy Guard (gpg) supplies the option --passphrase-fd. This instructs gpg to read the passphrase from the given file descriptor. Usually this file descriptor is opened before gpg is executed via execvp(3). Exactly that is what gpgwrap is doing. The passphrase may be passed to gpgwrap in 4 ways: * as file path, whereat the passphrase is stored as plain text in the file * it is piped from another program to the stdin of gpgwrap * through the GPGWRAP_PASSPHRASE environment variable * gpgwrap prompts for it With no precautions the first point undermines the secure infrastructure gpg provides. But in pure batch oriented environments this may be what you want. Otherwise if you are willing to enter passphrases once and don't want them to be stored as plain text in a file gpg-agent is what you are looking for. Another security objection could be the use of the environment variable GPGWRAP_PASSPHRASE which contains the passphrase and may be read by other processes of the same user. OPTIONS
-V, --version Print out version and exit. -P, --print Get the passphrase and print it mangled to stdout. -F, --file Read gpg commands from the given files. If <file> is - it is read from stdin. Exactly one command per line is expected. The given line is handled in the following way: * In the first place the passphrase is mangled. This means that unusual characters are replaced by their backslash escaped octal numbers. * Secondly the mangled passphrase is stored in the environment variable GPGWRAP_PASSPHRASE. * "exec gpgwrap -- " is prepended to each line, before the result is passed as argument to "sh -c". -h, --help Print out usage information. -v, --verbose Increase verbosity level. -i, --interactive Always prompt for passphrase (ignores -p and the environment variable). -a, --ask-twice Ask twice if prompting for a passphrase. -c, --check-exit-code While reading gpg commands from a file, gpgwrap ignores per default the exit code of its child processes. This option enables the check of the exit code. If a child terminates abnormal or with an exit code not equal 0 gpgwrap stops immediately and does return with this exit code. See also section BUGS. -p <file>, --passphrase-file <file> Read passphrase from <file>. If <file> is - it is read from stdin. The passphrase is expected to be in plain text. If this option is not given the passphrase will be taken either from the environment variable GPGWRAP_PASSPHRASE or it will be prompted on the controlling tty if the environment variable is not set. -o <name>, --option-name <name> Specify the name of the "--passphrase-fd" option understood by the program to be executed. This is useful if you want to use gpg- wrap in combination with other programs than gpg. LIMITATIONS
The given passphrase is subject to several limitations depending on the way it was passed to gpgwrap: * There is a size limitation: the passphrase should be not larger than some kilobytes (examine the source code for the exact limit). * gpgwrap allows you to use all characters in a passphrase even 00, but this does not mean that gpg will accept it. gpg may reject your passphrase or may only read a part of it, if it contains characters like 12 (in C also known as ). * If you set the environment variable GPGWRAP_PASSPHRASE you should take special care with the backslash character, because gpgwrap uses backslash to escape octal numbers, (see option -F). Therefore write backslash itself as octal number: 134. EXAMPLES
1. gpgwrap -p /path/to/a/secret/file gpg -c -z 0 --batch --no-tty --cipher-algo blowfish < infile > outfile Read passphrase from /path/to/a/secret/file and execute gpg to do symmetric encryption of infile and write it to outfile. 2. gpgwrap -i -a gpg -c -z 0 --batch --no-tty --cipher-algo blowfish < infile > outfile Same as above except that gpgwrap prompts twice for the passphrase. 3. gpgwrap -F -i - <<EOL gpg --decrypt --batch --no-tty < "$HOME/infile1" > "$HOME/outfile1" gpg --decrypt --batch --no-tty < "$HOME/infile2" > "$HOME/outfile2" gpg --decrypt --batch --no-tty < "$HOME/infile3" > "$HOME/outfile3" gpg --decrypt --batch --no-tty < "$HOME/infile4" > "$HOME/outfile4" EOL gpgwrap prompts for the passphrase and executes four instances of gpg to decrypt the given files. 4. GPGWRAP_PASSPHRASE="mysecretpassphrase" export GPGWRAP_PASSPHRASE gpgwrap -F -c -v /tmp/cmdfile1 - /tmp/cmdfile2 <<EOL gpg --decrypt --batch --no-tty < "$HOME/infile1" > "$HOME/outfile1" gpg --decrypt --batch --no-tty < "$HOME/infile2" > "$HOME/outfile2" gpg --decrypt --batch --no-tty < "$HOME/infile3" > "$HOME/outfile3" gpg --decrypt --batch --no-tty < "$HOME/infile4" > "$HOME/outfile4" EOL Same as above except that gpgwrap gets the passphrase via the environment variable, reads commands additionally from other files and checks the exit code of every gpg instance. This means if one gpg command has a non zero exit code, no further commands are executed. Furthermore gpgwrap produces verbose output. 5. GPGWRAP_PASSPHRASE="$(gpgwrap -P -i -a)" export GPGWRAP_PASSPHRASE find . -maxdepth 1 -type f | while read FILE; do FILE2="$FILE.bz2.gpg" bzip2 -c "$FILE" | gpgwrap gpg -c -z 0 --batch --no-tty --cipher-algo blowfish > "$FILE2" && touch -r "$FILE" "$FILE2" && rm -f "$FILE" done Read in passphrase, compress all files in the current directory, encrypt them and keep date from original file. 6. find . -maxdepth 1 -type f -name '*.bz2.gpg' | awk '{ printf("gpg --decrypt --batch --no-tty --quiet "); printf("--no-secmem-warning < %s ", $0); }' | gpgwrap -F -i -c - | bzip2 -d -c - | grep -i 'data' Decrypt all *.bz2.gpg files in the current directory, decompress them and print out all occurances of data. If you pipe the result to less you get into trouble because gpgwrap and less try to read from the TTY at the same time. In such a case it is better to use the environment variable to give the passphrase (the example above shows how to do this). 7. GPGWRAP_PASSPHRASE="$(gpgwrap -P -i -a)" export GPGWRAP_PASSPHRASE gpgwrap -P | ssh -C -x -P -l user host " GPGWRAP_PASSPHRASE="$(cat)" ... " Prompt for a passphrase twice and write it to the GPGWRAP_PASSPHRASE environment variable. 8. echo -n "Passphrase: " stty -echo read GPGWRAP_PASSPHRASE echo stty echo export GPGWRAP_PASSPHRASE Another way to prompt manually for the passphrase. It was needed in combination with older versions of gpgwrap, because they did not upport -P. Be aware that with this method no automatic conversion to backslash escaped octal numbers takes place. 9. echo "mysecretpassphrase" | gpg --batch --no-tty --passphrase-fd 0 --output outfile --decrypt infile Cheap method to give passphrase to gpg without gpgwrap. Note that you can't use stdin to pass a file to gpg, because stdin is already used for the passphrase. 10. gpg --batch --no-tty --passphrase-fd 3 3< /path/to/a/secret/file < infile > outfile This is a more advanced method to give the passphrase, it is equivalent to Option -p of gpgwrap. This example should at least work with the bash. 11. gpg --batch --no-tty --passphrase-fd 3 3< <(echo "mysecretpassphrase") < infile > outfile Like above, but the passphrase is given directly. This example should at least work with the bash. BUGS
In version 0.02 of gpgwrap the exit code of gpg was only returned if gpgwrap read the passphrase from a file. Since version 0.03, only -F omits exit code checking by default, but it can be enabled with -c. SEE ALSO
gpg, gpg-agent AUTHOR
Karsten Scheibler gpgwrap 0.04 gpgwrap(1)
All times are GMT -4. The time now is 08:45 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy