Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Output Multiple Files in the For Loop

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Output Multiple Files in the For Loop
# 1  
Old 05-13-2016
Output Multiple Files in the For Loop
Code:
while IFS= read -r line
do
# sV for version detection
	nmap -T4 -Pn -v -sS "$line" > "text/$line"
done < <(grep '' $file)

Hi,
where
Code:
line

represents the IP.

I am using NMAP to do scanning.

How can I set to execute that command in the loop several concurrently at a time instead of one at a time.
# 2  
Old 05-13-2016
Quote:
Originally Posted by alvinoo
How can I set to execute that command in the loop several concurrently at a time instead of one at a time.
This is a somewhat dangerous proposition. You should have some limiting factor (a "fanout value") to limit the number of jobs running at the same time.

A sketch for a possible solution could be:

Code:
typeset -i iFanOut=15           # maximum number of jobs, adjust!
typeset -i iRunJobs=0
typeset    chIP=""

while : ; do
                                        # read a new IP from <stdin>
     if [ -n "$chIP" ] ; then
          if ! read chIP ; then
               print -u2 "no more IPs to read, exiting"
               break
          fi
     fi
                                        # adjust number of running jobs
     (( iRunJobs -= $(jobs -n | wc -l) ))
                                        # start new job if slot is free
     if [ $iRunJobs -lt $iFanOut ] ; then
          Command $chIP &
          chIP=""
          (( iRunJobs += 1 ))
     fi
                                        # limit looping
     sleep 1
done < <list_of_IPs>

I hope this helps.

bakunin
Last edited by bakunin; 05-13-2016 at 11:14 AM..
# 3  
Old 05-13-2016
Try to put the commands in background
Code:
while IFS= read -r line
do
# sV for version detection
    nmap -T4 -Pn -v -sS "$line" > "text/$line" &
done < "$file"


Why not simply read from the file?

# 4  
Old 05-17-2016
Quote:
Originally Posted by bakunin
This is a somewhat dangerous proposition. You should have some limiting factor (a "fanout value") to limit the number of jobs running at the same time.

A sketch for a possible solution could be:

Code:
typeset -i iFanOut=15           # maximum number of jobs, adjust!
typeset -i iRunJobs=0
typeset    chIP=""

while : ; do
                                        # read a new IP from <stdin>
     if [ -n "$chIP" ] ; then
          if ! read chIP ; then
               print -u2 "no more IPs to read, exiting"
               break
          fi
     fi
                                        # adjust number of running jobs
     (( iRunJobs -= $(jobs -n | wc -l) ))
                                        # start new job if slot is free
     if [ $iRunJobs -lt $iFanOut ] ; then
          Command $chIP &
          chIP=""
          (( iRunJobs += 1 ))
     fi
                                        # limit looping
     sleep 1
done < <list_of_IPs>

I hope this helps.

bakunin
Does it work? No.. It keeps looping all the time without stopping.
Code:
typeset -i iFanOut=15
typeset -i iRunJobs=0
typeset    chIP=""

while : ; do
                                        # read a new IP from <stdin>
     if [ -n "$chIP" ] ; then
          if ! read chIP ; then
               print -u2 "no more IPs to read, exiting"
          fi
     fi
                                        # adjust number of running jobs
     (( iRunJobs -= $(jobs -n | wc -l) ))
                                        # start new job is slot is free
     if [ $iRunJobs -lt $iFanOut ] ; then
          echo $chIP &
          chIP=""
          (( iRunJobs += 1 ))
     fi
                                        # limit looping
     sleep 1
done < "IPAddressessorted.txt"

---------- Post updated at 09:02 PM ---------- Previous update was at 09:01 PM ----------

Quote:
Originally Posted by MadeInGermany
Try to put the commands in background
Code:
while IFS= read -r line
do
# sV for version detection
    nmap -T4 -Pn -v -sS "$line" > "text/$line" &
done < "$file"


Why not simply read from the file?

I read somewhere it will ignore the last line? So I have spent sometime to figure out it will work.
This solution does not work as well.

Moderator's Comments:
Mod Comment edit by bakunin: CODE-tags, for code, not QUOTE-tags, please
Last edited by bakunin; 05-18-2016 at 03:35 AM..
# 5  
Old 05-18-2016
Quote:
Originally Posted by alvinoo
Does it work? No.. It keeps looping all the time without stopping.
Code:
          if ! read chIP ; then
               print -u2 "no more IPs to read, exiting"
          fi

Yes: first of all, this is why i called it a SKETCH, rather than a SOLUTION in first place. And, comparing the original with your version, what you did was to remove the line break, (see the quoted part above) which is exactly the reason why it doesn't stop looping: you took the stop out!

Another thing: one of the reasons you only get - very generalised - sketches (instead of more specific solutions) here is that you didn't tell us anything about your requirements: OS, used version, used shell, .... But then again, maybe i shouldn't explain that but instead just answer
Quote:
your problem description does not work
and be done with this thread.

bakunin
# 6  
Old 05-18-2016
I am actually using CYGWIN on windows. Bash shell.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Execute multiple files in multiple folders and also output on same folder

How to execute multiple files in multiple folders and also output to be generated in the same folder? Hello Team, I have a path like Sanity_test/*/* and it has around 100+ folders inside with files. I would like to run/execute those files and output of execution to be placed on same /... (1 Reply)
Discussion started by: pushpabuzz
1 Replies

2. Shell Programming and Scripting

SBATCH trinity for multiple files and rename/move the output files

Hey guys, I have wrote the following script to apply a module named "trinity" on my files. (it takes two input files and spit a trinity.fasta as output) #!/bin/bash -l #SBATCH -p node #SBATCH -A <projectID> #SBATCH -n 16 #SBATCH -t 7-00:00:00 #SBATCH --mem=128GB #SBATCH --mail-type=ALL... (1 Reply)
Discussion started by: @man
1 Replies

3. Shell Programming and Scripting

Grep strings on multiple files and output to multiple files

Hi All, I want to use egrep on multiple files and the results should be output to multiple files. I am using the below code in my shell script(working in Ksh shell). However with this code I am not attaining the desired results. #!/bin/ksh ( a="/path/file1" b="path/file2" for file in... (4 Replies)
Discussion started by: am24
4 Replies

4. Shell Programming and Scripting

Create Multiple UNIX Files for Multiple SQL Rows output

Dear All, I am trying to write a Unix Script which fires a sql query. The output of the sql query gives multiple rows. Each row should be saved in a separate Unix File. The number of rows of sql output can be variable. I am able save all the rows in one file but in separate files. Any... (14 Replies)
Discussion started by: Rahul_Bhasin
14 Replies

5. Shell Programming and Scripting

While loop for multiple files

Hi I need to loop until some files are not there .If lck file is there I need to sleep.until its not there I am writing a code like this while && && do echo "Lck file exits" sleep 1 done But this one is not working.Can some one help here please (1 Reply)
Discussion started by: ginrkf
1 Replies

6. Shell Programming and Scripting

awk, multiple files input and multiple files output

Hi! I'm new in awk and I need some help. I have a folder with a lot of files and I need that awk do something in each file and print a new file with the output. The input file name should be modified when I print the outpu files. Thanks in advance for help! :-) ciao (5 Replies)
Discussion started by: gabrysfe
5 Replies

7. UNIX for Dummies Questions & Answers

Writing a loop to manipulate a script and store it in multiple output files

I have a script where the the 9th line looks like this: $filename=sprintf("250.1chr%d.ped", $N); I want to modify this script 1000 times, changing 250.1chr%d.ped to 250.2chr%d.ped, 250.3chr%.ped.......and so on all the way to 250.1000chr%d.ped and store each output in files called ... (4 Replies)
Discussion started by: evelibertine
4 Replies

8. Shell Programming and Scripting

Array output through a for loop problematic with multiple elements.

This code works perfect when using a machine with only one interface online. (Excluding the loopback of course) But when I have other interface up for vmware or a vpn the output gets mixed up. I know I had this working when I was just reading ip's from files so I know it is not a problem with... (8 Replies)
Discussion started by: Azrael
8 Replies

9. UNIX for Dummies Questions & Answers

Using AWK: Extract data from multiple files and output to multiple new files

Hi, I'd like to process multiple files. For example: file1.txt file2.txt file3.txt Each file contains several lines of data. I want to extract a piece of data and output it to a new file. file1.txt ----> newfile1.txt file2.txt ----> newfile2.txt file3.txt ----> newfile3.txt Here is... (3 Replies)
Discussion started by: Liverpaul09
3 Replies

10. Shell Programming and Scripting

Accessing Multiple files using for loop

Hi All, I have some files in my directory, and i want to pull all data using for loop....I am using following code but getting error..! for file in {file1, file2, file3, ..... filen} do L="$(tail -1 $file)";NUM=${L%%|*};DAT=${L##*|} echo $NUM>>filedata.txt done Error: tail:... (3 Replies)
Discussion started by: fidelis
3 Replies
Login or Register to Ask a Question