Help with script to fully occupied all available cpu


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Help with script to fully occupied all available cpu
# 1  
Old 09-10-2011
Help with script to fully occupied all available cpu

I have long list of file wanna processed by a program:
Code:
data_1.txt
data_2.txt
data_3.txt
data_4.txt
data_5.txt
data_6.txt
data_7.txt
data_8.txt
.
.
data_1_2.txt
data_2_2.txt
 data_3_2.txt
 data_4_2.txt
 data_5_2.txt
  data_6_2.txt
  .

Bash script that I run to auto the progress:
Code:
for f in *.txt
do
./progran_name $f > $f.out
done

My server have 16 cpu. My purpose is plan to write a script that able to utilize all the cpu instead of utilize only 1 cpu to run the progress. After first 16 files is completed, it will automatic run through the next 16 files, etc.
I got try to edit my bash script by adding "&":
Code:
for f in *.txt
do
./progran_name $f > $f.out &
done

Unfortunately, the above script will utilize the cpu based on total number of my *.txt. This is not advice since over cpu utilized will end up stuck my server Smilie
Thanks for any advice to solve my doubts.
# 2  
Old 09-10-2011
Perhaps GNU Parallel would be of some use to you. If you are interested in it (i.e., it seems promising), there is a video (part one of two) about it which you can watch on YouTube.
These 2 Users Gave Thanks to kermit For This Post:
# 3  
Old 09-10-2011
Try:
Code:
for f1 f2 f3 f4 in *.txt; do
  ./prog $f1 > $f1.out &
  ...
  ./prog $f4 > $f4.out &
  wait
done

But you really need GNU Parallel
This User Gave Thanks to yazu For This Post:
# 4  
Old 09-12-2011
Hi yazu,

When I trying your bash script, it shown the below error message:
Code:
line 1: syntax error near unexpected token `f2'

This is the list of my available input file:
Code:
data_1.txt
data_2.txt
data_3.txt
data_4.txt

Script that I edit based on your advice:
Code:
for f1 f2 f3 f4 in *.txt; do
  ./prog $f1 > $f1.out &
  ./prog $f2 > $f2.out &
  ./prog $f3 > $f3.out &
  ./prog $f4 > $f4.out &
  wait
done

Thanks for your advice.
# 5  
Old 09-12-2011
Code:
for f in *.txt; do
 ./prog $f > $f.out & 
 wait
done

--ahamed
This User Gave Thanks to ahamed101 For This Post:
# 6  
Old 09-12-2011
Quote:
Hi yazu,

When I trying your bash script, it shown the below error message:

Code:
line 1: syntax error near unexpected token `f2'
Sorry. I use zsh and I had checked it - it worked. But now I've checked it for bash and ksh - it doesn't.
This User Gave Thanks to yazu For This Post:
# 7  
Old 09-13-2011
Hi yazu,

Is ok about that.
Still thanks for your assist Smilie
You can update me anytime once you have better idea to write a script to utilize all the cpu in server for running a progress.
Thanks first!
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

C Shell Script: While function not fully looping

I am new to scripting and this is probably the 4th or 5th simple script I have written. I am working with a HUGE number of data that need to be organized into folders and named a certain way. I wrote the naming script using a while function to go through the 1000-some folders and rename the files... (0 Replies)
Discussion started by: notluckyhannah
0 Replies

2. Solaris

Can't change fully qualified host name

I tried changing my /etc/inet/hosts file for my server to: <ip address> <hostname> <fqdn> but when I go to reboot the file changes right back to: <ip address> <hostname> how do I get the <fqdn> to stick on a reboot. Thanks (2 Replies)
Discussion started by: jastanle84
2 Replies

3. UNIX for Dummies Questions & Answers

Space occupied by core

Guys, Can some one help me? I need to find the total percent of space occupied by core files on my unix system. I know df -k will give me the percent space utilization for a directory but how do we replicate the space for a file? (3 Replies)
Discussion started by: yabhi_22
3 Replies

4. Shell Programming and Scripting

script to display occupied and non occupied ports in unix

I want to extend this script. This must also be able to show me that a port is already assigned to a service but not running now there must be three different messages port 8949 is open but not listening port 8959 is open port 8999 hasn't been assigned to any service. I know that... (1 Reply)
Discussion started by: charan314
1 Replies

5. Shell Programming and Scripting

how to get fully qualified path name

hi actually i want to get fully qualified path name of the file when the file name is entered as command line argument while running a shell script ex. if i run the shell as $./test.sh ./nsdnet_file.csv the it should display me the full path of the file like /dialp/Release/bin/nsdnet_file.csv... (3 Replies)
Discussion started by: priyanka3006
3 Replies

6. Shell Programming and Scripting

Very Challenging Problem. Please read fully.

Hi, This is the Third thread i'm putting here for the same problem. :( Actually, i'm trying a script like this.. but its taking a long time.. about 3 days to complete fully.. #!/bin/ksh if then exit 1 fi while read i do while read j do field7=`echo $j|cut -d "|"... (12 Replies)
Discussion started by: RRVARMA
12 Replies

7. Solaris

Hostname not fully qualified..

Hi Friends.. I have a small problem with the hostname of my system.I had installed Solaris 10 X86 on Vmware in my windows 2000 system.After booting of my solaris system,if i give check-hostname command it says ,, hostname is not fully qualified ,,change the hostname to hostname.xxx.xxxxxx.com... (3 Replies)
Discussion started by: sdspawankumar
3 Replies

8. Solaris

Command used for checking space occupied by files & sub-direc's inside a mount in %?

Hi, I want to know the command which can be used for finding the % of disk space occupied by files & sub-folders inside a given mount in Sun Solaris For eg: I have /tmp/ folder when I sat df -k it will give the percentage of space used by /tmp/. Say if I want to see how much % the files &... (2 Replies)
Discussion started by: weblogicsupport
2 Replies

9. UNIX for Advanced & Expert Users

so there is no way to know the size occupied by data ?

I want to find out how many disk blocks are used by only data (and not metadata) by a file. But as far as I can tell, if the file has holes, then there is no way to know this. You can find out the logical size of the file (physical size + hole blocks). You can get the physical size of the... (3 Replies)
Discussion started by: the_learner
3 Replies

10. Programming

How to find out the stack size occupied for a process?

Hi, In Linux how to find out what will be the stack size allocated for a process? Actually i have to fork n number of processess, and will call exec. I will be execing an executable which is already multithreaded and each thread size is defined. My doubt is how to know if the size of the... (2 Replies)
Discussion started by: rvan
2 Replies
Login or Register to Ask a Question