|
|
Sponsored Content
Top Forums
Shell Programming and Scripting
help to parallelize work on thousands of files
Post 302436382 by vhope07 on Sunday 11th of July 2010 11:32:45 AM
07-11-2010
Code:
#!/bin/bash
for i in {1..10}
do
list[$i]=$(ls -lS | awk "NR>$i" | awk 'NR%10==1'{'print $8'})
done
for i in {1..10}
do
for file in ${list[$i]}; do genrep.pl $file ../test2 ;done &
doneThe only drawback is when i stop the script, i need another script to kill the background processes but thats ok.
---------- Post updated at 01:00 AM ---------- Previous update was at 12:59 AM ----------
LOL! now I have to try SCRUTINIZER's post and compare results!
---------- Post updated at 01:32 AM ---------- Previous update was at 01:00 AM ----------
Thanks All,
Initially scrutinizer's script was slower, but I got rid of the sleep and it runs like a charm. Advantage is that cancelling the script is easy.
Cheers!
- VH
|
|
10 More Discussions You Might Find Interesting
1. UNIX for Advanced & Expert Users
Hey all,
I have a box running SUSE SLES 8 and in the past few months the box will randomly spawn thousands of instances of /USR/SBIN/CRON to the point where the box will lock up entirely. Upwards of 14000 instances! I imagine it's using up all of the available files that can be opened at one... (10 Replies)
Discussion started by: sysera
10 Replies
2. Shell Programming and Scripting
Hi All,
I want to find a specific pattern from approximately 400000 files on solaris platform. Its very heavy for me to grep that pattern to each file individually.
Can anybody suggest me some way to search for specific pattern (alpha numeric) from these forty thousand files. Please note that... (6 Replies)
Discussion started by: aarora_98
6 Replies
3. Shell Programming and Scripting
Im transmiting thousands ftp files to a server, when type the command mput *, an error comes and say. args list to long. set to I. So ihave to transmit them in batch or blocks, but its too sloww. what shoul i do?. i need to do a program, or with a simple command i could solve the problem? (3 Replies)
Discussion started by: alexcol
3 Replies
4. Shell Programming and Scripting
Hi,
Trying to represent a number with thousands separator in AWK:
echo 1 12 123 1234 12345 123456 1234567 | awk --re-interval '{print gensub(/(])(]{3})/,"\\1,\\2","g")}'
1 12 123 1,234 1,2345 1,23456 1,234567
any idea what is wrong here ? (11 Replies)
Discussion started by: ynixon
11 Replies
5. UNIX for Advanced & Expert Users
There is a procedure I do here at work where I have to synchronize file systems. The source file system always has three or four directories of hundreds of thousands of tiny (1k or smaller) or empty files. Whenever my rsync command reaches these directories, I'm waiting for hours for those files... (3 Replies)
Discussion started by: deckard
3 Replies
6. Shell Programming and Scripting
I'm kinda stuck on this one, I have 7 files with 30.000 lines/file like this
050 0.023 0.504336
050 0.024 0.529521
050 0.025 0.538908
050 0.026 0.537035
I want to find the mean line by line of the third column from the files named like this:
Stat-f-1.dat .... Stat-f-7.dat
Stat-s-1.dat... (8 Replies)
Discussion started by: AriasFco
8 Replies
7. Shell Programming and Scripting
Dear all,
I'm a newbie in programming and I would like to know if it is possible to parallelize the script:
for l in {1..1000}
do
cut -f$l quase2 |tr "\n" "," |sed 's/$/\
/g' |sed '/^$/d' >a_$l.t
done
I tried:
for l in {1..1000}
do
cut -f$l quase2 |tr "\n" "," |sed 's/$/\
/g' |sed... (7 Replies)
Discussion started by: valente
7 Replies
8. Shell Programming and Scripting
Hi All,
I want to search for a certain string in thousands of files and these files are distributed over different directories created daily. For that I created a small script in bash but while running it I am getting the below error:
/ms.sh: xrealloc: subst.c:5173: cannot allocate... (17 Replies)
Discussion started by: danish0909
17 Replies
9. Shell Programming and Scripting
Hi to all,
I have thousand of files in a folder with names with format "FILE-YYYY-MM-DD-HHMM" for what I want to send the following AWK command
awk '/Code.*/' FILE-2014*
I'd like to separate all files that have the same date to a folder named with the corresponding date. For example, if I... (7 Replies)
Discussion started by: Ophiuchus
7 Replies
10. Shell Programming and Scripting
Hello,
I have a bunch of jobs (cp, cat or ln -s) on big files (10xGB in size):
# commands_script.sh:
cp file1 path1/newfile1
cp file2 path1/newfile2
cp file3 path1/newfile3
......
cat file11 path2/file21 path1/newfile11
cat file12 path2/file22 path1/newfile12
cat file13 path2/file23... (5 Replies)
Discussion started by: yifangt
5 Replies
LEARN ABOUT LINUX
set_color
set_color(1) fish set_color(1) NAME
set_color - set_color - set the terminal color set_color - set the terminal color Synopsis set_color [-v --version] [-h --help] [-b --background COLOR] [COLOR] Description Change the foreground and/or background color of the terminal. COLOR is one of black, red, green, brown, yellow, blue, magenta, purple, cyan, white and normal. o -b, --background Set the background color o -c, --print-colors Prints a list of all valid color names o -h, --help Display help message and exit o -o, --bold Set bold or extra bright mode o -u, --underline Set underlined mode o -v, --version Display version and exit Calling set_color normal will set the terminal color to whatever is the default color of the terminal. Some terminals use the --bold escape sequence to switch to a brighter color set. On such terminals, set_color white will result in a grey font color, while set_color --bold white will result in a white font color. Not all terminal emulators support all these features. This is not a bug in set_color but a missing feature in the terminal emulator. set_color uses the terminfo database to look up how to change terminal colors on whatever terminal is in use. Some systems have old and incomplete terminfo databases, and may lack color information for terminals that support it. Download and install the latest version of ncurses and recompile fish against it in order to fix this issue. Version 1.23.1 Sun Jan 8 2012 set_color(1)