|
|
Sponsored Content
Top Forums
Shell Programming and Scripting
help to parallelize work on thousands of files
Post 302436382 by vhope07 on Sunday 11th of July 2010 11:32:45 AM
07-11-2010
Code:
#!/bin/bash
for i in {1..10}
do
list[$i]=$(ls -lS | awk "NR>$i" | awk 'NR%10==1'{'print $8'})
done
for i in {1..10}
do
for file in ${list[$i]}; do genrep.pl $file ../test2 ;done &
doneThe only drawback is when i stop the script, i need another script to kill the background processes but thats ok.
---------- Post updated at 01:00 AM ---------- Previous update was at 12:59 AM ----------
LOL! now I have to try SCRUTINIZER's post and compare results!
---------- Post updated at 01:32 AM ---------- Previous update was at 01:00 AM ----------
Thanks All,
Initially scrutinizer's script was slower, but I got rid of the sleep and it runs like a charm. Advantage is that cancelling the script is easy.
Cheers!
- VH
|
|
10 More Discussions You Might Find Interesting
1. UNIX for Advanced & Expert Users
Hey all,
I have a box running SUSE SLES 8 and in the past few months the box will randomly spawn thousands of instances of /USR/SBIN/CRON to the point where the box will lock up entirely. Upwards of 14000 instances! I imagine it's using up all of the available files that can be opened at one... (10 Replies)
Discussion started by: sysera
10 Replies
2. Shell Programming and Scripting
Hi All,
I want to find a specific pattern from approximately 400000 files on solaris platform. Its very heavy for me to grep that pattern to each file individually.
Can anybody suggest me some way to search for specific pattern (alpha numeric) from these forty thousand files. Please note that... (6 Replies)
Discussion started by: aarora_98
6 Replies
3. Shell Programming and Scripting
Im transmiting thousands ftp files to a server, when type the command mput *, an error comes and say. args list to long. set to I. So ihave to transmit them in batch or blocks, but its too sloww. what shoul i do?. i need to do a program, or with a simple command i could solve the problem? (3 Replies)
Discussion started by: alexcol
3 Replies
4. Shell Programming and Scripting
Hi,
Trying to represent a number with thousands separator in AWK:
echo 1 12 123 1234 12345 123456 1234567 | awk --re-interval '{print gensub(/(])(]{3})/,"\\1,\\2","g")}'
1 12 123 1,234 1,2345 1,23456 1,234567
any idea what is wrong here ? (11 Replies)
Discussion started by: ynixon
11 Replies
5. UNIX for Advanced & Expert Users
There is a procedure I do here at work where I have to synchronize file systems. The source file system always has three or four directories of hundreds of thousands of tiny (1k or smaller) or empty files. Whenever my rsync command reaches these directories, I'm waiting for hours for those files... (3 Replies)
Discussion started by: deckard
3 Replies
6. Shell Programming and Scripting
I'm kinda stuck on this one, I have 7 files with 30.000 lines/file like this
050 0.023 0.504336
050 0.024 0.529521
050 0.025 0.538908
050 0.026 0.537035
I want to find the mean line by line of the third column from the files named like this:
Stat-f-1.dat .... Stat-f-7.dat
Stat-s-1.dat... (8 Replies)
Discussion started by: AriasFco
8 Replies
7. Shell Programming and Scripting
Dear all,
I'm a newbie in programming and I would like to know if it is possible to parallelize the script:
for l in {1..1000}
do
cut -f$l quase2 |tr "\n" "," |sed 's/$/\
/g' |sed '/^$/d' >a_$l.t
done
I tried:
for l in {1..1000}
do
cut -f$l quase2 |tr "\n" "," |sed 's/$/\
/g' |sed... (7 Replies)
Discussion started by: valente
7 Replies
8. Shell Programming and Scripting
Hi All,
I want to search for a certain string in thousands of files and these files are distributed over different directories created daily. For that I created a small script in bash but while running it I am getting the below error:
/ms.sh: xrealloc: subst.c:5173: cannot allocate... (17 Replies)
Discussion started by: danish0909
17 Replies
9. Shell Programming and Scripting
Hi to all,
I have thousand of files in a folder with names with format "FILE-YYYY-MM-DD-HHMM" for what I want to send the following AWK command
awk '/Code.*/' FILE-2014*
I'd like to separate all files that have the same date to a folder named with the corresponding date. For example, if I... (7 Replies)
Discussion started by: Ophiuchus
7 Replies
10. Shell Programming and Scripting
Hello,
I have a bunch of jobs (cp, cat or ln -s) on big files (10xGB in size):
# commands_script.sh:
cp file1 path1/newfile1
cp file2 path1/newfile2
cp file3 path1/newfile3
......
cat file11 path2/file21 path1/newfile11
cat file12 path2/file22 path1/newfile12
cat file13 path2/file23... (5 Replies)
Discussion started by: yifangt
5 Replies
LEARN ABOUT CENTOS
grub-render-label
GRUB-RENDER-LABEL(1) User Commands GRUB-RENDER-LABEL(1) NAME
grub-render-label - generate a .disk_label for Apple Macs. SYNOPSIS
grub-render-label [OPTION...] [OPTIONS] DESCRIPTION
Render Apple .disk_label. -b, --bgcolor=COLOR use COLOR for background -c, --color=COLOR use COLOR for text -f, --font=FILE use FILE as font (PF2). -i, --input=FILE read text from FILE. -o, --output=FILE set output filename. Default is STDOUT -t, --text=STRING set the label to render -v, --verbose print verbose messages. -?, --help give this help list --usage give a short usage message -V, --version print program version Mandatory or optional arguments to long options are also mandatory or optional for any corresponding short options. REPORTING BUGS
Report bugs to <bug-grub@gnu.org>. SEE ALSO
The full documentation for grub-render-label is maintained as a Texinfo manual. If the info and grub-render-label programs are properly installed at your site, the command info grub-render-label should give you access to the complete manual. grub-render-label (GRUB) 2.02-2ubuntu8.3 July 2018 GRUB-RENDER-LABEL(1)