Sponsored Content
Top Forums Shell Programming and Scripting help to parallelize work on thousands of files Post 302436382 by vhope07 on Sunday 11th of July 2010 11:32:45 AM
Old 07-11-2010
Code:
#!/bin/bash
for i in {1..10}
do
  list[$i]=$(ls -lS | awk "NR>$i" | awk 'NR%10==1'{'print $8'})
done
for i in {1..10}
do
  for file in ${list[$i]}; do genrep.pl $file ../test2 ;done &
done

The above is working for me now. Thanks
The only drawback is when i stop the script, i need another script to kill the background processes but thats ok.Smilie

---------- Post updated at 01:00 AM ---------- Previous update was at 12:59 AM ----------

LOL! now I have to try SCRUTINIZER's post and compare results!

---------- Post updated at 01:32 AM ---------- Previous update was at 01:00 AM ----------

Thanks All,

Initially scrutinizer's script was slower, but I got rid of the sleep and it runs like a charm. Advantage is that cancelling the script is easy.

Cheers!
- VH
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Multiple (thousands) of Cron Instances

Hey all, I have a box running SUSE SLES 8 and in the past few months the box will randomly spawn thousands of instances of /USR/SBIN/CRON to the point where the box will lock up entirely. Upwards of 14000 instances! I imagine it's using up all of the available files that can be opened at one... (10 Replies)
Discussion started by: sysera
10 Replies

2. Shell Programming and Scripting

Finding a specific pattern from thousands of files ????

Hi All, I want to find a specific pattern from approximately 400000 files on solaris platform. Its very heavy for me to grep that pattern to each file individually. Can anybody suggest me some way to search for specific pattern (alpha numeric) from these forty thousand files. Please note that... (6 Replies)
Discussion started by: aarora_98
6 Replies

3. Shell Programming and Scripting

trnsmiting thousands ftp files and get an error message

Im transmiting thousands ftp files to a server, when type the command mput *, an error comes and say. args list to long. set to I. So ihave to transmit them in batch or blocks, but its too sloww. what shoul i do?. i need to do a program, or with a simple command i could solve the problem? (3 Replies)
Discussion started by: alexcol
3 Replies

4. Shell Programming and Scripting

thousands separator

Hi, Trying to represent a number with thousands separator in AWK: echo 1 12 123 1234 12345 123456 1234567 | awk --re-interval '{print gensub(/(])(]{3})/,"\\1,\\2","g")}' 1 12 123 1,234 1,2345 1,23456 1,234567 any idea what is wrong here ? (11 Replies)
Discussion started by: ynixon
11 Replies

5. UNIX for Advanced & Expert Users

Copying Thousands of Tiny or Empty Files?

There is a procedure I do here at work where I have to synchronize file systems. The source file system always has three or four directories of hundreds of thousands of tiny (1k or smaller) or empty files. Whenever my rsync command reaches these directories, I'm waiting for hours for those files... (3 Replies)
Discussion started by: deckard
3 Replies

6. Shell Programming and Scripting

How to calculate mean in AWK? line by line several files, thousands of lines

I'm kinda stuck on this one, I have 7 files with 30.000 lines/file like this 050 0.023 0.504336 050 0.024 0.529521 050 0.025 0.538908 050 0.026 0.537035 I want to find the mean line by line of the third column from the files named like this: Stat-f-1.dat .... Stat-f-7.dat Stat-s-1.dat... (8 Replies)
Discussion started by: AriasFco
8 Replies

7. Shell Programming and Scripting

Parallelize a task that have for

Dear all, I'm a newbie in programming and I would like to know if it is possible to parallelize the script: for l in {1..1000} do cut -f$l quase2 |tr "\n" "," |sed 's/$/\ /g' |sed '/^$/d' >a_$l.t done I tried: for l in {1..1000} do cut -f$l quase2 |tr "\n" "," |sed 's/$/\ /g' |sed... (7 Replies)
Discussion started by: valente
7 Replies

8. Shell Programming and Scripting

Search for patterns in thousands of files

Hi All, I want to search for a certain string in thousands of files and these files are distributed over different directories created daily. For that I created a small script in bash but while running it I am getting the below error: /ms.sh: xrealloc: subst.c:5173: cannot allocate... (17 Replies)
Discussion started by: danish0909
17 Replies

9. Shell Programming and Scripting

Bash-awk to process thousands of files

Hi to all, I have thousand of files in a folder with names with format "FILE-YYYY-MM-DD-HHMM" for what I want to send the following AWK command awk '/Code.*/' FILE-2014* I'd like to separate all files that have the same date to a folder named with the corresponding date. For example, if I... (7 Replies)
Discussion started by: Ophiuchus
7 Replies

10. Shell Programming and Scripting

Parallelize bash commands/jobs

Hello, I have a bunch of jobs (cp, cat or ln -s) on big files (10xGB in size): # commands_script.sh: cp file1 path1/newfile1 cp file2 path1/newfile2 cp file3 path1/newfile3 ...... cat file11 path2/file21 path1/newfile11 cat file12 path2/file22 path1/newfile12 cat file13 path2/file23... (5 Replies)
Discussion started by: yifangt
5 Replies
JED(1)								   User Manuals 							    JED(1)

NAME
Jed - programmers editor SYNOPSIS
jed --version jed-script --version xjed --version jed [--secure] [--batch|--script|--help] [options] file ... jed-script [--secure] script file [script options] ... xjed [--secure] [X options] [--batch|--script|--help] [options] file ... DESCRIPTION
Jed - programmers editor Features: Color syntax highlighting. Emulation of Emacs, EDT, Wordstar, and Brief editors. Extensible in a language resembling C. Completely cus- tomizable. Editing TeX files with AUC-TeX style editing (BiBTeX support too). Folding support, and much more... For complete documentation, see GNU info files, this manual only provides brief tutorial. OPTIONS
major options --version prints the version and compiletime variables. --help prints usage information. --secure runs Jed in secure mode, e.g. you can't run any external commands with system() or run_shell_cmd(). --batch run Jed in batch mode. This is a non-interactive mode. --script this is a mode like --batch but jed does not eval the startup files. It behaves like slsh. You must give the file that should be evaluated as second argument. It's the same as calling jed-script. minor options -n do not load .jedrc file. -a 'file' load file as user configuration file instead of .jedrc. + 'n' goto line n in buffer (notice that in order to this option to take effect, if must appear before the file name in the command line, like 'jed +3 file') -g 'n' goto line n in buffer (notice that in order to this option to take effect, if must appear after the file name in the command line, like 'jed file -g 3') -l 'file' load file as S-Lang code. -f 'function' execute S-Lang function named function -s 'string' search forward for string -2 split window -i 'file' insert file into current buffer. X options xjed accapts the common options like -display, -name, -fn and -geometry. Additionaly it accepts -facesize SIZE, -fs SIZE if build with XRENDERFONT support, selects the font size SIZE. Use it with the option -fn to select a scalable font. -foreground COLOR, -fg COLOR sets the foreground color. -background COLOR, -bg COLOR sets the background color. -fgMouse COLOR, -mfg COLOR sets the foreground color of the mouse pointer. -bgMouse COLOR, -mbg COLOR sets the background color of the mouse pointer. -Iconic, -ic start iconified. -title NAME sets the window title to NAME. For more options look at xterm.c. CONFIGURATION
Emulating Other Editors JED's ability to create new functions using the S-Lang programming language as well as allowing the user to choose key bindings, makes the emulation of other editors possible. Currently, JED provides reasonable emulation of the Emacs, EDT, and Wordstar editors. Emacs Emulation Emacs Emulation is provided by the S-Lang code in emacs.sl. The basic functionality of Emacs is emulated; most Emacs users should have no problem with JED. To enable Emacs emulation in JED, make sure that the line: () = evalfile ("emacs"); is in your jed.rc (.jedrc) startup file. JED is distributed with this line already present in the default jed.rc file. EDT Emulation For EDT emulation, edt.sl must be loaded. This is accomplished by ensuring that the line: () = evalfile ("edt"); is in present in the jed.rc (.jedrc) Startup File. Wordstar Emulation wordstar.sl contains the S-Lang code for JED's Wordstar emulation. Adding the line () = evalfile ("wordstar"); to your jed.rc (.jedrc) startup file will enable JED's Wordstar emulation. RUN TIME
Status line and Windows JED supports multiple windows. Each window may contain the same buffer or different buffers. A status line is displayed immediately below each window. The status line contains information such as the JED version number, the buffer name, mode, etc. Please beware of the fol- lowing indicators: ** buffer has been modified since last save. %% buffer is read only. m Mark set indicator. This means a region is being defined. d File changed on disk indicator. This indicates that the file associated with the buffer is newer than the buffer itself. s spot pushed indicator. + Undo is enabled for the buffer. [Narrow] Buffer is narrowed to a region of LINES. [Macro] A macro is being defined. Mini-Buffer. The Mini-Buffer consists of a single line located at the bottom of the screen. Much of the dialog between the user and JED takes place in this buffer. For example, when you search for a string, JED will prompt you for the string in the Mini-Buffer. The Mini-Buffer also provides a direct link to the S-Lang interpreter. To access the interpreter, press Ctrl-X Esc and the S-Lang> prompt will appear in the Mini-Buffer. Enter any valid S-Lang expression for evaluation by the interpreter. It is possible to recall data previously entered into the Mini-Buffer by using the up and down arrow keys. This makes it possible to use and edit previous expressions in a convenient and efficient manner. Basic Editing Editing with JED is pretty easy - most keys simply insert themselves. Movement around the buffer is usually done using the arrow keys or page up and page down keys. If edt.sl is loaded, the keypads on VTxxx terminals function as well. Here, only the highlights are touched upon (cut/paste operations are not considered `highlights'). In the following, any character prefixed by the ^ character denotes a Control character. On keyboards without an explicit Escape key, Ctrl-[ will most likely generate and Escape character. A prefix argument to a command may be generated by first hitting the Esc key, then entering the number followed by pressing the desired key. Normally, the prefix argument is used simply for repetition. For example, to move to the right 40 characters, one would press Esc 4 0 followed immediately by the right arrow key. This illustrates the use of the repeat argument for repetition. However, the prefix argu- ment may be used in other ways as well. For example, to begin defining a region, one would press the Ctrl-@ key. This sets the mark and begins highlighting. Pressing the Ctrl-@ key with a prefix argument will abort the act of defining the region and to pop the mark. The following list of useful keybindings assumes that emacs.sl has been loaded. Ctrl-L Redraw screen. Ctrl-_ Undo (Control-underscore, also Ctrl-X u'). Esc q Reformat paragraph (wrap mode). Used with a prefix argument. will justify the paragraph as well. Esc n narrow paragraph (wrap mode). Used with a prefix argument will justify the paragraph as well. Esc ; Make Language comment (Fortran and C) Esc Trim whitespace around point Esc ! Execute shell command Esc $ Ispell word Ctrl-X ? Show line/column information. ` quoted_insert --- insert next char as is (backquote key) Esc s Center line. Esc u Upcase word. Esc d Downcase word. Esc c Capitalize word. Esc x Get M-x minibuffer prompt with command completion Ctrl-X Ctrl-B pop up a list of buffers Ctrl-X Ctrl-C exit JED Ctrl-X 0 Delete Current Window Ctrl-X 1 One Window. Ctrl-X 2 Split Window. Ctrl-X o Other window. Ctrl-X b switch to buffer Ctrl-X k kill buffer Ctrl-X s save some buffers Ctrl-X Esc Get "S-Lang>" prompt for interface to the S-Lang interpreter. Esc . Find tag Ctrl-@ Set Mark (Begin defining a region). Used with a prefix argument aborts the act of defining the region and pops the Mark. FILES
JED_ROOT/lib/*.sl these are the default runtime jed slang files (packages like jed-extra can define additional slang library directories) JED_ROOT/lib/site.sl This is the default startup file. /etc/jed.d/*.sl The system wide configuration files (this is a special Debian feature). ~/.jedrc Per user configuration file, or ~/.jed/jed.rc per user configuration file if the Jed_Home_Directory ~/.jed/ exists. AUTHOR
John E. Davis <davis@space.mit.edu> Jed's Author --- This document was translated to nroff by "Boris D. Beletsky" <borik@isracom.co.il> Debian OCT 1996 JED(1)
All times are GMT -4. The time now is 11:07 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy