Sponsored Content
Top Forums Shell Programming and Scripting Parallel Runs in UNIX/Linix Shell Scripting Post 302975465 by Don Cragun on Monday 13th of June 2016 03:55:09 PM
Old 06-13-2016
I do not understand what you're trying to do.

Are you saying that you want 100 copies of split running at the same to split a single file into 100 chunks? That is not the way split works! And, the split utility takes at most two file operands; not an unlimited number as indicated by the filec... in your example split command (and the comma in filea, will be used as the base part of the names of your output files (the first four of which will be named filea,aa, filea,ab, filea,ac, and filea,ad) assuming you remove the remaining file operands which would cause split
to fail with a syntax error.

You say your input file has about 1 million records. Are the records fixed length or variable length? Are you absolutely positive that files of 2,097,152 bytes will contain complete records (i.e., that the start of a record won't be in one of your split files and the end of that record won't be at the start of the next split file)?

After you have created a bunch of files (all but the last of which contain exactly 2,097,152 bytes), what do you want to do with these files? You say you want to run them. Are each of these 2,097,152 byte files a shell script???

Does your system have 100 CPUs? If not, trying to run 100 jobs in parallel will slow down processing; not provide faster processing!

Are you going to distribute the output files from split onto different disk drives? Do you have 100 different disk drives? Will running these 100 files in parallel overwhelm your I/O subsystem?

What operating system are you using?

What shell are you using?
 

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

UNIX problem? Unix programm runs windows 2000 CPU over 100%

Okee problems...!! What is happening: Unix server with some programms, workstations are windows 2000, the workstations work good but when you start a programm on the Unix server the CPU of the workstations go to 100% usage resulting that the system gets very slow. The programm well its running so... (2 Replies)
Discussion started by: zerocool
2 Replies

2. Shell Programming and Scripting

difference between AIX shell scripting and Unix shell scripting.

please give the difference between AIX shell scripting and Unix shell scripting. (2 Replies)
Discussion started by: haroonec
2 Replies

3. Shell Programming and Scripting

unix shell scripting

i have a file blah_blah_blah_yyyymmdd.dat . I need to write a shell script to extract this date and compare it with the current date. How do i do this?? Can someone provide me with the code please? (1 Reply)
Discussion started by: swarna bhat
1 Replies

4. Shell Programming and Scripting

Unix Shell Scripting

I 'm new to unix shell scripting can some one guide me to any e-book or link from where i can learn unix shell scripting .. i want to learn create interactive scripts for my day to day solaris work. Any help would be appreciated (1 Reply)
Discussion started by: fugitive
1 Replies

5. Shell Programming and Scripting

$# in unix shell scripting what does it mean

Hi can some one let me know what is the meaning of $# $@ $_ @_ in unix shell scripting. I really appreciate your time for replying to my post. Thanks In Advance (4 Replies)
Discussion started by: einsteinBrain
4 Replies

6. Shell Programming and Scripting

Unix shell scripting

Hi All, I have one file called date1.txt and it contains dates like 130112 140112 150112 160112 170112 180112 190112 201012 so i need a script to read this file line by line and find out the day of each date and assign this value in one variable. And validate Weekday="Mon" then... (4 Replies)
Discussion started by: vichuelaa
4 Replies

7. UNIX for Dummies Questions & Answers

Unix Shell Scripting( Calling from Unix to PLSQL)

Hello Experts, I have the following questions to be discussed here at this esteemed discussion forum. I have two Excel sheets which contain Unix Commands llike creating directory the structure/ftp/Copy/Zip etc to basically create an environment. I need help in understanding some of... (1 Reply)
Discussion started by: faizsaadq
1 Replies

8. UNIX for Dummies Questions & Answers

Unix Shell Scripting

I'm sorry if this doesn't go here, but I'm in depserate need of help with my last unix homework. Anyways, I'm taking summer classes, and one of them is UNIX. I've understood everything thus far, but I'm having a killer time with how my instructor has worded the problems for shell scripting. I... (3 Replies)
Discussion started by: dw15
3 Replies

9. Shell Programming and Scripting

Parallel processing in AIX (bash shell scripting)

Hi all, I am struggling to build a utility which can do the parallel processing. I achieved same in Linux using xargs -P but same is not working on AIX. I am building file copy utility where I will have all required info in a file (like souce file info and target location details), now i need... (2 Replies)
Discussion started by: ankur singh
2 Replies
SPLIT(1)						    BSD General Commands Manual 						  SPLIT(1)

NAME
split -- split a file into pieces SYNOPSIS
split [-a suffix_length] [-b byte_count[k|m] | -l line_count -n chunk_count] [file [name]] DESCRIPTION
The split utility reads the given file and breaks it up into files of 1000 lines each. If file is a single dash or absent, split reads from the standard input. file itself is not altered. The options are as follows: -a Use suffix_length letters to form the suffix of the file name. -b Create smaller files byte_count bytes in length. If 'k' is appended to the number, the file is split into byte_count kilobyte pieces. If 'm' is appended to the number, the file is split into byte_count megabyte pieces. -l Create smaller files line_count lines in length. -n Split file into chunk_count smaller files. If additional arguments are specified, the first is used as the name of the input file which is to be split. If a second additional argument is specified, it is used as a prefix for the names of the files into which the file is split. In this case, each file into which the file is split is named by the prefix followed by a lexically ordered suffix using suffix_length characters in the range ``a-z''. If -a is not speci- fied, two letters are used as the suffix. If the name argument is not specified, 'x' is used. STANDARDS
The split utility conforms to IEEE Std 1003.1-2001 (``POSIX.1''). HISTORY
A split command appeared in Version 6 AT&T UNIX. The -a option was introduced in NetBSD 2.0. Before that, if name was not specified, split would vary the first letter of the filename to increase the number of possible output files. The -a option makes this unnecessary. BSD
May 28, 2007 BSD
All times are GMT -4. The time now is 08:23 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy