Sponsored Content
Top Forums Shell Programming and Scripting Parallel Runs in UNIX/Linix Shell Scripting Post 302975443 by prawinmca on Monday 13th of June 2016 10:16:13 AM
Old 06-13-2016
Parallel Runs in UNIX/Linix Shell Scripting

HI,

I have a file(suppose 1 million Records), i wanted to divide file into chunks(100 small files), these all files should run parllell(e.x like threads) to utilize the process for fast processing.

Can you please provide a solution for the above issue.

Ex:
Code:
 Split -b 2m file.txt file, filea, fileb, filec...

Required to read all the above created chunk files(filea fileb, filec ...) parllell.

Thanks,
Praveen

Last edited by jim mcnamara; 06-13-2016 at 03:13 PM..
 

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

UNIX problem? Unix programm runs windows 2000 CPU over 100%

Okee problems...!! What is happening: Unix server with some programms, workstations are windows 2000, the workstations work good but when you start a programm on the Unix server the CPU of the workstations go to 100% usage resulting that the system gets very slow. The programm well its running so... (2 Replies)
Discussion started by: zerocool
2 Replies

2. Shell Programming and Scripting

difference between AIX shell scripting and Unix shell scripting.

please give the difference between AIX shell scripting and Unix shell scripting. (2 Replies)
Discussion started by: haroonec
2 Replies

3. Shell Programming and Scripting

unix shell scripting

i have a file blah_blah_blah_yyyymmdd.dat . I need to write a shell script to extract this date and compare it with the current date. How do i do this?? Can someone provide me with the code please? (1 Reply)
Discussion started by: swarna bhat
1 Replies

4. Shell Programming and Scripting

Unix Shell Scripting

I 'm new to unix shell scripting can some one guide me to any e-book or link from where i can learn unix shell scripting .. i want to learn create interactive scripts for my day to day solaris work. Any help would be appreciated (1 Reply)
Discussion started by: fugitive
1 Replies

5. Shell Programming and Scripting

$# in unix shell scripting what does it mean

Hi can some one let me know what is the meaning of $# $@ $_ @_ in unix shell scripting. I really appreciate your time for replying to my post. Thanks In Advance (4 Replies)
Discussion started by: einsteinBrain
4 Replies

6. Shell Programming and Scripting

Unix shell scripting

Hi All, I have one file called date1.txt and it contains dates like 130112 140112 150112 160112 170112 180112 190112 201012 so i need a script to read this file line by line and find out the day of each date and assign this value in one variable. And validate Weekday="Mon" then... (4 Replies)
Discussion started by: vichuelaa
4 Replies

7. UNIX for Dummies Questions & Answers

Unix Shell Scripting( Calling from Unix to PLSQL)

Hello Experts, I have the following questions to be discussed here at this esteemed discussion forum. I have two Excel sheets which contain Unix Commands llike creating directory the structure/ftp/Copy/Zip etc to basically create an environment. I need help in understanding some of... (1 Reply)
Discussion started by: faizsaadq
1 Replies

8. UNIX for Dummies Questions & Answers

Unix Shell Scripting

I'm sorry if this doesn't go here, but I'm in depserate need of help with my last unix homework. Anyways, I'm taking summer classes, and one of them is UNIX. I've understood everything thus far, but I'm having a killer time with how my instructor has worded the problems for shell scripting. I... (3 Replies)
Discussion started by: dw15
3 Replies

9. Shell Programming and Scripting

Parallel processing in AIX (bash shell scripting)

Hi all, I am struggling to build a utility which can do the parallel processing. I achieved same in Linux using xargs -P but same is not working on AIX. I am building file copy utility where I will have all required info in a file (like souce file info and target location details), now i need... (2 Replies)
Discussion started by: ankur singh
2 Replies
PT-FIFO-SPLIT(1p)					User Contributed Perl Documentation					 PT-FIFO-SPLIT(1p)

NAME
pt-fifo-split - Split files and pipe lines to a fifo without really splitting. SYNOPSIS
Usage: pt-fifo-split [options] [FILE ...] pt-fifo-split splits FILE and pipes lines to a fifo. With no FILE, or when FILE is -, read standard input. Read hugefile.txt in chunks of a million lines without physically splitting it: pt-fifo-split --lines 1000000 hugefile.txt while [ -e /tmp/pt-fifo-split ]; do cat /tmp/pt-fifo-split; done RISKS
The following section is included to inform users about the potential risks, whether known or unknown, of using this tool. The two main categories of risks are those created by the nature of the tool (e.g. read-only tools vs. read-write tools) and those created by bugs. pt-fifo-split creates and/or deletes the "--fifo" file. Otherwise, no other files are modified, and it merely reads lines from the file given on the command-line. It should be very low-risk. At the time of this release, we know of no bugs that could cause serious harm to users. The authoritative source for updated information is always the online issue tracking system. Issues that affect this tool will be marked as such. You can see a list of such issues at the following URL: http://www.percona.com/bugs/pt-fifo-split <http://www.percona.com/bugs/pt-fifo-split>. See also "BUGS" for more information on filing bugs and getting help. DESCRIPTION
pt-fifo-split lets you read from a file as though it contains only some of the lines in the file. When you read from it again, it contains the next set of lines; when you have gone all the way through it, the file disappears. This works only on Unix-like operating systems. You can specify multiple files on the command line. If you don't specify any, or if you use the special filename "-", lines are read from standard input. OPTIONS
This tool accepts additional command-line arguments. Refer to the "SYNOPSIS" and usage information for details. --config type: Array Read this comma-separated list of config files; if specified, this must be the first option on the command line. --fifo type: string; default: /tmp/pt-fifo-split The name of the fifo from which the lines can be read. --force Remove the fifo if it exists already, then create it again. --help Show help and exit. --lines type: int; default: 1000 The number of lines to read in each chunk. --offset type: int; default: 0 Begin at the Nth line. If the argument is 0, all lines are printed to the fifo. If 1, then beginning at the first line, lines are printed (exactly the same as 0). If 2, the first line is skipped, and the 2nd and subsequent lines are printed to the fifo. --pid type: string Create the given PID file. The file contains the process ID of the script. The PID file is removed when the script exits. Before starting, the script checks if the PID file already exists. If it does not, then the script creates and writes its own PID to it. If it does, then the script checks the following: if the file contains a PID and a process is running with that PID, then the script dies; or, if there is no process running with that PID, then the script overwrites the file with its own PID and starts; else, if the file contains no PID, then the script dies. --statistics Print out statistics between chunks. The statistics are the number of chunks, the number of lines, elapsed time, and lines per second overall and during the last chunk. --version Show version and exit. ENVIRONMENT
The environment variable "PTDEBUG" enables verbose debugging output to STDERR. To enable debugging and capture all output to a file, run the tool like: PTDEBUG=1 pt-fifo-split ... > FILE 2>&1 Be careful: debugging output is voluminous and can generate several megabytes of output. SYSTEM REQUIREMENTS
You need Perl, DBI, DBD::mysql, and some core packages that ought to be installed in any reasonably new version of Perl. BUGS
For a list of known bugs, see http://www.percona.com/bugs/pt-fifo-split <http://www.percona.com/bugs/pt-fifo-split>. Please report bugs at https://bugs.launchpad.net/percona-toolkit <https://bugs.launchpad.net/percona-toolkit>. Include the following information in your bug report: o Complete command-line used to run the tool o Tool "--version" o MySQL version of all servers involved o Output from the tool including STDERR o Input files (log/dump/config files, etc.) If possible, include debugging output by running the tool with "PTDEBUG"; see "ENVIRONMENT". DOWNLOADING
Visit http://www.percona.com/software/percona-toolkit/ <http://www.percona.com/software/percona-toolkit/> to download the latest release of Percona Toolkit. Or, get the latest release from the command line: wget percona.com/get/percona-toolkit.tar.gz wget percona.com/get/percona-toolkit.rpm wget percona.com/get/percona-toolkit.deb You can also get individual tools from the latest release: wget percona.com/get/TOOL Replace "TOOL" with the name of any tool. AUTHORS
Baron Schwartz ABOUT PERCONA TOOLKIT
This tool is part of Percona Toolkit, a collection of advanced command-line tools developed by Percona for MySQL support and consulting. Percona Toolkit was forked from two projects in June, 2011: Maatkit and Aspersa. Those projects were created by Baron Schwartz and developed primarily by him and Daniel Nichter, both of whom are employed by Percona. Visit <http://www.percona.com/software/> for more software developed by Percona. COPYRIGHT, LICENSE, AND WARRANTY This program is copyright 2007-2011 Baron Schwartz, 2011-2012 Percona Inc. Feedback and improvements are welcome. THIS PROGRAM IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 2; OR the Perl Artistic License. On UNIX and similar systems, you can issue `man perlgpl' or `man perlartistic' to read these licenses. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA. VERSION
pt-fifo-split 2.1.2 perl v5.14.2 2012-06-15 PT-FIFO-SPLIT(1p)
All times are GMT -4. The time now is 12:52 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy