Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Making use of multiple cores for running sed and awk scripts
Top Forums Shell Programming and Scripting Making use of multiple cores for running sed and awk scripts Post 302549015 by yazu on Saturday 20th of August 2011 11:56:39 AM
Old 08-20-2011
You can try GNU parallel. This command should help:
Code:
cat wikipedia_dump.xml | parallel --pipe --recstart '<page>' awk '...'

===

No... I'm afraid it will use a fresh counter on each chunk and new files would overwrite older ones... You should split somehow your file on 16 files and then do something like this
Code:
cat filelist | parallel awk '/<page>/{c++} {print > FILENAME "-" c ".dat"}

---------- Post updated at 10:56 PM ---------- Previous update was at 10:04 PM ----------

And there are almost one million articles. It would be hard to work with them as separated files. It better use some xml-streamed parser and put information to database. But this is another story, of course.
Last edited by yazu; 08-20-2011 at 12:15 PM.. Reason: Wrong advise
This User Gave Thanks to yazu For This Post:
shoaibjameel123
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

How to open multiple shells while the scripts keeps running.

Hello, I've tried for a while now to run a bash script that continues to the end, while opening new shells as needed. I've tried xterm -e "somecommand"; & xterm -e " somecommand"; I've also tried screen -S "somecommand"; & screen -S "somecommand"; All without any luck, they... (5 Replies)
Discussion started by: Closed_Socket
5 Replies

2. Shell Programming and Scripting

running multiple scripts

Hi all I have a requirement where I have a flow like Script1 script2 Script3 Script 4 Script 5 Script 6 script7 where script2 to script6 will... (3 Replies)
Discussion started by: nvuradi
3 Replies

3. Shell Programming and Scripting

multiple child scripts running in backgroud, how to use grep on the parent?

Hi I have a shell script A which calls another 10 shell scripts which run in background. How do i make the parent script wait for the child scripts complete, or in other words, i must be able to do a grep of parent script to find out if the child scripts are still running. My Code: ... (5 Replies)
Discussion started by: albertashish
5 Replies

4. Shell Programming and Scripting

Running Multiple scripts based on file size.

Hi, I have created 3 shell scripts which has to run one by one first two shell scripts will create a .txt files...which are used by the third shell script.Now I want to create a master script and run all these in a single script. Please give a pseudo code on how to so the same. ... (4 Replies)
Discussion started by: gaur.deepti
4 Replies

5. Shell Programming and Scripting

making the first character of word using uppercase using awk and sed

I want to make the first character of some words to be uppercase. I have a file like the one below. uid,givenname,sn,cn,mail,telephonenumber mattj,matt,johnson,matt johnson,mattj@gmail.com markv,mark,vennet,matt s vennet,markv@gmail.com mikea,mike,austi,mike austin,mike@gmail.com I want... (3 Replies)
Discussion started by: matt12
3 Replies

6. Shell Programming and Scripting

Help with Shell Scripts Using sed in multiple files.

Hi, I was hoping that someone could help me. I have a problem that i am trying to work on and it requires me to change text within multiple files using sed. I use the program to change an occurance of a word throughout different files that are being tested. At first i had to Create a new script,... (1 Reply)
Discussion started by: Johnny2518
1 Replies

7. UNIX for Dummies Questions & Answers

Execution problem in running multiple scripts

hi all, I have 3 individual scripts to perform the task . 2nd script should run only after the 1st script and 3rd script must run only after first 2 scripts are executed successfully. i want to have a single script that calls all this 3 scripts .this single script should execute the 2nd script... (1 Reply)
Discussion started by: Rahul619
1 Replies

8. Shell Programming and Scripting

Issue with SUNOS running sed scripts

Hi I probably dont have GNU extended sed in my SUNOS . and its creating lot of problems ex: a simple sed command like this is not working sed '/WORD/ a\ sample text line 1 \ sample text line 1 ' filename sed: command garbled: /WORD/ a I took precaution to have a new line after... (11 Replies)
Discussion started by: vash
11 Replies

9. UNIX for Advanced & Expert Users

Running Multiple Scripts for different business date

Hi Team, I have the below 4 scripts which I will be running in sequential order. This run will start for today's business date. If all the 4 scripts are success for today that means script has ran succesfully. Howver if any one of these 4 scripts failed then it has to take the next... (1 Reply)
Discussion started by: Deena1984
1 Replies

10. Shell Programming and Scripting

Running Multiple scripts at a time

Hello! I have a scriptA.ksh and in this script I need to call script1.ksh, script2.ksh, script3.ksh, script4.ksh and script5.ksh. But want to run in two batches like 1st script1.ksh, script2.ksh, script3.ksh, once all 3 are completed then script4.ksh script5.ksh I have given the syntax... (1 Reply)
Discussion started by: karumudi7
1 Replies

LEARN ABOUT MOJAVE

locale::codes::langfam

Locale::Codes::LangFam(3pm)				 Perl Programmers Reference Guide			       Locale::Codes::LangFam(3pm)

NAME

       Locale::Codes::LangFam - standard codes for language extension identification

SYNOPSIS

	  use Locale::Codes::LangFam;

	  $lext = code2langfam('apa');		       # $lext gets 'Apache languages'
	  $code = langfam2code('Apache languages');    # $code gets 'apa'

	  @codes   = all_langfam_codes();
	  @names   = all_langfam_names();

DESCRIPTION

       The "Locale::Codes::LangFam" module provides access to standard codes used for identifying language families, such as those as defined in
       ISO 639-5.

       Most of the routines take an optional additional argument which specifies the code set to use. If not specified, the default ISO 639-5
       language family codes will be used.

SUPPORTED CODE SETS

       There are several different code sets you can use for identifying language families. A code set may be specified using either a name, or a
       constant that is automatically exported by this module.

       For example, the two are equivalent:

	  $lext = code2langfam('apa','alpha');
	  $lext = code2langfam('apa',LOCALE_LANGFAM_ALPHA);

       The codesets currently supported are:

       alpha
	   This is the set of three-letter (lowercase) codes from ISO 639-5 such as 'apa' for Apache languages.

	   This is the default code set.

ROUTINES

       code2langfam ( CODE [,CODESET] )
       langfam2code ( NAME [,CODESET] )
       langfam_code2code ( CODE ,CODESET ,CODESET2 )
       all_langfam_codes ( [CODESET] )
       all_langfam_names ( [CODESET] )
       Locale::Codes::LangFam::rename_langfam  ( CODE ,NEW_NAME [,CODESET] )
       Locale::Codes::LangFam::add_langfam  ( CODE ,NAME [,CODESET] )
       Locale::Codes::LangFam::delete_langfam  ( CODE [,CODESET] )
       Locale::Codes::LangFam::add_langfam_alias  ( NAME ,NEW_NAME )
       Locale::Codes::LangFam::delete_langfam_alias  ( NAME )
       Locale::Codes::LangFam::rename_langfam_code  ( CODE ,NEW_CODE [,CODESET] )
       Locale::Codes::LangFam::add_langfam_code_alias  ( CODE ,NEW_CODE [,CODESET] )
       Locale::Codes::LangFam::delete_langfam_code_alias  ( CODE [,CODESET] )
	   These routines are all documented in the Locale::Codes::API man page.

SEE ALSO

       Locale::Codes
	   The Locale-Codes distribution.

       Locale::Codes::API
	   The list of functions supported by this module.

       http://www.loc.gov/standards/iso639-5/id.php
	   ISO 639-5 .

AUTHOR

       See Locale::Codes for full author history.

       Currently maintained by Sullivan Beck (sbeck@cpan.org).

COPYRIGHT

	  Copyright (c) 2011-2013 Sullivan Beck

       This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

perl v5.18.2							    2013-11-04					       Locale::Codes::LangFam(3pm)
All times are GMT -4. The time now is 02:04 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy