Sponsored Content
Top Forums Shell Programming and Scripting Multi thread awk command for faster performance Post 302631899 by bartus11 on Saturday 28th of April 2012 09:06:47 AM
Old 04-28-2012
You can split the files that you need to process into a number of batches and run your code concurrently on each of them.
 

9 More Discussions You Might Find Interesting

1. Programming

Multi threading using posix thread library

hi all, can anyone tell me some good site for the mutithreading tutorials, its application, and some code examples. -sushil (2 Replies)
Discussion started by: shushilmore
2 Replies

2. UNIX for Dummies Questions & Answers

Which command will be faster? y?

i)wc -c/etc/passwd|awk'{print $1}' ii)ls -al/etc/passwd|awk'{print $5}' (4 Replies)
Discussion started by: karthi_g
4 Replies

3. Programming

Multi thread data sharing problem in uclinux

hello, I have wrote a multi thread application to run under uclinux. the problem is that threads does not share data. using the ps command it shows a single process for each thread. I test the application under Ubuntu 8.04 and Open Suse 10.3 with 2.6 kernel and there were no problems and also... (8 Replies)
Discussion started by: mrhosseini
8 Replies

4. Shell Programming and Scripting

Multi thread shell programming

I have a unix directory where a million of small text files getting accumulated every week. As of now there is a shell batch program in place which merges all the files in this directory into a single file and ftp to other system. Previously the volume of the files would be around 1 lakh... (2 Replies)
Discussion started by: vk39221
2 Replies

5. Shell Programming and Scripting

Faster way to use this awk command

awk "/May 23, 2012 /,0" /var/tmp/datafile the above command pulls out information in the datafile. the information it pulls is from the date specified to the end of the file. now, how can i make this faster if the datafile is huge? even if it wasn't huge, i feel there's a better/faster way to... (8 Replies)
Discussion started by: SkySmart
8 Replies

6. Shell Programming and Scripting

Making a faster alternative to a slow awk command

Hi, I have a large number of input files with two columns of numbers. For example: 83 1453 99 3255 99 8482 99 7372 83 175 I only wish to retain lines where the numbers fullfil two requirements. E.g: =83 1000<=<=2000 To do this I use the following... (10 Replies)
Discussion started by: s052866
10 Replies

7. Shell Programming and Scripting

How to substract selective values in multi row, multi column file (using awk or sed?)

Hi, I have a problem where I need to make this input: nameRow1a,text1a,text2a,floatValue1a,FloatValue2a,...,floatValue140a nameRow1b,text1b,text2b,floatValue1b,FloatValue2b,...,floatValue140b look like this output: nameRow1a,text1b,text2a,(floatValue1a - floatValue1b),(floatValue2a -... (4 Replies)
Discussion started by: nricardo
4 Replies

8. Shell Programming and Scripting

How to make awk command faster?

I have the below command which is referring a large file and it is taking 3 hours to run. Can something be done to make this command faster. awk -F ',' '{OFS=","}{ if ($13 == "9999") print $1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12 }' ${NLAP_TEMP}/hist1.out|sort -T ${NLAP_TEMP} |uniq>... (13 Replies)
Discussion started by: Peu Mukherjee
13 Replies

9. Shell Programming and Scripting

How to make awk command faster for large amount of data?

I have nginx web server logs with all requests that were made and I'm filtering them by date and time. Each line has the following structure: 127.0.0.1 - xyz.com GET 123.ts HTTP/1.1 (200) 0.000 s 3182 CoreMedia/1.0.0.15F79 (iPhone; U; CPU OS 11_4 like Mac OS X; pt_br) These text files are... (21 Replies)
Discussion started by: brenoasrm
21 Replies
SEND-UUCP(8)						    InterNetNews Documentation						      SEND-UUCP(8)

NAME
send-uucp - Send Usenet articles via UUCP SYNOPSIS
send-uucp [site ...] DESCRIPTION
The send-uucp program processes batch files written by innd(8) to send Usenet articles to UUCP sites. It reads a configuration file to control how it behaves with various sites. Normally, it is run periodically out of cron to put together batches and send them to remote UUCP sites. It makes it possible to reduce bandwidth usage and to send news to remote UUCP sites which cannot receive a real-time feed (for instance if they are over dial-up connections). OPTIONS
Any arguments provided to the program are interpreted as a list of sites specified in send-uucp.cf for which batches should be generated. If no arguments are supplied then batches will be generated for all sites listed in that configuration file. CONFIGURATION
The sites to which articles are to be sent must be configured in the configuration file send-uucp.cf in pathetc as set in inn.conf. Each site is specified with a line of the form: site[:host[:funnel]] [compressor [maxsize [batchtime]]] site The news site name being configured. This must match a site name from newsfeeds(5). host The UUCP host name to which batches should be sent for this site. If omitted, the news site name will be used as the UUCP host name. funnel In the case of a site configured as a funnel, send-uucp needs to flush the channel (or exploder) being used as the target of the funnel instead of flushing the site. This is the way to tell send-uucp the name of the channel or exploder to flush for this site. If not specified, default to flushing the site. compressor The compression method to use for batches. This should be one of "bzip2", "compress", "gzip" or "none". Arguments for the compression command may be specified by using "_" instead of spaces. For example, "gzip_-9". The default value is "gzip". maxsize The maximum size in bytes of a single batch before compression. The default value is 500000 bytes. batchtime A comma separated list of hours during which batches should be generated for a given site. When send-uucp runs, a site will only be processed if the current hour matches one of the hours in batchtime. The default is no limitation on when to generate batches. Fields are separated by spaces and only the site name needs to be specified, with defaults being used for unspecified values. If the first character on a line is a hash sign ("#") then the rest of the line is ignored. EXAMPLE
Here is an example for the send-uucp.cf configuration file: zoetermeer gzip 1048576 5,18,22 hoofddorp gzip 1048576 5,18,22 pa3ebv gzip 1048576 5,18,22 drinkel bzip2 1048576 5,6,18,20,22,0,2 manhole compress 1048576 5,18,22 owl compress 1048576 able pern::MYFUNNEL! This defines eight UUCP sites. The first three and the last two use "gzip" compression, the fourth site ("drinkel") uses "bzip2" and the remaining sites ("manhole" and "owl") use "compress". The first six use a batch size of 1 MB, and the two last sites ("able" and "pern") use the default of 500,000 bytes. The "zoetermeer", "hoofddorp", "pa3ebv", and "manhole" sites will only have batches generated for them during the hours of 05:00, 18:00, and 22:00, and the "drinkel" site will only have batches generated during those hours and 06:00, 20:00, 00:00, and 02:00. There are no restrictions on when batches will be generated for "owl", "able" and "pern". The "pern" site is configured as a funnel into "MYFUNNEL!". send-uucp will issue "ctlinnd flush MYFUNNEL!" instead of "ctlinnd flush pern". As for the newsfeeds file, the usual flags used for a UUCP feed are "Tf,Wnb". Here is a typical entry for "zoetermeer", where the batching is kept between 4 KB and 1 KB: zoetermeer :*,!junk,!control,!control.*/!foo :Tf,Wnb,B4096/1024: FILES
pathbin/send-uucp The Perl script itself used to create news batches from the outgoing files. pathetc/send-uucp.cf The configuration file which specifies a list of sites to be processed. HISTORY
This program was originally written by Edvard Tuinder <ed@elm.net> and then maintained and extended by Miquel van Smoorenburg <miquels@cistron.nl>. Marco d'Itri <md@linux.it> cleaned up the code for inclusion in INN. This manual page was written by Mark Brown <broonie@sirena.org.uk>. SEE ALSO
innd(8), newsfeeds(5), uucp(8). INN 2.5.2 2009-08-16 SEND-UUCP(8)
All times are GMT -4. The time now is 12:44 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy