Sponsored Content
Full Discussion: batching files together
Top Forums Shell Programming and Scripting batching files together Post 86033 by pookie on Monday 10th of October 2005 09:44:43 AM
Old 10-10-2005
batching files together

I'm looking after part of an application which batches up files concatanates them together and then moves the concatanated files to another directory. The problem I have is that the box runs Windows 2003 and UNIX shell emulation is provided by MKS Toolkit, when 10,000 or more files arive in the directory the script I have running to batch the files grinds to a halt when using an ls -l on the directory. Does anyone know how I could write a script that doesn't need to use ls -l or indeed the fastest way to do this in a korn compatible shell ?. Also the files need to be human readable so I'm not sure that cpio could be used.
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

batching files

HI, Suppose I have thousand files in one directory and they can keep increasing ie /home/rooh Then I want to create a batch of 100 files and then compress that batch file and ftp them somewhere else. And morever the files in the batch will be moved to somewhere else. Can you please help... (21 Replies)
Discussion started by: rooh
21 Replies

2. Shell Programming and Scripting

Grep causing long delay (batching) whilst piping

Hi all. I have a problem at work which I have managed to break down into a simple test scenario: I have written a monitoring script that outputs every second the status of various processes, but for now, lets just print the date input.sh: while true do date sleep 1 done This... (9 Replies)
Discussion started by: spudtheimpaler
9 Replies

3. Shell Programming and Scripting

How to retrieve all the linked script files/ctl files/sql files?

Hi I am going to migrate our datawarehouse system from HP Tru 64 Unix to the Red Hat Linux. Inside the box, it is running around 40 cron jobs; inside each cron job, it is calling other shell script files, and the shell script files may again call other shell script files or ctl files(for... (1 Reply)
Discussion started by: franksubramania
1 Replies

4. UNIX for Dummies Questions & Answers

write a program in c in unix that display the files(includ sub-direc and files within) in a sorted

the sorting is based on name of file, file size modification time stamps o f file it should dislay the output in the following format "." and ".." enteries should be ignored please give some idea how to do it (1 Reply)
Discussion started by: pappu kumar jha
1 Replies

5. Shell Programming and Scripting

need a shell script to extract the files from source file and check whether those files existonserve

Hi, I am new to shell scripting.Please help me on this.I am using solaris 10 OS and shell i am using is # echo $0 -sh My requirement is i have source file say makefile.I need to extract files with extensions (.c |.cxx |.h |.hxx |.sc) from the makefile.after doing so i need to check whether... (13 Replies)
Discussion started by: muraliinfy04
13 Replies

6. Shell Programming and Scripting

newbie needs help batching awk, tabitha

Hi guys, I need a little help learning to batch an awk script. All the examples I found on line are too complicated for me. Here's the awk command that I want to run on lots of files. awk 'NR==FNR{a=$0;next$3 in a{print $0 a " " a}' inputfile_A_1.out inputfile_B_1.out > outputfile1.txt ... (11 Replies)
Discussion started by: atjurhs
11 Replies

7. Shell Programming and Scripting

How to create zip/gz/tar files for if the files are older than particular days in UNIX or Linux?

I need a script file for backup (zip or tar or gz) of old log files in our unix server (causing the space problem). Could you please help me to create the zip or gz files for each log files in current directory and sub-directories also? I found one command which is to create gz file for the... (4 Replies)
Discussion started by: Mallikgm
4 Replies

8. Shell Programming and Scripting

Append string to all the files inside a directory excluding subdirectories and .zip files

Hii, Could someone help me to append string to the starting of all the filenames inside a directory but it should exclude .zip files and subdirectories. Eg. file1: test1.log file2: test2.log file3 test.zip After running the script file1: string_test1.log file2: string_test2.log file3:... (4 Replies)
Discussion started by: Ravi Kishore
4 Replies

9. Shell Programming and Scripting

Shell script for field wise record count for different Files .csv files

Hi, Very good wishes to all! Please help to provide the shell script for generating the record counts in filed wise from the .csv file My question: Source file: Field1 Field2 Field3 abc 12f sLm 1234 hjd 12d Hyd 34 Chn My target file should generate the .csv file with the... (14 Replies)
Discussion started by: Kirands
14 Replies

10. UNIX for Beginners Questions & Answers

Automate splitting of files , scp files as each split completes and combine files on target server

i use the split command to split a one terabyte backup file into 10 chunks of 100 GB each. The files are split one after the other. While the files is being split, I will like to scp the files one after the other as soon as the previous one completes, from server A to Server B. Then on server B ,... (2 Replies)
Discussion started by: malaika
2 Replies
BATCHER(8)						    InterNetNews Documentation							BATCHER(8)

NAME
batcher - Article batching for InterNetNews SYNOPSIS
batcher [-rv] [-a articles] [-A total-articles] [-b size] [-B total-size] [-i string] [-N batches] [-p process] [-s separator] host [input] DESCRIPTION
batcher reads a list of files and prepares news batches for the specified host. It is generally used to prepare UUCP feeds, but the resulting batches can be used by any application that uses rnews(8) to inject the articles. It is normally invoked by a script run out of cron that uses shlock to lock the host, followed by ctlinnd to flush the batch file. See send-uucp(8) for a front-end for batcher. batcher reads the file input, or standard input if no file is given. If input is a relative file name, it is assumed to be in pathoutgoing as set in inn.conf. Blank lines and lines starting with a number sign ("#") are ignored. All other lines in the input should consist of one or two fields separated by a single space. The first field is the storage API token of an article. The second field, if present, specifies the size of the article in bytes. By default, batches are written to standard output (which isn't very useful if more than one batch is output), but see the -p option. OPTIONS
-a articles This flag limits the number of articles included in each batch. The default is no limit. A new batch will be started when either the total bytes or the number of articles written exceeds the specified limits. -A total-articles Limits the total number of articles written for all batches. As soon as the total number of articles written to batches reaches or exceeds total-articles, all additional articles in the input will be deferred. The default is no limit. -b size This flag sets the size limit for each batch; as soon as at least this much data has been written out, a new batch will be started. The default size is 60 KB. Using "-b 0" will allow unlimited batch sizes. -B total-size Limits the total number of bytes written for all batches. As soon as the total bytes written to batches reaches or exceeds total-size, all additional articles in the input will be deferred. The default is no limit. -i string A batch starts with an identifying line to specify the unpacking method to be used on the receiving end. When this flag is used, string, followed by a newline, will be output at the start of each batch. The default is to have no initial string (under the assumption that either the processor specified with the -p flag or some other later process will add the appropriate line). -N batches Limits the total number of batches written. As soon as the number of batches written reaches or exceeds batches, all additional articles in the input will be deferred. The default is no limit. -p process By default, batches are written to standard output, which is not useful when more than one output batch is created. If this option is given, each batch will instead be fed via a pipe to the shell command process. The process argument must be an sprintf(3) format string, which may have a single %s parameter that will be replaced with the host name. A common value is: ( echo '#! gunbatch' ; exec gzip -c ) | uux - -r -z %s!rnews which generates gzip-compressed batches and feeds them to uux. -r By default, batcher reports errors to pathlog/errlog. To suppress this redirection and report errors to standard error, use the -r flag. -s separator Each article in a batch starts with a separator line that indicates the size of the article. separator must be an sprintf(3) string, which may have a single %ld in the string that will be replaced with the size of the article. If the separator is not empty, a newline will also be appended to it when it is added to the beginning of each article. The default separator is: #! rnews %ld and this should rarely be changed. -v Upon exit, batcher reports statistics via syslog. With this flag, the statistics will also be printed to standard output. EXIT STATUS
If the input is exhausted and all batches are created successfully, batcher will exit with a zero status. If any of the limits specified with -A, -B, or -N flags are reached, or if there is an error in writing a batch, batcher will try to spool the remaining input by copying it to a file as follows: o If there was no input filename, the remaining input will be copied to pathoutgoing/host. o If an input filename was given, the remaining input will be copied to a temporary file named by appending ".bch" to the end of input (and qualified by adding pathoutgoing if input was not a fully qualified path). If this happens successfully, batcher will then try to rename this temporary file to input (thus replacing input with a copy of itself with all of lines for the successfully batched articles removed). Upon receipt of an interrupt or termination signal, batcher will finish batching the current article, close the batch, and then rewrite the batch file as described above. HISTORY
Written by Rich $alz <rsalz@uunet.uu.net> for InterNetNews. Rewritten by Russ Allbery <rra@stanford.edu> in POD. $Id: batcher.pod 8774 2009-11-15 09:23:27Z iulius $ SEE ALSO
ctlinnd(8), inn.conf(5), newsfeeds(5), rnews(8), send-uucp(8), shlock(1). INN 2.5.2 2009-11-15 BATCHER(8)
All times are GMT -4. The time now is 12:35 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy