Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: File queue script
Top Forums Shell Programming and Scripting File queue script Post 74255 by GMMike on Wednesday 8th of June 2005 03:26:00 PM
Old 06-08-2005
File queue script
Dear Gurus,
I have an application that sends me thousands of small files.
I need to write a script that sleeps for some time untile enough files have arrived then concatenate (n) files into 1 big file, then send that big file to a program to process it.

For example, if I have the following files
f1, f2, f3, f4, f5

assuming n=3
the script will do noting until f1, f2, f3 arrive, then:

It would do something like this:
cat f1 f2 f3 > bigfile and then executes another program to execute bigfile

then it would wait for the next 3 files to be available or a time period has elapsed (to cover for the case of only x < n files ever arriving) - In this example, it would wait some time after f5 arrives, then does something like this:
cat f4 f5 > bigfile and then executes another program to process bigfile

In short I need to write a script that deals with input files as a FIFO queue. I am sure some one some where must have written a similar one.

Could you point me to some source?

Thanks.
Last edited by GMMike; 06-08-2005 at 04:32 PM..
 

10 More Discussions You Might Find Interesting

1. Programming

Queue

:confused: I have some problem with the technology Queue in Unix Can someone explain me the way queue is made and how it excute ,how it run Thanks in advance (1 Reply)
Discussion started by: iwbasts
1 Replies

2. Shell Programming and Scripting

shell script needed for mail queue notification

Hi shell experts, I would like to have a shell script running in a redhat server for monitoring the mailqueue status. I have already installed the qmHandle and I am using it to get a status of the mail queue in daily basis. I am executing the qmhandle in the cron. Now I am planning to execute... (10 Replies)
Discussion started by: Nightman
10 Replies

3. Shell Programming and Scripting

Need Help Writing Script To Email When Fax Queue Goes Down

I need a script that would email me when the fax queue goes down. How would I go about doing this? Our fax queue drops a few times a week and if I don't catch it fairly quickly it gets backed up really fast which makes the users a little unhappy... I am new to Unix-AIX. We are running... (6 Replies)
Discussion started by: ITAdmin08
6 Replies

4. AIX

AIX 6.1 Error - "A file, file system or message queue is no longer available."

Getting a strange error with a log file below. This is just a normal text log file that is written to by an application on the server. The applicaiton is down, however, I cannot read, remove or touch this file. The file shows on an ls listing but not an ls -al listing. The file system is JFS2... (0 Replies)
Discussion started by: troym72
0 Replies

5. AIX

AIX Bunch of printers queue creation script - HELP

I'd seek for help on how to create a bunch of printers in AIX 6.x or equal or above in one go – say like I have 35 printers to create in 4 different AIX Nodes every month – I currently create it manually like below:- How can I automatic this creation on all the 4-5 Nodes – not actually automatic... (3 Replies)
Discussion started by: shiv2001in
3 Replies

6. Shell Programming and Scripting

Detect if script starts from queue

Dear community, what I'm try to do is deny users to run a script without parameters from command bash, but the same script should run without parameters only from crontab. Example runs by crontab:*/5 * * * * /tmp/script.sh Here the normal execution starts every 5 minutes Example #1 runs by... (16 Replies)
Discussion started by: Lord Spectre
16 Replies

7. AIX

Help with clear printer queue script in AIX 5.3

Good day UNIX forum, could you help me with my clear printer queue script, i have problems with the consistency of this function, sometimes it works sometimes it doesn't. Thanks in advance } preRemovePrintQ(){ clear; echo; echo... (1 Reply)
Discussion started by: beware187
1 Replies

8. Shell Programming and Scripting

How to write Perl Script to Get MQ Queue Depth?

Hi , I got the following script from internet to display queue depth using Perl Script. However, when I execute it , im getting following error. Can anyone shed light on what is going wrong? #!/usr/bin/perl ## 07/23/01 ## Depth Inquiry sample program. ## Arguments: ## Connects to... (11 Replies)
Discussion started by: srkmish
11 Replies

9. Shell Programming and Scripting

Shell Script for Websphere MQ Queue Manager start/stop

Hello All, I am completely new to shell scripting. I had to write a script that starts and stop the queue manager in Websphere MQ. We are on Linux 64-bit patform. The script should stop the queue manager and all the processes related to websphere MQ. It should be a clean stop. Once the queue... (3 Replies)
Discussion started by: pady1224
3 Replies

10. Shell Programming and Scripting

MQ Queue Manager SSL Checking Script

Hello everyone, Newbie to the forum. I have a requirement for a script to check Queue Manager SSL certificates - However I have no previous scripting skills what so ever. I need a script which from high level point of view does the following: #For each Queue Manager on the server go to... (1 Reply)
Discussion started by: MQAdmin
1 Replies

LEARN ABOUT DEBIAN

flow-cat

flow-cat(1)						      General Commands Manual						       flow-cat(1)

NAME

       flow-cat -- Concatenate flow files

SYNOPSIS

       flow-cat  [-aghmp]   [-b  big|little]   [-C  comment]   [-d  debug_level]   [-o	filename]   [-t start_time]  [-T start_time]  [-z z_level]
       [file|directory ...]

DESCRIPTION

       The flow-cat utility processes files and/or directories of files in the flow-tools format.  The resulting concatenated data set is  written
       to the standard output or file specified by -o.	If file is a single dash (`-') or absent, flow-cat will read from the standard input.

OPTIONS

       -a	 Do not ignore filenames that begin with tmp.

       -b big|little
		 Byte order of output.

       -C Comment
		 Add a comment.

       -d debug_level
		 Enable debugging.

       -g	 Sort file list by capture start time before processing.

       -h	 Display help.

       -m	 Disable the use of mmap().

       -p	 Preload headers.  Use to preserve meta information such as lost flows.

       -o file	 Write to file instead of the standard out.

       -t start_time
		 Select flow files up to start_time.  If used with -T select files between start_time and end_time.

       -T end_time
		 Select flow files after end_time.  If used with -t select files between start_time and end_time.

       -z z_level
		 Configure compression level to  z_level.  0 is disabled (no compression), 9 is highest compression.

       file|directory...
		 Process the files and/or directory.

TIME
/DATE parsing
       start_time  and	end_time  parsing  is  implemented with getdate.y, a commonly used function to process free-form time date specifications.
       Example usage borrowed from cvs:
	   1 month ago
	   2 hours ago
	   400000 seconds ago
	   last year
	   last Monday
	   yesterday
	   a fortnight ago
	   3/31/92 10:00:07 PST
	   January 23, 1987 10:05pm
	   22:00 GMT

EXAMPLES

       Concatenate all flow files begining with ft-v05.2001-05.01, use flow-print to display the results.

	   flow-cat ft-v05.2001-05-01.* | flow-print

       Concatenate flow files in /flows/krc4, store store the output in compressed.flows at compression level 9 (best).  The headers are preloaded
       so  various  metadata  such  as	the flow count is correct in the result.  Filenames begining with tmp which are typically in-progress flow
       files from flow-capture are not processed.

	   flow-cat -p -z9 /flows/krc4 > compressed.flows

BUGS

       None known.

AUTHOR

       Mark Fullmer maf@splintered.net

SEE ALSO

       flow-tools(1)

																       flow-cat(1)
All times are GMT -4. The time now is 10:36 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy