Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Ensure file copy is complete before starting process
Top Forums Shell Programming and Scripting Ensure file copy is complete before starting process Post 302476637 by adi_2_chaos on Thursday 2nd of December 2010 06:45:06 AM
Old 12-02-2010
Ensure file copy is complete before starting process
Hi experts,

I have a requirement wherein a user is uploading a file to the Landing directory on one of our Linux servers. A cron job is scheduled to run after every 5 minutes which will pick up the files from the source (Landing) dir and copy to the target dir, and once successfully copied to the target, should remove that file from the source folder. For ensuring that the files which are stil being uploaded (when the cron script starts) arent deleted, I am first creating a file list, and the copy command should go in a loop, copying all the files which are there in the list to the target. Once the copy command gives a successfuly exit for all the files, I'll capture the byte-count for each file at both source and target dirs, and if the sizes match, I'll delete the source file.

The issue I face is with large files. I tried to upload a file called abc (a binary file) to the server using WinSCP, and I noticed that before the file is completely uploaded to the source dir, it is named as "abc.filepart". So, assuming a situation wherein the file is being uploaded when the cron starts, it will identify the filename as "abc.filepart" and copy the contents to the target dir. But if the upload is complete before the copy command finishes, the file name changes back to "abc". Consequently, the loop which runs to compare the byte-counts, wont be able to locate the file "abc.filepart" in the source dir, and the script fails.

I may also try to ignore the "filepart" extension, but then I'm not sure what all extensions can possibly be suffixed to the files while they are being uploaded.

Can anyone give me any pointer on how to ensure that I can check this particular clause.

Regards,
Sriram
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

capture the process id when starting a background process

Hello all, How do I start a background process and save the process id to a file on my system. For example %wait 5 & will execute and print the process id. I can't figure out how to get it to a file. I've tried: > filename 0>filename 1>filename. Any assistance is most appreciated. Thanks, Jim... (10 Replies)
Discussion started by: jleavitt
10 Replies

2. Filesystems, Disks and Memory

some process writin file - check if complete

Hi folks... some process is writing a file.... as soon as the process starts the file comes there, and its growin.. now i in another script want to ftp the file. i don't know if the file is complete or not. the process which writes the file is some other application and hence can't... (0 Replies)
Discussion started by: sade
0 Replies

3. Shell Programming and Scripting

PERL: wait for process to complete

I'm using PERL on windows NT to try to run an extract of data. I have multiple zip files in multiple locations. I am extracting "*.t" from zip files and subsequently adding that file to one zip file so when the script is complete I should have one zip file with a whole bunch of ".t" files in it. ... (2 Replies)
Discussion started by: dangral
2 Replies

4. UNIX for Dummies Questions & Answers

Ensure FTP is complete before using file

Hi, I have a program that checks a directory for new files. A file may be placed in the directory only via FTP from another system. The files are long, the FTP can take several minutes to complete. my program sences that a file has arrived but can't tell if the FTP process that sent it is... (10 Replies)
Discussion started by: GMMike
10 Replies

5. Shell Programming and Scripting

Ensure the file completion ?

Hi, I would like to copy a file from one location to another. But that particular file is not fully loaded. I like to copy a file once it's fully loaded or complete file. How to ensure whether file is fully loaded or complete file?. -Thambi (5 Replies)
Discussion started by: thambi
5 Replies

6. IP Networking

A test to ensure TCP/IP configuration is complete

Greetings. I have Debian lenny, and at the moment only the bare bones install, no GUI. I'm connecting via wireless to a Linksys router on a home network, and I manually configured a fixed ip etc during the install. I have not attempted to use any higher level apps such as ftp or telnet or apt yet.... (3 Replies)
Discussion started by: fguy
3 Replies

7. Shell Programming and Scripting

[KSH/Bash] Starting a parent process from a child process?

Hey all, I need to launch a script from within 2 other scripts that can run independently of the two parent scripts... Im having a hard time doing this, if anyone knows how please let me know. More detail. ScriptA (bash), ScriptB (ksh), ScriptC (bash) ScriptA, launches ScriptB ScirptB,... (7 Replies)
Discussion started by: trey85stang
7 Replies

8. Shell Programming and Scripting

check if some file is in copy process, then transfer it

my user copy large files, and it's take 10min for file to be copied to the server (/tmp/user/ files/), if in the meantime start my scheduled script, then it will copy a part of some file to server1 my idea is to check the file size twice in a short period (1-2 seconds) of time, then compare, if... (5 Replies)
Discussion started by: waso
5 Replies

9. UNIX for Advanced & Expert Users

How to copy a binary file while the file is being written to by another process

Hello, Can I copy a binary file while the file is being written to by another process? Another process (program) “P1” creates and opens (for writing) binary file “ABC” on local disk. Process P1 continuously write into ABC file every couple of seconds, adding 512-byte blocks of data. ABC file... (1 Reply)
Discussion started by: mbuki
1 Replies

10. Shell Programming and Scripting

Script that waits for process to complete

Hello, I am in need of running an executable provided by a vendor that basically syncs files to a db. This tool can only be run against one folder at a time and it cannot have more than one instance running at a time. However, I need to run this tool against multiple folders. Each run of the... (5 Replies)
Discussion started by: vipertech
5 Replies

LEARN ABOUT OPENDARWIN

queuedefs

queuedefs(4)							   File Formats 						      queuedefs(4)

NAME

       queuedefs - queue description file for at, batch, and cron

SYNOPSIS

       /etc/cron.d/queuedefs

DESCRIPTION

       The queuedefs file describes the characteristics of the queues managed by cron(1M). Each non-comment line in this file describes one queue.
       The format of the lines are as follows:

	      q.[njobj][nicen][nwaitw]

       The fields in this line are:

       q	The name of the queue. a is the default queue for jobs started by at(1); b is the default queue for jobs  started  by  batch  (see
		at(1)); c is the default queue for jobs run from a crontab(1) file.

       njob	The  maximum  number of jobs that can be run simultaneously in that queue; if more than njob jobs are ready to run, only the first
		njob jobs will be run, and the others will be run as jobs that are currently running terminate.  The default value is  100.

       nice	The nice(1) value to give to all jobs in that queue that are not run with a user ID of super-user.  The default value is 2.

       nwait	The number of seconds to wait before rescheduling a job that was deferred because more than njob jobs were running in  that  job's
		queue, or because the system-wide limit of jobs executing has been reached.  The default value is 60.

       Lines beginning with # are comments, and are ignored.

EXAMPLES

       Example 1: A sample file.

       #
       #
       a.4j1n
       b.2j2n90w

       This  file  specifies that the a queue, for at jobs, can have up to 4 jobs running simultaneously; those jobs will be run with a nice value
       of 1.  As no nwait value was given, if a job cannot be run because too many other jobs are running cron will wait 60 seconds before  trying
       again to run it.

       The  b  queue, for batch(1) jobs, can have up to 2 jobs running simultaneously; those jobs will be run with a nice(1) value of 2.  If a job
       cannot be run because too many other jobs are running, cron(1M) will wait 90 seconds before trying again to run it. All	other  queues  can
       have up to 100 jobs running simultaneously; they will be run with a nice value of 2, and if a job cannot be run because too many other jobs
       are running cron will wait 60 seconds before trying again to run it.

FILES

       /etc/cron.d/queuedefs	       queue description file for  at, batch, and  cron.

SEE ALSO

       at(1), crontab(1), nice(1), cron(1M)

SunOS 5.10							    1 Mar 1994							      queuedefs(4)
All times are GMT -4. The time now is 11:54 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy