I am pulling my hair out here. I have a file that comes in, once it finishes i want to move it to a new location. This sounds all very easy but my solution is failing and moves the file before it has finished generating.
i test this on the command line it works fine. while the file is generating the result is greater than 0 . But i just ran this and it grabbed the file before it finished generating. Can someone explain what i am doing wrong and or give me a simple solution. I'm sure there maybe an easier way (maybe using fuser) but it drives me insane when everything tells me it should work and it fails to. I am self taught on UNIX so im still learning, any help would be aprreciated
Is the file being generated byyour account - i.e., the process runs under your username.
If this is Linux, you can use the /proc filesystem to see if there exists a file descriptor open for that file. When the descriptor goes away, the file is done generating.
You can also use inotify, lsof, or fuser to see if that process has the file open. Depends on the OS.
Please identify the OS and shell you are using. Then we can give you exact help.
This User Gave Thanks to jim mcnamara For This Post:
First you need to precisely define "done". Then how do you handle error conditions?
And you need to take into account the only entity that knows for certain that the file is "done" - and correct - is whatever is writing it.
Even checking if a process has the file open isn't going to tell you the file is "done" with certainty - that ignores error conditions, such as when a network connection fails.
Thanks for the replies, Jim the system I am using is a Solaris. With very strict restrictions on plugins. So I think fuser maybe the way to go? Could you suggest away to use that?
Also achenle thanks for your input. Good points made. Will have to consider the error handling when I get past this little snagget.
---------- Post updated at 10:39 PM ---------- Previous update was at 10:30 PM ----------
Just had another thought. Could I use cksum in some way?
cksum file grab it using awk '{print $1}' then checking it again and comparing the cksums? Not sure on how to do that but maybe an approach?
The easiest way to know that a file is complete is to also write a marker file, so if you are sending it with ftp or similar, you would do the following on the sending:-
On the receiving server, you write you script to look for $file.OK and then you know you have the file.
Of course, you should put an error check after you think you have sent the data file before writing the flag file else the receiving server will just grab it.
If you want to use a checksum, you could use this as content to the flag file, assuming that the sender and the receiver are using the same check-sum algorithm, i.e. the output from sum is very different to cksum and it depends what each side has.
So, on the sender, something like:-
And on the receiver:-
Of course you could use the m5sum functions to make this neater if you wish with:-
...and...
Does this logic help?
You didn't reply to jim's first question. "Is the file being generated byyour account - i.e., the process runs under your username.?"
If yes, you need to use the correct way of implementation. i.e. Only move the complete file to the final directory. It means, use some temp location while it being written, when finished, move to final location.
Thanks, great replies. Alas the file is sent from a 3rd party and all I do with it is wait till its complete and send it onto someone else. This is a manual task so I thought I would automate it. I am surprised at the problems I ran into. In my head I am thinking easy. Got to love unix and how you can do 1 thing several ways.
is the file there
has it finished generating
send the fiile.
why cant I think of an elegant way to do that.
Thanks guys for taking the time to help a self taught noob.
Last edited by rbatte1; 03-17-2015 at 12:23 PM..
Reason: Converted to LIST=1 tags from plain text
Hi All
Below code is working as expected but creating too many child processes when the url is not up and every minute that process is sending false email alerts
any help with the logic not to generate child process and not to send duplicate alerts
app="https://url"
appresult=$(wget... (2 Replies)
i have a unix script which generates the csv file. the data in csv file is dynamic. how can i convert/move the data from csv file to xml. please suggest (1 Reply)
Hi Guys,
I have a simple request. I have a file in w3c format. Each file has 2 header lines. Rest of the lines are 16 columns each. They are separated by Tab. I need to discard the first 2 lines and then write each column of the txt file into a seperate column of CSV. I tried the command below... (1 Reply)
Hi Frndz,
I have req in which i need to run the query stored in some file and then store the resultset of the query in some other file.
PFB the code i have written:
#----Start job log---
SQLLOG='/opt/app/vertica2/cdr_dev/logs/conn_orac_db.log'
ORA_USER='abc'
ORA_PASSWD='xyz'
... (7 Replies)
HI,
I want to append some configuration statements to the existing file in unix to it's 3 line , 7th line and 28 line.
There is a file generated by our system job, to this I need to add some lines as mentioned below.
DIR=/usr/CDR - line 3
Source=/usr/src - line 7
Target=/usr/tgt - line... (2 Replies)
Hi Gurus,
I have a requirement as below
I have text file a.txt which contains
hi hello
process update
status
Output for file Ok to Proceed no issues good
data arrangement
My requirement here is i need to read the file and check for the words
"OK to Proceed" and if it is available... (14 Replies)
Hello Gurus,
We are facing some performance issue in UNIX. If someone had faced such kind of issue in past please provide your suggestions on this .
Problem Definition:
/Few of load processes of our Finance Application are facing issue in UNIX when they uses a shell script having below... (19 Replies)
Hi,
Our requirement is we have to create file from one file where all the rows from source file is converted into columns of the target file.
For example
Source file is :
Status Report ABC
Generated: 2009-05-05 08:49:47
Job start time=2009-05-05 08:43:43
Job end time=2009-05-05... (1 Reply)
I am trying to find for the pattern in first 5 bytes of every line in a text file and then generate a new file with the pattern found.
Example: expected pattern is '-' to be serached in first 5 bytes of file only.
Input File
ab-cd jan-09
ddddd jan09
cc-ww jan09
dsgdq jan-09
... (2 Replies)
Forgive the daft requests - I'm still a learner :D
I need a script so that I can test another script (I'm confused already) The script I am looking for should generate a new file in the same directory (called newfile1 or what ever) and also generate text within the new file (Hello world? What... (1 Reply)