How to deal with incomplete image files


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers How to deal with incomplete image files
# 1  
Old 06-29-2011
How to deal with incomplete image files

Sorry for the odd title, but I couldn't think of an easy way to describe my issue.

Background
I have a home security system that continually sends (via FTP) 4 different still images to a directory on my personal website - cam0.jpg, cam1.jpg, etc. I've written an extremely basic html script to display the images on a single page, and I can access that from anywhere (using a DynDNS address).

Problem
In general the web page works fine, but the problem I have is that one of the images is always in the middle of being overwritten by an updated one. The result is either half an image, a corrupt image or sometimes even an error if I catch the file at just the right spot (maybe as its writing the header or something...)

Question
Is there any way to structure the directory, or some other technique that will "cache" the existing image before it's overwritten? The goal is to have all four images intact each time I enter the web page.

I don't think I have any control over how the security software writes to the FTP site, so maybe I'd need to have some script running that only copies intact files to a secondary directory... I really don't know about this one...

Thanks for any help!

# 2  
Old 06-29-2011
Quote:
Originally Posted by CinciJeff
Is there any way to structure the directory, or some other technique that will "cache" the existing image before it's overwritten?
You could rename the file before you upload, or upload as a different name.
Quote:
I don't think I have any control over how the security software writes to the FTP site, so maybe I'd need to have some script running that only copies intact files to a secondary directory... I really don't know about this one...
But then you have the problem of how to identify intact files. How long does it take to upload one file? Would a lag of double that time before the updated picture becomes visible be acceptable?
# 3  
Old 06-29-2011
The software uploads images every second or so. I'd be fine with every 5 or 10 seconds. I can dig around the software settings and see if I can increase the time between uploads, though I seem to remember that this wasn't an option. If I do increase the lag time, the probability of an incomplete file decreases, but it doesn't guarantee anything.

Regarding renaming the files, I don't have a way to do this though the software.
# 4  
Old 06-29-2011
You have several options to solve your issue.

Here is one suggestion:

1) Change your FTP script as follows:
1.a) Before it starts to copy the images, it deletes a flag file (example: All_Done.txt) if it exists.
1.b) Copy all files to your directory.
1.c) The last step is to create the flag file in the same directory.

2) Write a script/program as follows:
2.a) If it sees the flag file in the directory:
2.a.I) Rename all FTP files to append a suffix to it (example: "cam0_view.jpg", "cam1_view.jpg").
2.a.II) Delete the flag file.

3) Change your html script to display the images from the "XXXX_view.jpg".

The only issue is the split milisecond that will take to rename the files to "view"
where the FTP could be trying to send the new files, otherwise, the above
is a safe procedure.

Hope it helps.
# 5  
Old 06-30-2011
Quote:
Originally Posted by CinciJeff
The software uploads images every second or so. I'd be fine with every 5 or 10 seconds.
Hmmm. If you could increase that to 5 or more that'd be good. This should work in BASH or KSH:
Code:
#!/bin/bash

# Minimum age of a file to replace.
# There's a lag/jitter since time's measured in seconds,
# so set an AGE of at most 3 for a file updated every 5 seconds.
AGE=3

# File to watch for updates on
WATCH=./recent

# File to keep up to date
OUT=./complete

# Create temp files of increasing age
for ((N=0; N<AGE; N++)) ; do touch /tmp/$$-$N ; sleep 1 ; done

# Delete temp files on exit no matter what
trap "rm -f /tmp/$$-[0-9]" EXIT

while true
do
        sleep 1

        # Update oldest file
        touch /tmp/$$-$((N%AGE))
        ((N++)) # N is now the next-oldest file
        OLDEST=/tmp/$$-$((N%AGE))

        [ -f "$WATCH" ] || continue

        # File is old enough to be done
        if [ "${WATCH}" -ot "$OLDEST" ]
        then
                # do NOT replace:  Move COMPLETE files into its place.
                # incomplete files will never be seen.

                ln "${WATCH}" "${WATCH}-tmp"
                rm -f "${OUT}" "${WATCH}"
                mv "${WATCH}-tmp" "${OUT}"
        fi
done

Running this and a 'while true ; do sleep 5 ; touch recent ; done' loop, it worked like clockwork, waiting until 'recent' was at least 4 seconds old before replacing complete and deleting recent. Which is cutting it a bit close I suppose -- possible it could jitter one second once in a while, wait 5 seconds, get caught the instant an upload happens and grab an incomplete. (Which wouldn't be the end of the world, since it's not COPYING incomplete data, it's LINKING it: Data will continue to be appended.) AGE=2 would be better for 5 seconds.

complete is kept complete at all times that it exists, so the worst you'd get is a 404. The odds of seeing the 404 are miniscule. And you might even be able to do a workaround with a custom 404 redirect that just sends it to the same image.

Last edited by Corona688; 06-30-2011 at 06:50 PM..
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Split Command Generating Incomplete Output Files

Hello All, May i please know how do i ensure my split command would NOT generate incomplete output files like below, the last lines in each file is missing some columns or last line is complete. split -b 50GB File File_ File_aa |551|70210203|xxxxxxx|12/22/2010 20:44:58|11/01/2010... (1 Reply)
Discussion started by: Ariean
1 Replies

2. Shell Programming and Scripting

Exclude incomplete files in ls -rlt

Hi All, I am bit puzzled with this requirement where I need to list the files in a directory. However, files are being continuously written to this folder through FTP. Hence I need to exclude the file which is being written at the time of listing the directory. I thought of using file time... (5 Replies)
Discussion started by: angshuman
5 Replies

3. Shell Programming and Scripting

How to deal with replaced files?

My task is to copy all files from many directories in one. The big problem i encounter is that some files in different directory have the same names. Is they are way to copy the files that have same names in a sub directory ( need to preserve the name of the files unchanged ) I have list with... (6 Replies)
Discussion started by: gogok_bg
6 Replies

4. Shell Programming and Scripting

matching image files to create one image

Hi, I have two sets of image files. Both sets have names A to Z but set 1 ends with .cdt.png and set 2 ends with .matrix.png. I want set 1 to match with set 2 if the names match (i.e. A.cdt.png will match with A.matrix.png) and with the convert image tool (program for images), it will merge the... (6 Replies)
Discussion started by: kylle345
6 Replies

5. Shell Programming and Scripting

How to deal files with space in names?

I have a bash file as following: #!/bin/sh deal_file(){ printf $1 printf "\t is a file" echo } main(){ for file in `find "$1" ` do deal_file $file done } main $1 then I run ./t.sh .,get the following results: (1 Reply)
Discussion started by: 915086731
1 Replies

6. Shell Programming and Scripting

How to deal with .tar.gz files in Linux?x

Dear all, there are more than 10 files with .tar.gz extension in my folder i didnt want to extract them i just want to run the query to fetch my necessary data from all the files. but when i run the command it untar the files in that folder. gunzip -c abc.tar.gz | tar -xf - | grep REC |... (0 Replies)
Discussion started by: jojo123
0 Replies

7. Solaris

How to ignore incomplete files

On Solaris, suppose there is a directory 'dir'. Log files of size approx 1MB are continuously being deposited here by scp command. I have a script that scans this dir every 5 mins and moves away the log files that have been deposited so far. How do I design my script so that I pick up *only*... (6 Replies)
Discussion started by: sentak
6 Replies

8. Shell Programming and Scripting

How to ignore incomplete files

On Solaris & AIX, suppose there is a directory 'dir'. Log files of size approx 1MB are continuously being deposited here by scp command. I have a script that scans this dir every 5 mins and moves away the log files that have been deposited so far. How do I design my script so that I pick up... (6 Replies)
Discussion started by: sentak
6 Replies

9. Shell Programming and Scripting

Join of files is incomplete?!

Hi folks, I am using the join command to join two files on a common field as follows: File1.txt Adsorption|H01.181.529.047 Adult|M01.060.116 Children|M01.055 File2.txt 5|Adsorption|C0001674 7|Adult|C000001 6|Children|C00002 join -i -t "|" -a 2 -1 1 -2 2 File1.txt File2.txt This... (7 Replies)
Discussion started by: s0460205
7 Replies

10. UNIX for Dummies Questions & Answers

append newline to files with incomplete last line

Hi all, Is there any way I can check a file for the linefeed character at the end of the file, and append one only if it is missing (ie. Incomplete last line)? Need to do this because I need to write a script to process files FTP-ed over from various machines, which may or may not be... (1 Reply)
Discussion started by: ziyi
1 Replies
Login or Register to Ask a Question