Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: How to deal with incomplete image files
Top Forums UNIX for Dummies Questions & Answers How to deal with incomplete image files Post 302535451 by Corona688 on Thursday 30th of June 2011 05:42:11 PM
Old 06-30-2011
Quote:
Originally Posted by CinciJeff
The software uploads images every second or so. I'd be fine with every 5 or 10 seconds.
Hmmm. If you could increase that to 5 or more that'd be good. This should work in BASH or KSH:
Code:
#!/bin/bash

# Minimum age of a file to replace.
# There's a lag/jitter since time's measured in seconds,
# so set an AGE of at most 3 for a file updated every 5 seconds.
AGE=3

# File to watch for updates on
WATCH=./recent

# File to keep up to date
OUT=./complete

# Create temp files of increasing age
for ((N=0; N<AGE; N++)) ; do touch /tmp/$$-$N ; sleep 1 ; done

# Delete temp files on exit no matter what
trap "rm -f /tmp/$$-[0-9]" EXIT

while true
do
        sleep 1

        # Update oldest file
        touch /tmp/$$-$((N%AGE))
        ((N++)) # N is now the next-oldest file
        OLDEST=/tmp/$$-$((N%AGE))

        [ -f "$WATCH" ] || continue

        # File is old enough to be done
        if [ "${WATCH}" -ot "$OLDEST" ]
        then
                # do NOT replace:  Move COMPLETE files into its place.
                # incomplete files will never be seen.

                ln "${WATCH}" "${WATCH}-tmp"
                rm -f "${OUT}" "${WATCH}"
                mv "${WATCH}-tmp" "${OUT}"
        fi
done

Running this and a 'while true ; do sleep 5 ; touch recent ; done' loop, it worked like clockwork, waiting until 'recent' was at least 4 seconds old before replacing complete and deleting recent. Which is cutting it a bit close I suppose -- possible it could jitter one second once in a while, wait 5 seconds, get caught the instant an upload happens and grab an incomplete. (Which wouldn't be the end of the world, since it's not COPYING incomplete data, it's LINKING it: Data will continue to be appended.) AGE=2 would be better for 5 seconds.

complete is kept complete at all times that it exists, so the worst you'd get is a 404. The odds of seeing the 404 are miniscule. And you might even be able to do a workaround with a custom 404 redirect that just sends it to the same image.
Last edited by Corona688; 06-30-2011 at 06:50 PM..
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

append newline to files with incomplete last line

Hi all, Is there any way I can check a file for the linefeed character at the end of the file, and append one only if it is missing (ie. Incomplete last line)? Need to do this because I need to write a script to process files FTP-ed over from various machines, which may or may not be... (1 Reply)
Discussion started by: ziyi
1 Replies

2. Shell Programming and Scripting

Join of files is incomplete?!

Hi folks, I am using the join command to join two files on a common field as follows: File1.txt Adsorption|H01.181.529.047 Adult|M01.060.116 Children|M01.055 File2.txt 5|Adsorption|C0001674 7|Adult|C000001 6|Children|C00002 join -i -t "|" -a 2 -1 1 -2 2 File1.txt File2.txt This... (7 Replies)
Discussion started by: s0460205
7 Replies

3. Shell Programming and Scripting

How to ignore incomplete files

On Solaris & AIX, suppose there is a directory 'dir'. Log files of size approx 1MB are continuously being deposited here by scp command. I have a script that scans this dir every 5 mins and moves away the log files that have been deposited so far. How do I design my script so that I pick up... (6 Replies)
Discussion started by: sentak
6 Replies

4. Solaris

How to ignore incomplete files

On Solaris, suppose there is a directory 'dir'. Log files of size approx 1MB are continuously being deposited here by scp command. I have a script that scans this dir every 5 mins and moves away the log files that have been deposited so far. How do I design my script so that I pick up *only*... (6 Replies)
Discussion started by: sentak
6 Replies

5. Shell Programming and Scripting

How to deal with .tar.gz files in Linux?x

Dear all, there are more than 10 files with .tar.gz extension in my folder i didnt want to extract them i just want to run the query to fetch my necessary data from all the files. but when i run the command it untar the files in that folder. gunzip -c abc.tar.gz | tar -xf - | grep REC |... (0 Replies)
Discussion started by: jojo123
0 Replies

6. Shell Programming and Scripting

How to deal files with space in names?

I have a bash file as following: #!/bin/sh deal_file(){ printf $1 printf "\t is a file" echo } main(){ for file in `find "$1" ` do deal_file $file done } main $1 then I run ./t.sh .,get the following results: (1 Reply)
Discussion started by: 915086731
1 Replies

7. Shell Programming and Scripting

matching image files to create one image

Hi, I have two sets of image files. Both sets have names A to Z but set 1 ends with .cdt.png and set 2 ends with .matrix.png. I want set 1 to match with set 2 if the names match (i.e. A.cdt.png will match with A.matrix.png) and with the convert image tool (program for images), it will merge the... (6 Replies)
Discussion started by: kylle345
6 Replies

8. Shell Programming and Scripting

How to deal with replaced files?

My task is to copy all files from many directories in one. The big problem i encounter is that some files in different directory have the same names. Is they are way to copy the files that have same names in a sub directory ( need to preserve the name of the files unchanged ) I have list with... (6 Replies)
Discussion started by: gogok_bg
6 Replies

9. Shell Programming and Scripting

Exclude incomplete files in ls -rlt

Hi All, I am bit puzzled with this requirement where I need to list the files in a directory. However, files are being continuously written to this folder through FTP. Hence I need to exclude the file which is being written at the time of listing the directory. I thought of using file time... (5 Replies)
Discussion started by: angshuman
5 Replies

10. UNIX for Beginners Questions & Answers

Split Command Generating Incomplete Output Files

Hello All, May i please know how do i ensure my split command would NOT generate incomplete output files like below, the last lines in each file is missing some columns or last line is complete. split -b 50GB File File_ File_aa |551|70210203|xxxxxxx|12/22/2010 20:44:58|11/01/2010... (1 Reply)
Discussion started by: Ariean
1 Replies

LEARN ABOUT REDHAT

pnmindex

pnmindex(1)						      General Commands Manual						       pnmindex(1)

NAME

       pnmindex - build a visual index of a bunch of anymaps

SYNOPSIS

       pnmindex [-size N] [-across N] [-colors N] [-black] [-title T] [-quant|-noquant] pnmfile ...

DESCRIPTION

       pnmindex creates an index image containing thumbnail (small) versions of a bunch of PNM files you supply.

       pnmindex labels each thumbnail and, optionally, contains a title.

OPTIONS

       -size N
	      The  size  of  each  thumbnail.	The  image  is scaled to fit maximally inside a N x N pixel box without changing its aspect ratio.
	      Default is 100.

       -across N
	      The number of thumbnails in each row.  Default is 6.

       -colors N
	      The maximum number of colors allowed in the overall image.  If it would otherwise have more colors than  these,  pnmindex  quantizes
	      the result.  The default is 256.

	      However, this value is meaningless if you specify the -noquant option.

       -black This  controls  the  color  of the padding between the images; normally it's white and the labels are black lettering on white back-
	      ground, but the -black flag reverses this.

       -title title
	      Specifies a title top place at the top of the image.  Default is no title.

       -quant Enables quanization (to the number of colors specified by -colors ).  Quantization is on by default but  you  can  disable  it  with
	      -noquant.

       -noquant
	      See -quant.

SEE ALSO

       pnmscale(1), pnmcat(1), pbmtext(1), ppmquant(1), pnm(5)

AUTHOR

       Copyright (C) 1991 by Jef Poskanzer.  -title and -noquant added 2000 by John Heidemann.

								  9 January 1991						       pnmindex(1)
All times are GMT -4. The time now is 04:13 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy