Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: How to deal with incomplete image files
Top Forums UNIX for Dummies Questions & Answers How to deal with incomplete image files Post 302535451 by Corona688 on Thursday 30th of June 2011 05:42:11 PM
Old 06-30-2011
Quote:
Originally Posted by CinciJeff
The software uploads images every second or so. I'd be fine with every 5 or 10 seconds.
Hmmm. If you could increase that to 5 or more that'd be good. This should work in BASH or KSH:
Code:
#!/bin/bash

# Minimum age of a file to replace.
# There's a lag/jitter since time's measured in seconds,
# so set an AGE of at most 3 for a file updated every 5 seconds.
AGE=3

# File to watch for updates on
WATCH=./recent

# File to keep up to date
OUT=./complete

# Create temp files of increasing age
for ((N=0; N<AGE; N++)) ; do touch /tmp/$$-$N ; sleep 1 ; done

# Delete temp files on exit no matter what
trap "rm -f /tmp/$$-[0-9]" EXIT

while true
do
        sleep 1

        # Update oldest file
        touch /tmp/$$-$((N%AGE))
        ((N++)) # N is now the next-oldest file
        OLDEST=/tmp/$$-$((N%AGE))

        [ -f "$WATCH" ] || continue

        # File is old enough to be done
        if [ "${WATCH}" -ot "$OLDEST" ]
        then
                # do NOT replace:  Move COMPLETE files into its place.
                # incomplete files will never be seen.

                ln "${WATCH}" "${WATCH}-tmp"
                rm -f "${OUT}" "${WATCH}"
                mv "${WATCH}-tmp" "${OUT}"
        fi
done

Running this and a 'while true ; do sleep 5 ; touch recent ; done' loop, it worked like clockwork, waiting until 'recent' was at least 4 seconds old before replacing complete and deleting recent. Which is cutting it a bit close I suppose -- possible it could jitter one second once in a while, wait 5 seconds, get caught the instant an upload happens and grab an incomplete. (Which wouldn't be the end of the world, since it's not COPYING incomplete data, it's LINKING it: Data will continue to be appended.) AGE=2 would be better for 5 seconds.

complete is kept complete at all times that it exists, so the worst you'd get is a 404. The odds of seeing the 404 are miniscule. And you might even be able to do a workaround with a custom 404 redirect that just sends it to the same image.
Last edited by Corona688; 06-30-2011 at 06:50 PM..
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

append newline to files with incomplete last line

Hi all, Is there any way I can check a file for the linefeed character at the end of the file, and append one only if it is missing (ie. Incomplete last line)? Need to do this because I need to write a script to process files FTP-ed over from various machines, which may or may not be... (1 Reply)
Discussion started by: ziyi
1 Replies

2. Shell Programming and Scripting

Join of files is incomplete?!

Hi folks, I am using the join command to join two files on a common field as follows: File1.txt Adsorption|H01.181.529.047 Adult|M01.060.116 Children|M01.055 File2.txt 5|Adsorption|C0001674 7|Adult|C000001 6|Children|C00002 join -i -t "|" -a 2 -1 1 -2 2 File1.txt File2.txt This... (7 Replies)
Discussion started by: s0460205
7 Replies

3. Shell Programming and Scripting

How to ignore incomplete files

On Solaris & AIX, suppose there is a directory 'dir'. Log files of size approx 1MB are continuously being deposited here by scp command. I have a script that scans this dir every 5 mins and moves away the log files that have been deposited so far. How do I design my script so that I pick up... (6 Replies)
Discussion started by: sentak
6 Replies

4. Solaris

How to ignore incomplete files

On Solaris, suppose there is a directory 'dir'. Log files of size approx 1MB are continuously being deposited here by scp command. I have a script that scans this dir every 5 mins and moves away the log files that have been deposited so far. How do I design my script so that I pick up *only*... (6 Replies)
Discussion started by: sentak
6 Replies

5. Shell Programming and Scripting

How to deal with .tar.gz files in Linux?x

Dear all, there are more than 10 files with .tar.gz extension in my folder i didnt want to extract them i just want to run the query to fetch my necessary data from all the files. but when i run the command it untar the files in that folder. gunzip -c abc.tar.gz | tar -xf - | grep REC |... (0 Replies)
Discussion started by: jojo123
0 Replies

6. Shell Programming and Scripting

How to deal files with space in names?

I have a bash file as following: #!/bin/sh deal_file(){ printf $1 printf "\t is a file" echo } main(){ for file in `find "$1" ` do deal_file $file done } main $1 then I run ./t.sh .,get the following results: (1 Reply)
Discussion started by: 915086731
1 Replies

7. Shell Programming and Scripting

matching image files to create one image

Hi, I have two sets of image files. Both sets have names A to Z but set 1 ends with .cdt.png and set 2 ends with .matrix.png. I want set 1 to match with set 2 if the names match (i.e. A.cdt.png will match with A.matrix.png) and with the convert image tool (program for images), it will merge the... (6 Replies)
Discussion started by: kylle345
6 Replies

8. Shell Programming and Scripting

How to deal with replaced files?

My task is to copy all files from many directories in one. The big problem i encounter is that some files in different directory have the same names. Is they are way to copy the files that have same names in a sub directory ( need to preserve the name of the files unchanged ) I have list with... (6 Replies)
Discussion started by: gogok_bg
6 Replies

9. Shell Programming and Scripting

Exclude incomplete files in ls -rlt

Hi All, I am bit puzzled with this requirement where I need to list the files in a directory. However, files are being continuously written to this folder through FTP. Hence I need to exclude the file which is being written at the time of listing the directory. I thought of using file time... (5 Replies)
Discussion started by: angshuman
5 Replies

10. UNIX for Beginners Questions & Answers

Split Command Generating Incomplete Output Files

Hello All, May i please know how do i ensure my split command would NOT generate incomplete output files like below, the last lines in each file is missing some columns or last line is complete. split -b 50GB File File_ File_aa |551|70210203|xxxxxxx|12/22/2010 20:44:58|11/01/2010... (1 Reply)
Discussion started by: Ariean
1 Replies

LEARN ABOUT REDHAT

age

AGE(4)                                                     BSD Kernel Interfaces Manual                                                     AGE(4)

NAME

     age -- Attansic/Atheros L1 Gigabit Ethernet driver

SYNOPSIS

     To compile this driver into the kernel, place the following lines in your kernel configuration file:

           device miibus
           device age

     Alternatively, to load the driver as a module at boot time, place the following line in loader.conf(5):

           if_age_load="YES"

DESCRIPTION

     The age device driver provides support for Attansic/Atheros L1 PCI Express Gigabit Ethernet controllers.

     All LOMs supported by the age driver have TCP/UDP/IP checksum offload for both transmit and receive, TCP segmentation offload (TSO), hardware
     VLAN tag stripping/insertion features and an interrupt moderation mechanism as well as a 64-bit multicast hash filter.

     The L1 also supports Jumbo Frames (up to 10240 bytes), which can be configured via the interface MTU setting.  Selecting an MTU larger than
     1500 bytes with the ifconfig(8) utility configures the adapter to receive and transmit Jumbo Frames.

     The age driver supports the following media types:

     autoselect   Enable autoselection of the media type and options.  The user can manually override the autoselected mode by adding media
                  options to rc.conf(5).

     10baseT/UTP  Set 10Mbps operation.

     100baseTX    Set 100Mbps (Fast Ethernet) operation.

     1000baseTX   Set 1000baseTX operation over twisted pair.

     The age driver supports the following media options:

     full-duplex  Force full duplex operation.

     half-duplex  Force half duplex operation.

     For more information on configuring this device, see ifconfig(8).

HARDWARE

     The age driver provides support for LOMs based on Attansic/Atheros L1 Gigabit Ethernet controller chips, including:

     o   ASUS M2N8-VMX
     o   ASUS M2V
     o   ASUS M3A
     o   ASUS P2-M2A590G
     o   ASUS P5B-E
     o   ASUS P5B-MX/WIFI-AP
     o   ASUS P5B-VMSE
     o   ASUS P5K
     o   ASUS P5KC
     o   ASUS P5KPL-C
     o   ASUS P5KPL-VM
     o   ASUS P5K-SE
     o   ASUS P5K-V
     o   ASUS P5L-MX
     o   ASUS P5DL2-VM
     o   ASUS P5L-VM 1394
     o   ASUS G2S

LOADER TUNABLES

     Tunables can be set at the loader(8) prompt before booting the kernel or stored in loader.conf(5).

     hw.age.msi_disable
             This tunable disables MSI support on the Ethernet hardware.  The default value is 0.

     hw.age.msix_disable
             This tunable disables MSI-X support on the Ethernet hardware.  The default value is 0.

SYSCTL VARIABLES

     The following variables are available as both sysctl(8) variables and loader(8) tunables:

     dev.age.%d.int_mod
             Maximum amount of time to delay interrupt processing in units of 2us.  The accepted range is 0 to 65000, the default is 50 (100us).
             Value 0 completely disables the interrupt moderation.

     dev.age.%d.process_limit
             Maximum amount of Rx events to be processed in the event loop before rescheduling a taskqueue.  The accepted range is 30 to 255, the
             default value is 128 events.  The interface does not need to be brought down and up again before a change takes effect.

     dev.age.%d.stats
             Display lots of useful MAC counters maintained in the driver.

SEE ALSO

     altq(4), arp(4), miibus(4), netintro(4), ng_ether(4), vlan(4), ifconfig(8)

HISTORY

     The age driver was written by Pyun YongHyeon <yongari@FreeBSD.org>.  It first appeared in FreeBSD 7.1.

BSD                                                             September 18, 2008                                                             BSD
All times are GMT -4. The time now is 06:58 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy