Visit Our UNIX and Linux User Community


Building a better mouse trap, or How many lines of code does it take to trap a mouse?


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Building a better mouse trap, or How many lines of code does it take to trap a mouse?
# 1  
Old 04-06-2006
Building a better mouse trap, or How many lines of code does it take to trap a mouse?

Hello all,

I'm hoping to get a little insight from some of the wily veterans amongst you.

I've written a script to check for new outgoing files to our vendors located on our ssl server. It seems to be working ok, but the final question here, will be one of logic, and/or a better way to do it.

First a little background; The program is run every 5 minutes from cron. The files are uploaded via NFS or CIFS. So file dates can't be fully trusted. So, I use find -cmin for the dates. Files remain on the server for 10 days.

Process;
1) Check for PID file. If PID file exist, exit. (program still running) If not generate PID file.

2) Check filesystem size for changes since the last run. If no changes, clean up PID file and exit. (No new files) If changed sleep 1 minute. (File(s) may still be transferring) Loop until changes stop. Add total sleep time to find time. Continue to step 3 (Transfer done)

3) Using the find command. Build a file containing the list of new files in ftp directory newer than specified cmin time.

4) Filter through the file built in step 3. Generate email for each vendor with file names and send to contact for vendor.

5) Clean up PID file. Copy stat files to backups for comparison on the next program run. exit.

Like I said, this is working, but a few files slip through the cracks.

What I would like to know is: If you have any thoghts on better ways to do this.

One Idea I've been looking into is:
Generate a full file list every 5 minutes and use diff to generate the outgoing file list?

Also, This started out as a small server. So, checking for filesystem changes was no problem. Now I have roughly 180 vendors accessing the site. With all the changes to the filesystem size the program will somtimes run for 15 - 20 minutes. Regardless of how the list is built. I would think that once it is generated I could just check file sizes on those files for changes. Once they finish transferring, generate the mail, and wait for the next go-round to pick up additional files.

So what's the general consensus? Thoughts, Ideas, Opinions?

Thanks in Advance,
MPH

I'd rather have a bottle in front of me, than a frontal labotomy.
# 2  
Old 04-06-2006
Not sure that I understand. Is this one directory or a directory tree? How the the files get removed? Anyway...

I would loop through all the files getting name and size (if date cannot be trusted, ignore it). Add name and size to a little database somewhere, timestamping this addition. Or if the entry is present, update size and timestamp. Then loop through database and find entries with old timestamps; process these; remove from database and directory (removal not possible? --- mark as processed in the database.)
# 3  
Old 04-06-2006
Perderabo,

Quote:
Not sure that I understand. Is this one directory or a directory tree? How the the files get removed? Anyway...
This is a directory tree /ftp. Under this there are the users and their incoming and outgoing directories. Each user has their own directory for security reasons. Our customers don't want their data availible to the wrong vendors.
Files get removed by another daily cron job that finds files older than 10 days. The date can't be trusted as far as how many minutes old they are. So, find works fine for removing old the files. If they're transferred via CIFS it holds the creation date previous to the transfer. That's why I use the -cmin. It seems to work well and uses the access time of the transfer. But I think that's where some files fall through. I had to setup ntp on the server due to clock variations between the server and the clients causing problems with file times. Another reason to use the "find all files and diff them" logic.
Quote:
I would loop through all the files getting name and size (if date cannot be trusted, ignore it). Add name and size to a little database somewhere, timestamping this addition. Or if the entry is present, update size and timestamp. Then loop through database and find entries with old timestamps; process these; remove from database and directory (removal not possible? --- mark as processed in the database.)
This is simular to what (I guess) I was trying to say with the idea I was looking into. That is to say, find all the files under /ftp/*/outgoing and diff them for additions against the file list built 5 minutes ago. Using the diffed file names, the "database" would simply be a temp file containing the name and size. Grep for the file, awk the $NF for the size and compair till they're the same, sleeping for bit between checks to avoid frantic looping. When the run is finished delete the temp database. Removed files won't be an issue, since I'm only looking for added files between runs. If the file reapears, there's usually a good reason for it (corrupted IGES files, etc...) and the vendor should be re-notified.

I hope this makes sense. My fingers are too well connected to my brain.
# 4  
Old 04-06-2006
Smilie Hmmmm... I gotta learn to learn to leave these hyper-abstract problems alone.
# 5  
Old 04-06-2006
Smilie I knew I shouldn't have gone to the picasso school of communication Smilie

Previous Thread | Next Thread
Test Your Knowledge in Computers #686
Difficulty: Medium
In reverse Polish notation, the operators follow their operands; for instance, to add 3 and 4, one would write 3 4 + rather than 3 + 4.
True or False?

8 More Discussions You Might Find Interesting

1. Homework & Coursework Questions

VM trap may work differently than a pure install trap.

Use and complete the template provided. The entire template must be completed. If you don't, your post may be deleted! 1. The problem statement, all variables and given/known data: That is the last reply I received from my instructor, and I'm looking for some alternatives. When using... (2 Replies)
Discussion started by: newuser45
2 Replies

2. What is on Your Mind?

Keyboard vs mouse

Which Input device do you use the most ? for me... keyboard ofcourse !! (56 Replies)
Discussion started by: vpraveen84
56 Replies

3. Shell Programming and Scripting

Cntl+z Trap is not detecting ??? Help required to add a trap detection ???

Hi folks, I have tried to add some trap detection in the below script....this script is used to monitor database activities...in a rather awkward way :rolleyes:.... The idea behind adding trap is that....this script creates lots of temporary files in the running folder to store the count... (1 Reply)
Discussion started by: frozensmilz
1 Replies

4. UNIX for Dummies Questions & Answers

Changing middle mouse button for pasting to right mouse button in cygwin rxvt

Hi, I'm using rxvt in Cygwin and I'm wondering how to change my mouse bindings from the middle button for pasting to the right button. The main reason why I want to do this is because my laptop doesn't have a middle mouse button. Thanks for any help! (2 Replies)
Discussion started by: sayeo
2 Replies

5. HP-UX

reset mouse

I have a Belkin KVM switcher, but when I switch from Unix to Windows, then back to Unix, the mouse stops working is there any way to reset the mouse without restarting the workstation? thanks! (0 Replies)
Discussion started by: xRonin
0 Replies

6. UNIX for Advanced & Expert Users

Indigo mouse

A few years ago i found i the Net a "homework" about how to convert ps2 mouse to Indigo mouse. But i do not save it. Anybody helps me? There are about 10 Indigo mice in all of my country. Thanks in advance (2 Replies)
Discussion started by: mikidimov
2 Replies

7. Linux

Mouse Problems

I just got the other computer working again and i put Fedora Core 2 on it but theres a problem. It wont recongnize my mouse at all. Not in installation or in the actual desktop... it doesnt even know its there. It says No - Mouse.... does anyone know how i can fix this? (4 Replies)
Discussion started by: FordGuy
4 Replies

8. UNIX for Dummies Questions & Answers

Mouse wheel...

Hello. I've got round to installing RedHat 7.1, and i have this slight problem. I'm not able to use my mouse wheel. Is this just not possible in RedHat or is there a way to configure my mouse so i can use my mouse wheel. Thank you. (1 Reply)
Discussion started by: Mr-Pixie
1 Replies

Featured Tech Videos