Sponsored Content
Top Forums Shell Programming and Scripting SFTP - accidentally removing unprocessed records Post 302714115 by jeffvansan on Thursday 11th of October 2012 08:17:49 PM
Old 10-11-2012
CPU & Memory SFTP - accidentally removing unprocessed records

I have an automated sftp process running on a Linux server that is securely connecting to an insurance company server whereby the insurance company places work assignments into a directory on their windows server (running VanDyke Vshell).

My unattended (background) process runs every minute. There is private/public key authentication that is working fine.

Its goal is to retrieve any new records in the Assignments directory, and after retrieving them, post them back to a directory called ProcessedAssignments. After sending them back, the process clears the Assignments directory.

This process works perfectly 99.9% of the time. Once in a great while, I accidentally remove a new record that was just placed in the Assignments directory between the time I did a 'get' and a 'put' and the time I do an rm * on the Assignments directory.

I have searched everywhere for a 'best practices' for this type of recurring file transfer with no results.

I'm trying to find a snippet of code somewhere that will allow me to create a list of the records I got/put, and only remove those records from the Assignments directory.

Code to generate script from Basic program here:
Code:
SCRIPT = ""
SCRIPT<-1> = "cd /Assignments" ;* Change external directory
SCRIPT<-1> = "get * ":CUSTOMER.INBOUND.DIRECTORY
SCRIPT<-1> = "cd /ProcessedAssignments":
SCRIPT<-1> = "put ":CUSTOMER.INBOUND.DIRECTORY:"/* ":
SCRIPT<-1> = "cd /Assignments"
SCRIPT<-1> = "rm * "
SCRIPT<-1> = "quit"
SCRIPT<-1> = "eof"


My problem is with the rm * statement.

So, I need to somehow pass a list to my script generator after performing the get, or I need to somehow tag the records I 'got' and only delete those...
Any help would be greatly appreciated.

Last edited by jim mcnamara; 10-11-2012 at 09:24 PM.. Reason: code tags please
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Removing spaces between records

Hi I have an XML file. Which has spaces between different records.... current file( Has many lines, like this... I want to delete all the spaces between > and <, if there are only spaces between them) input file <xyzr> <abc>1234</xyzr> <aaa> <bbb> ayz mnz</bbb> <sen>KEA... (6 Replies)
Discussion started by: thanuman
6 Replies

2. Linux

Need awk script for removing duplicate records

I have huge txt file having millions of trade data. For e.g Trade.txt (first 8 lines in the file is header info) COB_DATE,TRADE_ID,SOURCE_SYSTEM_TRADE_ID,TRADE_GROUP_ID, TRADE_TYPE,DEALER_NAME,EXTERNAL_COUNTERPARTY_ID, EXTERNAL_COUNTERPARTY_NAME,DB_COUNTERPARTY_ID,... (6 Replies)
Discussion started by: nmumbarkar
6 Replies

3. Shell Programming and Scripting

Removing bad records from a text file

Hi, I have an requirement where i need to remove few bad records(bad records I mean email id's are part of the 1st field, where a numeric value expected) from the text file delimited by ",". file1.txt --------- 1234,,DAVID,MAX abc@email.com,,JOHN,SMITH 234,,ROBERT,SEN I need to remove... (3 Replies)
Discussion started by: naveen_sangam
3 Replies

4. Shell Programming and Scripting

Removing duplicate records from 2 files

Can anyone help me to removing duplicate records from 2 separate files in UNIX? Please find the sample records for both the files cat Monday.dat 3FAHP0JA1AR319226MOHMED ATEK 966504453742 SAU2010DE 3LNHL2GC6AR636361HEA DEUK CHOI 821057314531 KOR2010LE 3MEHM0JG7AR652083MUTLAB NAL-NAFISAH... (4 Replies)
Discussion started by: zooby
4 Replies

5. Linux

Need awk script for removing duplicate records

I have log file having Traffic line 2011-05-21 15:11:50.356599 TCP (6), length: 52) 10.10.10.1.3020 > 10.10.10.254.50404: 2011-05-21 15:11:50.652739 TCP (6), length: 52) 10.10.10.254.50404 > 10.10.10.1.3020: 2011-05-21 15:11:50.652558 TCP (6), length: 89) 10.10.10.1.3020 >... (1 Reply)
Discussion started by: Rastamed
1 Replies

6. Shell Programming and Scripting

Removing duplicate records in a file based on single column

Hi, I want to remove duplicate records including the first line based on column1. For example inputfile(filer.txt): ------------- 1,3000,5000 1,4000,6000 2,4000,600 2,5000,700 3,60000,4000 4,7000,7777 5,999,8888 expected output: ---------------- 3,60000,4000 4,7000,7777... (5 Replies)
Discussion started by: G.K.K
5 Replies

7. Shell Programming and Scripting

Removing non matching records

Hi all I have a file with records starting with "Page" has a first column. some of the records have some other junk characters has first column. so pls help me to remove rows which is not having "Page" has a first column. Thanks, Baski (2 Replies)
Discussion started by: baskivs
2 Replies

8. Shell Programming and Scripting

removing duplicate records comparing 2 csv files

Hi All, I want to remove the rows from File1.csv by comparing a column/field in the File2.csv. If both columns matches then I want that row to be deleted from File1 using shell script(awk). Here is an example on what I need. File1.csv: RAJAK,ACTIVE,1 VIJAY,ACTIVE,2 TAHA,ACTIVE,3... (6 Replies)
Discussion started by: rajak.net
6 Replies

9. Shell Programming and Scripting

Removing duplicate records in a file based on single column explanation

I was reading this thread. It looks like a simpler way to say this is to only keep uniq lines based on field or column 1. https://www.unix.com/shell-programming-scripting/165717-removing-duplicate-records-file-based-single-column.html Can someone explain this command please? How are there no... (5 Replies)
Discussion started by: cokedude
5 Replies

10. Shell Programming and Scripting

Removing specific records from files when duplicate key

Hello I have been trying to remove a row from a file which has the same first three columns as another row - I have tried lots of different combinations of suggestion on this forum but can't get it exactly right. what I have is 900 - 1000 = 0 900 - 1000 = 2562 1000 - 1100 = 0 1000 - 1100... (7 Replies)
Discussion started by: tinytimmay
7 Replies
SERVICE(8)						      System Manager's Manual							SERVICE(8)

NAME
service - run a System V init script SYNOPSIS
service SCRIPT COMMAND [OPTIONS] service --status-all service --help | -h | --version DESCRIPTION
service runs a System V init script or systemd unit in as predictable an environment as possible, removing most environment variables and with the current working directory set to /. The SCRIPT parameter specifies a System V init script, located in /etc/init.d/SCRIPT, or the name of a systemd unit. The existence of a systemd unit of the same name as a script in /etc/init.d will cause the unit to take precedence over the init.d script. The supported val- ues of COMMAND depend on the invoked script. service passes COMMAND and OPTIONS to the init script unmodified. For systemd units, start, stop, status, and reload are passed through to their systemctl/initctl equivalents. All scripts should support at least the start and stop commands. As a special case, if COMMAND is --full-restart, the script is run twice, first with the stop command, then with the start command. service --status-all runs all init scripts, in alphabetical order, with the status command. The status is [ + ] for running services, [ - ] for stopped services and [ ? ] for services without a status command. This option only calls status for sysvinit jobs. EXIT CODES
service calls the init script and returns the status returned by it. FILES
/etc/init.d The directory containing System V init scripts. /{lib,run,etc}/systemd/system The directories containing systemd units. ENVIRONMENT
LANG, LANGUAGE, LC_CTYPE, LC_NUMERIC, LC_TIME, LC_COLLATE, LC_MONETARY, LC_MESSAGES, LC_PAPER, LC_NAME, LC_ADDRESS, LC_TELEPHONE, LC_MEA- SUREMENT, LC_IDENTIFICATION, LC_ALL, TERM, PATH The only environment variables passed to the init scripts. SEE ALSO
/etc/init.d/skeleton update-rc.d(8) init(8) invoke-rc.d(8) systemctl(1) AUTHOR
Miloslav Trmac <mitr@redhat.com>, Petter Reinholdtsen <pere@hungry.com> Licence: GNU Public Licence v2 (GPLv2) COPYRIGHT
2006 Red Hat, Inc., Petter Reinholdtsen <pere@hungry.com> Jan 206 SERVICE(8)
All times are GMT -4. The time now is 01:07 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy