Sponsored Content
Top Forums Shell Programming and Scripting SFTP - accidentally removing unprocessed records Post 302714115 by jeffvansan on Thursday 11th of October 2012 08:17:49 PM
Old 10-11-2012
CPU & Memory SFTP - accidentally removing unprocessed records

I have an automated sftp process running on a Linux server that is securely connecting to an insurance company server whereby the insurance company places work assignments into a directory on their windows server (running VanDyke Vshell).

My unattended (background) process runs every minute. There is private/public key authentication that is working fine.

Its goal is to retrieve any new records in the Assignments directory, and after retrieving them, post them back to a directory called ProcessedAssignments. After sending them back, the process clears the Assignments directory.

This process works perfectly 99.9% of the time. Once in a great while, I accidentally remove a new record that was just placed in the Assignments directory between the time I did a 'get' and a 'put' and the time I do an rm * on the Assignments directory.

I have searched everywhere for a 'best practices' for this type of recurring file transfer with no results.

I'm trying to find a snippet of code somewhere that will allow me to create a list of the records I got/put, and only remove those records from the Assignments directory.

Code to generate script from Basic program here:
Code:
SCRIPT = ""
SCRIPT<-1> = "cd /Assignments" ;* Change external directory
SCRIPT<-1> = "get * ":CUSTOMER.INBOUND.DIRECTORY
SCRIPT<-1> = "cd /ProcessedAssignments":
SCRIPT<-1> = "put ":CUSTOMER.INBOUND.DIRECTORY:"/* ":
SCRIPT<-1> = "cd /Assignments"
SCRIPT<-1> = "rm * "
SCRIPT<-1> = "quit"
SCRIPT<-1> = "eof"


My problem is with the rm * statement.

So, I need to somehow pass a list to my script generator after performing the get, or I need to somehow tag the records I 'got' and only delete those...
Any help would be greatly appreciated.

Last edited by jim mcnamara; 10-11-2012 at 09:24 PM.. Reason: code tags please
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Removing spaces between records

Hi I have an XML file. Which has spaces between different records.... current file( Has many lines, like this... I want to delete all the spaces between > and <, if there are only spaces between them) input file <xyzr> <abc>1234</xyzr> <aaa> <bbb> ayz mnz</bbb> <sen>KEA... (6 Replies)
Discussion started by: thanuman
6 Replies

2. Linux

Need awk script for removing duplicate records

I have huge txt file having millions of trade data. For e.g Trade.txt (first 8 lines in the file is header info) COB_DATE,TRADE_ID,SOURCE_SYSTEM_TRADE_ID,TRADE_GROUP_ID, TRADE_TYPE,DEALER_NAME,EXTERNAL_COUNTERPARTY_ID, EXTERNAL_COUNTERPARTY_NAME,DB_COUNTERPARTY_ID,... (6 Replies)
Discussion started by: nmumbarkar
6 Replies

3. Shell Programming and Scripting

Removing bad records from a text file

Hi, I have an requirement where i need to remove few bad records(bad records I mean email id's are part of the 1st field, where a numeric value expected) from the text file delimited by ",". file1.txt --------- 1234,,DAVID,MAX abc@email.com,,JOHN,SMITH 234,,ROBERT,SEN I need to remove... (3 Replies)
Discussion started by: naveen_sangam
3 Replies

4. Shell Programming and Scripting

Removing duplicate records from 2 files

Can anyone help me to removing duplicate records from 2 separate files in UNIX? Please find the sample records for both the files cat Monday.dat 3FAHP0JA1AR319226MOHMED ATEK 966504453742 SAU2010DE 3LNHL2GC6AR636361HEA DEUK CHOI 821057314531 KOR2010LE 3MEHM0JG7AR652083MUTLAB NAL-NAFISAH... (4 Replies)
Discussion started by: zooby
4 Replies

5. Linux

Need awk script for removing duplicate records

I have log file having Traffic line 2011-05-21 15:11:50.356599 TCP (6), length: 52) 10.10.10.1.3020 > 10.10.10.254.50404: 2011-05-21 15:11:50.652739 TCP (6), length: 52) 10.10.10.254.50404 > 10.10.10.1.3020: 2011-05-21 15:11:50.652558 TCP (6), length: 89) 10.10.10.1.3020 >... (1 Reply)
Discussion started by: Rastamed
1 Replies

6. Shell Programming and Scripting

Removing duplicate records in a file based on single column

Hi, I want to remove duplicate records including the first line based on column1. For example inputfile(filer.txt): ------------- 1,3000,5000 1,4000,6000 2,4000,600 2,5000,700 3,60000,4000 4,7000,7777 5,999,8888 expected output: ---------------- 3,60000,4000 4,7000,7777... (5 Replies)
Discussion started by: G.K.K
5 Replies

7. Shell Programming and Scripting

Removing non matching records

Hi all I have a file with records starting with "Page" has a first column. some of the records have some other junk characters has first column. so pls help me to remove rows which is not having "Page" has a first column. Thanks, Baski (2 Replies)
Discussion started by: baskivs
2 Replies

8. Shell Programming and Scripting

removing duplicate records comparing 2 csv files

Hi All, I want to remove the rows from File1.csv by comparing a column/field in the File2.csv. If both columns matches then I want that row to be deleted from File1 using shell script(awk). Here is an example on what I need. File1.csv: RAJAK,ACTIVE,1 VIJAY,ACTIVE,2 TAHA,ACTIVE,3... (6 Replies)
Discussion started by: rajak.net
6 Replies

9. Shell Programming and Scripting

Removing duplicate records in a file based on single column explanation

I was reading this thread. It looks like a simpler way to say this is to only keep uniq lines based on field or column 1. https://www.unix.com/shell-programming-scripting/165717-removing-duplicate-records-file-based-single-column.html Can someone explain this command please? How are there no... (5 Replies)
Discussion started by: cokedude
5 Replies

10. Shell Programming and Scripting

Removing specific records from files when duplicate key

Hello I have been trying to remove a row from a file which has the same first three columns as another row - I have tried lots of different combinations of suggestion on this forum but can't get it exactly right. what I have is 900 - 1000 = 0 900 - 1000 = 2562 1000 - 1100 = 0 1000 - 1100... (7 Replies)
Discussion started by: tinytimmay
7 Replies
SCRIPT(1)						    BSD General Commands Manual 						 SCRIPT(1)

NAME
script -- make typescript of terminal session SYNOPSIS
script [-adkpqr] [-F pipe] [-t time] [file [command ...]] DESCRIPTION
The script utility makes a typescript of everything printed on your terminal. It is useful for students who need a hardcopy record of an interactive session as proof of an assignment, as the typescript file can be printed out later with lpr(1). If the argument file is given, script saves all dialogue in file. If no file name is given, the typescript is saved in the file typescript. If the argument command is given, script will run the specified command with an optional argument vector instead of an interactive shell. The following options are available: -a Append the output to file or typescript, retaining the prior contents. -d When playing back a session with the -p flag, do not sleep between records when playing back a timestamped session. -F pipe Immediately flush output after each write. This will allow a user to create a named pipe using mkfifo(1) and another user may watch the live session using a utility like cat(1). -k Log keys sent to the program as well as output. -p Play back a session recorded with the -r flag in real time. -q Run in quiet mode, omit the start, stop and command status messages. -r Record a session with input, output, and timestamping. -t time Specify the interval at which the script output file will be flushed to disk, in seconds. A value of 0 causes script to flush after every character I/O event. The default interval is 30 seconds. The script ends when the forked shell (or command) exits (a control-D to exit the Bourne shell (sh(1)), and exit, logout or control-D (if ignoreeof is not set) for the C-shell, csh(1)). Certain interactive commands, such as vi(1), create garbage in the typescript file. The script utility works best with commands that do not manipulate the screen. The results are meant to emulate a hardcopy terminal, not an addressable one. ENVIRONMENT
The following environment variables are utilized by script: SCRIPT The SCRIPT environment variable is added to the sub-shell. If SCRIPT already existed in the users environment, its value is overwrit- ten within the sub-shell. The value of SCRIPT is the name of the typescript file. SHELL If the variable SHELL exists, the shell forked by script will be that shell. If SHELL is not set, the Bourne shell is assumed. (Most shells set this variable automatically). SEE ALSO
csh(1) HISTORY
The script command appeared in 3.0BSD. The -d, -p and -r options first appeared in NetBSD 2.0 and were ported to FreeBSD 9.2. BUGS
The script utility places everything in the log file, including linefeeds and backspaces. This is not what the naive user expects. It is not possible to specify a command without also naming the script file because of argument parsing compatibility issues. When running in -k mode, echo cancelling is far from ideal. The slave terminal mode is checked for ECHO mode to check when to avoid manual echo logging. This does not work when the terminal is in a raw mode where the program being run is doing manual echo. If script reads zero bytes from the terminal, it switches to a mode when it only attempts to read once a second until there is data to read. This prevents script from spinning on zero-byte reads, but might cause a 1-second delay in processing of user input. BSD
December 4, 2013 BSD
All times are GMT -4. The time now is 08:09 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy