Table Cleanup Script


 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users Table Cleanup Script
# 1  
Old 10-08-2010
Table Cleanup Script

I needed some help with a script to fetch and delete all records prior to 3 days from now connecting to sybase from sunos. I wrote the following script but not working..can someone please guide me with my code.

Thanks

Code:
#!/bin/ksh
##GET PREVIOUS DAY DATE
dt=`date | awk '{printf"%4d%2d%2d\n",$6,$2,($3-3)}' | sed 's/ /0/g'`

#GET THE RECORDS FROM 3 DAYS OLD
SQL_QUERY="select * from CLEANUPTABLE where convert(varchar(12), ORDER_DATE, 3) <= '${dt}'"
echo $SQL_QUERY
LOADER=`isql -U${dbuid} -P${dbpwd} -S${dbsrv} << EOF
${LOADER}
${SQL_QUERY} >>cleanup.out
go
EXIT;
EOF

if [ $? -ne 0 ]
then
echo "Failed to execute query: ${SQL_QUERY}">> $LOG_FILE
exit -10 ;
fi


Last edited by moe458; 10-08-2010 at 11:46 AM..
# 2  
Old 10-08-2010
Last I checked, truncate table does it all only for the table owner.


Quote:
Sybase Data Integration Suite 1.1 - ETL > Sybase ETL 4.2 User's Guide > Components > Destination components > DB Bulk Load Sybase IQ > Optional properties
Image Image
Chapter 6: Components
Truncate

Activate this option to remove all records from the destination table when initializing the transformation process.
Quote:
TRUNCATE statement




Deletes all rows from a table without deleting the table definition.
Image Syntax

TRUNCATE TABLE [ owner.]table-name| MATERIALIZED VIEW [ owner.]materialized-view-name
Image Remarks

The TRUNCATE statement deletes all rows from the table or materialized view.







My favorite archival delete is something like:
  1. select the delete target rows to a global temp table.
  2. bcp that table out.
  3. Verify the row and line counts match.
  4. Using a script or stored procedure or TSQL loop, for 128 lines/rows at a time in clustered index column order, delete original rows and commit.
This ensures minimal target table impact, and nothing deleted is ever noy archived, even if the unattended script has a problem.

Missing the last go & eof linefeed, second ` misplaced, misspelled table name?

Must be a code frag, fi but not if.

You are removing partial days (three days to the runtime microsecond) if your date column is a datetime type:



Maybe you want distinct, but I always say, if you think distinct, reconsider as count(*) group by.

I am not a big fan of <<, as the behavior of echo '...' is more predictable and it flows better reading left to right. I haven't quantified if a tmp file create for << is cheaper than a pipe and fork for echo, just cleaner piped stream layout. I found your second ` in the middle, when in needed to be a line below EOF.

Code:
 
#!/bin/ksh
 
echo ' CLEANUPTABLE
' >cleanup.out
 
results=$(
echo "
delete CLEANUPTABLE
where datetime_col < dateadd ('dd',-3, getdate())
go
select datetime_col, count(*) ct
from CLEANUPTABLE
group by datetime_col
go
' | isql -U${dbuid} -P${dbpwd} -S${dbsrv} -w3000 -E vi
)


Last edited by vbe; 10-08-2010 at 01:39 PM.. Reason: typo
This User Gave Thanks to DGPickett For This Post:
# 3  
Old 10-08-2010
I'm able to run the query in the sybase and it works fine but when i try to put it in the script it gives errors..not sure what the problem is..i get the following error


Code:
sunos-moe > cleanup.sh
select cleanup.out cleanup.sh cleanup.tmp test.sh truncateEOL.sh truncateEOL.sql from CLEANUPTABLE  where convert(varchar(12), ORDER_DATE, 3) <= '20100005'
cleanup.sh[6]: syntax error at line 6 : ``' unmatched
sunos-moe >



---------- Post updated at 10:51 AM ---------- Previous update was at 10:49 AM ----------

Code:
1> select count(*) from CLEANUPTABLE  where convert(varchar(12), ORDER_DATE, 3) <= '20100810'
2> go
             
 ----------- 
       97039 

(1 row affected)
1>


here is the what I had pulled from the database...i don't want to truncate anything unless i'm able to select..there are 130K records in there...just want to clean up what i want..let's do the select and deal with the truncate later ;-)

---------- Post updated at 01:37 PM ---------- Previous update was at 10:51 AM ----------

Okay I got it resolved.

Code:
#!/bin/ksh

dt=`date | awk '{printf"%4d%2d%2d\n",$6,$2,($3-3)}' | sed 's/ /0/g'`
isql -U${dbuid} -P${dbpwd} -S${dbsrv} << EOF
select * from CLEANUPTABLE where convert(varchar(12), ORDER_DATE, 3) <= '${dt}' 
go
output to cleanup.out
EXIT;
EOF

# 4  
Old 10-12-2010
At the low level, see man page for date + option.

The problem with date and getdate() is that you do not want to be tied to the clock at any low level, ever, except writing history rows and similar real time activities. You should be able to test with arbitrary data/dates, or if there is an outage, clean up on a later day.

You probably want each archive file to hold all of one day. You could select each distinct date old enough to archive and loop for each date at a high level to archive and delete, so it automatically does more days if it fails to run some days.

T-SQL can can convert date strings using select without from eith in a scalar subquery to load a variable.

You do not want to convert ever row in the table if you can avoid it, you want to convert your bound(s) constants, especially if there is a useful index. BTW, a table clustered on time is lock-prone for non-batch use, as all users are churning at the end of the table.

You want to ensure that either:
  • the date_time_col is a string, or
  • the date_time_col is always loaded with a date and no time, or
  • the date_time_col is processed in a range for the one day, e.g., "date_time_col >= day and date_time_col < (day + 1)".
Wasting a date-time datatype with just date is confusing, too, so often designers use either a CCYYMMDD string, a CCYYMMDD as literal int or a int days since epoch date. We all have 8 byte wide CPUs these days, so getting it down to 4 bytes is mostly useful for space saving.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Web Development

Getting Rid of Annoying Bootstrap Table Borders and Wayward Table Lines

Bootstrap is great; but we have had some issues with Bootstrapped <tables> (and legacy <fieldset> elements) showing annoying, wayward lines. I solved that problem today with this simple jQuery in the footer: <script> $(function(){ $('tr, td, fieldset,... (0 Replies)
Discussion started by: Neo
0 Replies

2. Shell Programming and Scripting

UNIX script for cleanup

Hello, I need some help from unix guru's here..I am looking for some advanced level script to cleanup the directories and files from specific directories under a file system.. The folders are created under /opt/modules And under modules, there are multiple subfolders with the application... (6 Replies)
Discussion started by: mb525
6 Replies

3. Shell Programming and Scripting

Suggestion with script to cleanup

I need help with sed and awk scripts to search for Symmetrix ID=000090009902 and then grep its child disk devices associated to the dead paths and display them only, so that those dead devices can be removed. test01:/#powermt display dev=all Pseudo name=hdiskpower0 Symmetrix ID=000090009902... (0 Replies)
Discussion started by: aix_admin_007
0 Replies

4. UNIX for Dummies Questions & Answers

Creating a condensed table from a pre-existing table in putty

Hello, I'm working with putty on Windows 7 professional and I'd like to know if there's a way to gather specific lines from a pre-existing table and make a new table with that information. More specifically, I'd like the program to look at a specific column, say column N, and see if any of the... (5 Replies)
Discussion started by: Deedee393
5 Replies

5. Shell Programming and Scripting

pid.cleanup script.

Hi guys! I have a directory in the production environment from which i have to delete files older then 40 minutes with .pid extention. I wrote a script below for the purpose. #!/bin/bash # # Script to delete specific file older than N minutes. # OLDERTHAN="40" #40 minutes ... (6 Replies)
Discussion started by: sajid.shah
6 Replies

6. Shell Programming and Scripting

Mail cleanup from ksh script, keeping 50 most recent msgs

I found some posts describing how to completely clean out a mailbox in Unix/Linux. But I want to keep the 50 most recent messages. Any ideas out there? Thanks! (3 Replies)
Discussion started by: OPTIMUS_prime
3 Replies

7. Shell Programming and Scripting

Suggestions/cleanup Bash script

Hello, beginner bash scripter here.. I was able to write a script and it works just fine. I'm just wondering if someone could chime in or any suggestions to make it cleaner or tighter so to speak. I have a disk to disk backup solution which uses 250GB disks. When one gets full I just po in a new... (7 Replies)
Discussion started by: woodson2
7 Replies

8. Shell Programming and Scripting

Cleanup script

Hi! I would like to write a script which remove some files, all beginning with the same prefix : prefix.1 doc/prefix.2 ../prefix.3 etc. So, I would create a file and chmod it executable. But I dont know how to pass a variable to a script. I would like to write something like ... (2 Replies)
Discussion started by: tipi
2 Replies

9. Shell Programming and Scripting

User Cleanup Script

Hi Guys, I've got an system setup to act as an sftp server. I have a script that allows me to create chroot users running a custom shell within their home directory, it also creates a subdirectory that they can write into. I'm trying to write a script (that I can cron at a later date) that checks... (3 Replies)
Discussion started by: King_Brucie
3 Replies

10. Shell Programming and Scripting

awk/sed/ksh script to cleanup /etc/group file

Many of my servers' /etc/group file have many userid's that does not exist in /etc/passwd file and they need to be deleted. This happened due to manual manipulation of /etc/passwd files. I need to do this for 40 servers. Can anyone help me in achieving this? Even reducing a step or two will be... (6 Replies)
Discussion started by: pdtak
6 Replies
Login or Register to Ask a Question