Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Calculating expiry date using date,sed,grep
Top Forums Shell Programming and Scripting Calculating expiry date using date,sed,grep Post 302567488 by r4v3n on Monday 24th of October 2011 10:19:13 AM
Old 10-24-2011
Calculating expiry date using date,sed,grep
Hi,

I would greatly appreciate it if someone can help me with my problem.

I have a crawler which collects spam URLs everyday & this data needs to be published in a blacklist.

Here's the catch:

The "Time To Live" (TTL) for each URL is 3 months (or whatever for that matter). If i see the same URL again within the expiry of its TTL, I need to update that URLs TTL, so it stays in the blacklist for another 3 months (TTL).

The URLs which were never seen after the TTL need to be removed from the list after the TTL expires, so I don't have old data & can manage the size of my blacklist.

Here's an example current URL list which my crawler would have got today:
[URL followed by TTL of 3 days or whatever for that matter]
Code:
b.com 23
e.com 23
f.com 23

Here's an example of the current master URL file used for comparison:
Code:
a.com 19
b.com 20
c.com 21
d.com 21

Here's an example of the updated master URL file after comparison:
Code:
b.com 23
c.com 21
d.com 21
e.com 23
f.com 23

Here's what the final blacklist should look like:
Code:
b.com
c.com
d.com
e.com
f.com

How do can do this using using sed/grep/date (if it is indeed possible)? Unfortunately, I can't install any SQL db on this machine, which I realize would make things easy.

Again, any help would be much appreciated.

Thanks in advance
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

domain expiry date

Hi all, I want to write a shell script that read line by line of a file that have all domain names written, after reading complete file it will give us following output. Domain Name Expiry Days-Left abcd.com 20-Feb-200 6 10 How can i acheive that script... (6 Replies)
Discussion started by: learnbash
6 Replies

2. UNIX for Advanced & Expert Users

Digital Certificate Expiry Date

HI, I want to know the certificate expiry dates and who is owner of the certificate installed in UNIX. example certiface name 1)SSL 2)Veri Sign Trust Network (1 Reply)
Discussion started by: RG18173
1 Replies

3. Shell Programming and Scripting

Extract Expiry date and server_name from the following file

Hi Guys, I am trying to write one ksh script where I have to go to lots of unix boxes and read one text file which will have data like this though this is not complete data. And I just have to extract expire date ( which is 2nd date value 08/31/2009 in the first one ) and the server name ( astro... (0 Replies)
Discussion started by: pareshan
0 Replies

4. UNIX for Dummies Questions & Answers

Find user password expiry date

Does anyone know a simple way to find out the expiry date of my UNIX user password? I do not have root access to the box. Surely there is a way to so this... (4 Replies)
Discussion started by: oztoma
4 Replies

5. SuSE

Disable the password expiry date

Hi, We are going to create the new user, using that user we are automate the work, but every 90 days password get expired how can i create or avoid the expiry the password. (1 Reply)
Discussion started by: kingganesh04
1 Replies

6. UNIX for Advanced & Expert Users

expiry date for files?

Hi, Is there any way i can give an expiry date for files that i create? For example, i would like to get a file deleted automatically after 60 days from the creation date. Is there any possibility for this in Unix? Thanks, RRVARMA (3 Replies)
Discussion started by: RRVARMA
3 Replies

7. Shell Programming and Scripting

Expiry date of shell script

Hello folks, I have question about my expiry of shell script, today date is 2009-11-11, i will have to deploy a script today, how i should setup that my script will not work after 2010-05-11, Between if someone will change the date of server then script will again not work, please give... (3 Replies)
Discussion started by: learnbash
3 Replies

8. Shell Programming and Scripting

Password expiry date check for hp servers

Hi All, Can anybody help me, to find the command to check for the password expiry date for the hp servers. Thanks, Deepak (3 Replies)
Discussion started by: dswain
3 Replies

9. Shell Programming and Scripting

Calculating using date

I need to help to calculating using date in a script. One application is licensed by date, some month at a time. I can read the date from system and get an output like this: echo $status 6A34 System4 01.01.11-31.01.11 My goal is to use license date 31.01.11 and subtract todays date... (7 Replies)
Discussion started by: Jotne
7 Replies

10. Solaris

expiry date of an account

can an user see the expiry date of its own account,also can the user know whether it is having peer access or not. Thanks, Megh (2 Replies)
Discussion started by: megh
2 Replies

LEARN ABOUT FREEBSD

dnsblog

DNSBLOG(8postfix)														 DNSBLOG(8postfix)

NAME

       dnsblog - Postfix DNS white/blacklist logger

SYNOPSIS

       dnsblog [generic Postfix daemon options]

DESCRIPTION

       The  dnsblog(8)	server	implements  an ad-hoc DNS white/blacklist lookup service. This may eventually be replaced by an UDP client that is
       built directly into the postscreen(8) server.

PROTOCOL

       With each connection, the dnsblog(8) server receives a DNS white/blacklist domain name, an IP address, and an ID.  If  the  IP  address	is
       listed  under  the DNS white/blacklist, the dnsblog(8) server logs the match and replies with the query arguments plus an address list with
       the resulting IP addresses, separated by whitespace, and the reply TTL.	Otherwise it replies  with  the  query	arguments  plus  an  empty
       address	list  and  the reply TTL; the reply TTL is -1 if there is no reply, or a negative reply that contains no SOA record.  Finally, the
       dnsblog(8) server closes the connection.

DIAGNOSTICS

       Problems and transactions are logged to syslogd(8).

CONFIGURATION PARAMETERS

       Changes to main.cf are picked up automatically, as dnsblog(8) processes run for only a limited amount of time.  Use  the  command  "postfix
       reload" to speed up a change.

       The text below provides only a parameter summary. See postconf(5) for more details including examples.

       config_directory (see 'postconf -d' output)
	      The default location of the Postfix main.cf and master.cf configuration files.

       daemon_timeout(18000s)
	      How much time a Postfix daemon process may take to handle a request before it is terminated by a built-in watchdog timer.

       postscreen_dnsbl_sites (empty)
	      Optional list of DNS white/blacklist domains, filters and weight factors.

       ipc_timeout(3600s)
	      The time limit for sending or receiving information over an internal communication channel.

       process_id (read-only)
	      The process ID of a Postfix command or daemon process.

       process_name (read-only)
	      The process name of a Postfix command or daemon process.

       queue_directory (see 'postconf -d' output)
	      The location of the Postfix top-level queue directory.

       syslog_facility (mail)
	      The syslog facility of Postfix logging.

       syslog_name (see 'postconf -d' output)
	      A prefix that is prepended to the process name in syslog records, so that, for example, "smtpd" becomes "prefix/smtpd".

       Available in Postfix 3.3 and later:

       service_name (read-only)
	      The master.cf service name of a Postfix daemon process.

SEE ALSO

       smtpd(8), Postfix SMTP server
       postconf(5), configuration parameters
       syslogd(5), system logging

LICENSE

       The Secure Mailer license must be distributed with this software.

HISTORY

       This service was introduced with Postfix version 2.8.

AUTHOR(S)
       Wietse Venema
       IBM T.J. Watson Research
       P.O. Box 704
       Yorktown Heights, NY 10598, USA

       Wietse Venema
       Google, Inc.
       111 8th Avenue
       New York, NY 10011, USA

																 DNSBLOG(8postfix)
All times are GMT -4. The time now is 04:15 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy