Sponsored Content
Top Forums Shell Programming and Scripting Calculating expiry date using date,sed,grep Post 302567488 by r4v3n on Monday 24th of October 2011 10:19:13 AM
Old 10-24-2011
Calculating expiry date using date,sed,grep

Hi,

I would greatly appreciate it if someone can help me with my problem.

I have a crawler which collects spam URLs everyday & this data needs to be published in a blacklist.

Here's the catch:

The "Time To Live" (TTL) for each URL is 3 months (or whatever for that matter). If i see the same URL again within the expiry of its TTL, I need to update that URLs TTL, so it stays in the blacklist for another 3 months (TTL).

The URLs which were never seen after the TTL need to be removed from the list after the TTL expires, so I don't have old data & can manage the size of my blacklist.

Here's an example current URL list which my crawler would have got today:
[URL followed by TTL of 3 days or whatever for that matter]
Code:
b.com 23
e.com 23
f.com 23

Here's an example of the current master URL file used for comparison:
Code:
a.com 19
b.com 20
c.com 21
d.com 21

Here's an example of the updated master URL file after comparison:
Code:
b.com 23
c.com 21
d.com 21
e.com 23
f.com 23

Here's what the final blacklist should look like:
Code:
b.com
c.com
d.com
e.com
f.com

How do can do this using using sed/grep/date (if it is indeed possible)? Unfortunately, I can't install any SQL db on this machine, which I realize would make things easy.

Again, any help would be much appreciated.

Thanks in advance
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

domain expiry date

Hi all, I want to write a shell script that read line by line of a file that have all domain names written, after reading complete file it will give us following output. Domain Name Expiry Days-Left abcd.com 20-Feb-200 6 10 How can i acheive that script... (6 Replies)
Discussion started by: learnbash
6 Replies

2. UNIX for Advanced & Expert Users

Digital Certificate Expiry Date

HI, I want to know the certificate expiry dates and who is owner of the certificate installed in UNIX. example certiface name 1)SSL 2)Veri Sign Trust Network (1 Reply)
Discussion started by: RG18173
1 Replies

3. Shell Programming and Scripting

Extract Expiry date and server_name from the following file

Hi Guys, I am trying to write one ksh script where I have to go to lots of unix boxes and read one text file which will have data like this though this is not complete data. And I just have to extract expire date ( which is 2nd date value 08/31/2009 in the first one ) and the server name ( astro... (0 Replies)
Discussion started by: pareshan
0 Replies

4. UNIX for Dummies Questions & Answers

Find user password expiry date

Does anyone know a simple way to find out the expiry date of my UNIX user password? I do not have root access to the box. Surely there is a way to so this... (4 Replies)
Discussion started by: oztoma
4 Replies

5. SuSE

Disable the password expiry date

Hi, We are going to create the new user, using that user we are automate the work, but every 90 days password get expired how can i create or avoid the expiry the password. (1 Reply)
Discussion started by: kingganesh04
1 Replies

6. UNIX for Advanced & Expert Users

expiry date for files?

Hi, Is there any way i can give an expiry date for files that i create? For example, i would like to get a file deleted automatically after 60 days from the creation date. Is there any possibility for this in Unix? Thanks, RRVARMA (3 Replies)
Discussion started by: RRVARMA
3 Replies

7. Shell Programming and Scripting

Expiry date of shell script

Hello folks, I have question about my expiry of shell script, today date is 2009-11-11, i will have to deploy a script today, how i should setup that my script will not work after 2010-05-11, Between if someone will change the date of server then script will again not work, please give... (3 Replies)
Discussion started by: learnbash
3 Replies

8. Shell Programming and Scripting

Password expiry date check for hp servers

Hi All, Can anybody help me, to find the command to check for the password expiry date for the hp servers. Thanks, Deepak (3 Replies)
Discussion started by: dswain
3 Replies

9. Shell Programming and Scripting

Calculating using date

I need to help to calculating using date in a script. One application is licensed by date, some month at a time. I can read the date from system and get an output like this: echo $status 6A34 System4 01.01.11-31.01.11 My goal is to use license date 31.01.11 and subtract todays date... (7 Replies)
Discussion started by: Jotne
7 Replies

10. Solaris

expiry date of an account

can an user see the expiry date of its own account,also can the user know whether it is having peer access or not. Thanks, Megh (2 Replies)
Discussion started by: megh
2 Replies
FSVS - URL format(5)						       fsvs						      FSVS - URL format(5)

NAME
Format of URLs - FSVS can use more than one URL; the given URLs are overlaid according to their priority. FSVS can use more than one URL; the given URLs are overlaid according to their priority. For easier managing they get a name, and can optionally take a target revision. Such an extended URL has the form ['name:'{name},]['target:'{t-rev},]['prio:'{prio},]URL where URL is a standard URL known by subversion -- something like http://...., svn://... or svn+ssh://.... The arguments before the URL are optional and can be in any order; the URL must be last. Example: name:perl,prio:5,svn://... or, using abbreviations, N:perl,P:5,T:324,svn://... Please mind that the full syntax is in lower case, whereas the abbreviations are capitalized! Internally the : is looked for, and if the part before this character is a known keyword, it is used. As soon as we find an unknown keyword we treat it as an URL, ie. stop processing. The priority is in reverse numeric order - the lower the number, the higher the priority. (See url__current_has_precedence() ) Why a priority? When we have to overlay several URLs, we have to know which URL takes precedence - in case the same entry is in more than one. (Which is not recommended!) Why a name? We need a name, so that the user can say 'commit all outstanding changes to the repository at URL x', without having to remember the full URL. After all, this URL should already be known, as there's a list of URLs to update from. You should only use alphanumeric characters and the underscore here; or, in other words, w or [a-zA-Z0-9_]. (Whitespace, comma and semicolon get used as separators.) What can I do with the target revision? Using the target revision you can tell fsvs that it should use the given revision number as destination revision - so update would go there, but not further. Please note that the given revision number overrides the -r parameter; this sets the destination for all URLs. The default target is HEAD. Note: In subversion you can enter URL@revision - this syntax may be implemented in fsvs too. (But it has the problem, that as soon as you have a @ in the URL, you must give the target revision every time!) There's an additional internal number - why that? This internal number is not for use by the user. It is just used to have an unique identifier for an URL, without using the full string. On my system the package names are on average 12.3 characters long (1024 packages with 12629 bytes, including newline): COLUMNS=200 dpkg-query -l | cut -c5- | cut -f1 -d' ' | wc So if we store an id of the url instead of the name, we have approx. 4 bytes per entry (length of strings of numbers from 1 to 1024). Whereas using the needs name 12.3 characters, that's a difference of 8.3 per entry. Multiplied with 150 000 entries we get about 1MB difference in filesize of the dir-file. Not really small ... And using the whole URL would inflate that much more. Currently we use about 92 bytes per entry. So we'd (unnecessarily) increase the size by about 10%. That's why there's an url_t::internal_number. Author Generated automatically by Doxygen for fsvs from the source code. Version trunk:2424 11 Mar 2010 FSVS - URL format(5)
All times are GMT -4. The time now is 06:55 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy