Sponsored Content
Full Discussion: big file processeing
Top Forums Shell Programming and Scripting big file processeing Post 35435 by omran on Saturday 12th of April 2003 09:31:00 AM
Old 04-12-2003
big file processeing

hi,

i have a very big file that holding data, how could i pick line by line from this file.
the following process can illustrate better:

file
-------------------
123444444 |
122314567 |-----------data
146689000 |

c=123444444 ---------- c is variable

process c ----------- passing c to script (sh,sql....etc)

c=122314567 ---------- c is variable

process c ----------- passing c to script (sh,sql....etc)

c=146689000 ---------- c is variable

process c ----------- passing c to script (sh,sql....etc)


i think u understand what i want to do
please help

thanx in advance
 

8 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

How to view a big file(143M big)

1 . Thanks everyone who read the post first. 2 . I have a log file which size is 143M , I can not use vi open it .I can not use xedit open it too. How to view it ? If I want to view 200-300 ,how can I implement it 3 . Thanks (3 Replies)
Discussion started by: chenhao_no1
3 Replies

2. Shell Programming and Scripting

bath processeing script

am currently working on a batch processing script and i am stuck I am not very familiar with the korn shell I need to do the following: Process an input file with the following information: SOURCE FILE 533650_MSCIEUROPE_AvgWeight_YTD_EXP.XLS/Daily/test/Ceurope/EuropeFactset/YTD/... (1 Reply)
Discussion started by: chambala5
1 Replies

3. Solaris

wtmpx file is too big

Hi, I am using Sun Solaris 5.9 OS. I have found a file called wtmpx having a size of 5.0 GB. I want to clear this file using :>/var/adm/wtmpx. My query is, would it cause any problem to the running live system. Could anyone suggest the best method to clear the file without causing problem to... (6 Replies)
Discussion started by: Vijayakumarpc
6 Replies

4. Shell Programming and Scripting

Inserting a column from one file into another big file

Hi I have two files, one is 1.6 GB. I would like to add one extra column of information to the large file at a specific location (after its 2nd column). For example: File 1 has two columns more than 1000 rows like this MM009987 1 File 2 looks like this MM00098 MM00076 3 4 2 4 2... (1 Reply)
Discussion started by: sogi
1 Replies

5. UNIX for Dummies Questions & Answers

How big is too big a config.log file?

I have a 5000 line config.log file with several "maybe" errors. Any reccomendations on finding solvable problems? (2 Replies)
Discussion started by: NeedLotsofHelp
2 Replies

6. Shell Programming and Scripting

parsing data from a big file using keys from another smaller file

Hi, I have 2 files format of file 1 is: a1 b2 a2 c2 d1 f3 format of file 2 is (tab delimited): a1 1.2 0.5 0.06 0.7 0.9 1 0.023 a3 0.91 0.007 0.12 0.34 0.45 1 0.7 a2 1.05 2.3 0.25 1 0.9 0.3 0.091 b1 1 5.4 0.3 9.2 0.3 0.2 0.1 b2 3 5 7 0.9 1 9 0 1 b3 0.001 1 2.3 4.6 8.9 10 0 1 0... (10 Replies)
Discussion started by: Lucky Ali
10 Replies

7. Emergency UNIX and Linux Support

Getting VALUE from Big XML File -- That's All

We got data that was supposed to be CSV, but was sent in a huge XML file. I've downloaded xmlstarlet, but I'm darned if I can get it to operate the "sel" feature to look down a path and get any sort of value. I see pieces of what should be paths, but they seem to have extraneous characters, and... (7 Replies)
Discussion started by: gmark99
7 Replies

8. UNIX for Beginners Questions & Answers

How to convert CR to LF in a big file?

Hello Friends, I have a big file that is transferred to my UNIX system and it seems it has CR as the line delimiter When I run file <filename> <filename>: ASCII text, with CR line terminators How do I convert the file to one with LF as terminators so that my code that runs on UNIX can... (3 Replies)
Discussion started by: mehimadri12
3 Replies
dspam_maintenance(1)						   User Commands					      dspam_maintenance(1)

NAME
dspam_maintenance - remove old signatures and unimportant tokens from the DSPAM storage backend SYNOPSIS
dspam_maintenance [--profile=PROFILE] [--logdays=no_of_days] [--signatures=no_of_days] [--neutral=no_of_days] [--unused=no_of_days] [--hapaxes=no_of_days] [--hits1s=no_of_days] [--hits1i=no_of_days] [--without-sql-purge] [--with-sql-automatization] [--with-sql-autoup- date] [--purgescriptdir=directory] [--with-all-drivers] [--verbose] DESCRIPTION
dspam_maintenance is used to remove old signatures/tokens and unimportant tokens from the DSPAM storage backend and purge old log entries in user and system logs. This script can be called from the command line or run from within cron. Either add a line in your crontab or add this script in your cron.{hourly,daily,weekly,monthly}. Running this script every hour might not be the best idea but it's your decision if you want to do so. OPTIONS
--profile=PROFILE Specify a storage profile from dspam.conf. The storage profile selected will be used for all database connectivity. NOTE: See dspam.conf for more information. --logdays=no_of_days All log entries older than no_of_days days will be removed. NOTE: Default is 31 days. (for more info: man dspam_logrotate) --signatures=no_of_days All signatures older than no_of_daysdays will be removed. NOTE: Default value is set by the PurgeSignatures option in dspam.conf, if this option is not set, the default value is 14 days. This option is only used for the Hash driver. --neutral=no_of_days Deletes all tokens from the target users database whose probability is between 0.35 and 0.65 (fairly neutral, useless data). NOTE: Default value is set by the PurgeNeutral option in dspam.conf, if this option is not set, the default value is 90 days. --unused=no_of_days Remove stale tokens which have not been used for a long period of time. NOTE: Default value is set by the PurgeUnused option in dspam.conf, if this option is not set, the default value is 90 days. --hapaxes=no_of_days Remove tokens with a total hit count below 5 (which will be assigned a hapaxial value by DSPAM). NOTE: Default value is set by the PurgeHapaxes option in dspam.conf, if this option is not set, the default value is 30 days. --hits1s=no_of_days Remove tokens with a single SPAM hit. NOTE: Default value is set by the PurgeHits1S option in dspam.conf, if this option is not set, the default value is 15 days. --hits1i=no_of_days Remove tokens with a single INNOCENT hit. NOTE: Default value is set by the PurgeHits1I option in dspam.conf, if this option is not set, the default value is 15 days. --without-sql-purge Do not use SQL based purging. Only run dspam_clean. NOTE: Default is off (aka: use SQL based purging). --with-sql-optimization Run VACUUM (for PostgreSQL/SQLite) and/or OPTIMIZE (for MySQL). NOTE: Default is off (aka: do not use optimizations). --with-sql-autoupdate Run SQL based purging with purge day values passed to dspam_maintenance. NOTE: Default is off (aka: do not attempt to modify SQL instructions). --purgescriptdir=directory Space separated list of directories where to search for SQL files used for the SQL based purging. --with-all-drivers Process all installed storage drivers (not just the active driver). NOTE: Default is true (aka: process all installed drivers). --verbose Verbose output while running maintenance script. EXIT VALUE
0 Operation was successful. other Operation resulted in an error. Run the script with --verbose to get more information about error. COPYRIGHT
Copyright (C) 2002-2010 DSPAM Project All rights reserved. For more information, see http://dspam.sourceforge.net. SEE ALSO
dspam(1), dspam_admin(1), dspam_clean(1), dspam_crc(1), dspam_dump(1), dspam_logrotate(1), dspam_merge(1), dspam_stats(1), dspam_train(1) dspam_maintenance Apr 26, 2010 dspam_maintenance(1)
All times are GMT -4. The time now is 07:16 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy