what is the most effective way to process a large logfile?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Process multiple large files with awk

Hi there, I'm camor and I'm trying to process huge files with bash scripting and awk. I've got a dataset folder with 10 files (16 millions of row each one - 600MB), and I've got a sorted file with all keys inside. For example: a sample_1 200 a.b sample_2 10 a sample_3 10 a sample_1 10 a...

2. Shell Programming and Scripting

Logfile monitoring with logfile replacement

Bonjour, I've wrote a script to monitor a logfile in realtime. It is working almost perfeclty except for two things. The script use the following technique : tail -fn0 $logfile | \ while read line ; do ... some stuff done First one, I'd like a way to end the monitoring script if a...

3. Cybersecurity

Password rules not effective

I was looking for a good list of words to exclude people from using as passwords, i.e. those that could be guessed easily. I'm working through a whole bunch of suggestions from skullsecurity.org, but I managed to find this page that seems to suggest I have more options than I thought. :b: I...

4. UNIX for Dummies Questions & Answers

Real and Effective IDs

Can anyone explain me in details of Real and Effective IDs

5. Shell Programming and Scripting

how to track a log file of one process and mail that logfile to a group?

Hi Friendz, I have 14 DB load scripts say 1,2,3....14. I want a script to call each script automatically, and after completion of every script, it needs to track the logfile and mail that log file to a group.and again it should run the next script in sequence 1,2,3...14 Please help me,need...

6. Red Hat

Fork wait in background process - large delay

hi all, We are trying to run a process in the background and in the process we call fork ;and wait for the child process to finish .We find that the died = wait(&status); happens after 10 seconds randomly and sometimes completes in time (within 1 sec) This behavior is seen only when the...

7. Red Hat

Fork wait in background process - large delay

8. Shell Programming and Scripting

Scripting the process to edit a large file

Hi, I need to make a script to edit a file. File is a large file in below format Version: 2008120101 ;$INCLUDE ./abc/xyz/Delhi ;$INCLUDE ./abc/xyz/London $INCLUDE ./abc/xyz/New York First line in the file is version number which is in year,month,date and serial number format. Each...

9. Shell Programming and Scripting

get last section from large logfile

Hi all, I need to pull the (last section) from the header to the end of file. This logfile will have many sections appended, but I only need to capture the last one and email it to someone. Any ideas? Thanks Kathy Example Logfile: ============= = BOX : SAPCFI = JOB : SAPCFI =...

10. UNIX for Dummies Questions & Answers

most effective search ?

what's the most efficient and effective search for a file in a dir ? I see many guys use this # find - print or something as such ? and sometimes pipe it to something else ? Is there a better way of using "grep" in all of this ? thanks simon2000

Shell Programming and Scripting