Sponsored Content
Full Discussion: Parsing a large log
Top Forums Shell Programming and Scripting Parsing a large log Post 302199935 by asth on Wednesday 28th of May 2008 06:35:36 AM
Old 05-28-2008
Parsing a large log

I need to parse a large log say 300-400 mb
The commands like awk and cat etc are taking time.
Please help how to process.
I need to process the log for certain values of current date.
But I am unbale to do so.
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Splitting a large log file

Okay, absolute newbie here... I'm on a Mac trying to split an almost 2 Gig log file on a Unix box into manageable chunks for my web-based log analysis tool. What do I need to do, what programs do I need to do it? All and any help appreciated/needed :-) Cheers (8 Replies)
Discussion started by: simmonet
8 Replies

2. Shell Programming and Scripting

Problem with parsing a large file

Hi All, Following is the sample file and following is the op desired that is the last entry of each unique first field is required. My solution is as follows However the original file has around a million entries and around a 100,000 uniques first fields, so this soln.... (6 Replies)
Discussion started by: gauravgoel
6 Replies

3. Shell Programming and Scripting

Cutting a large log file in to smaller ones

I have a very large (150 megs) IRC log file from 2000-2001 which I want to cut down to individual daily log files. I have a very basic knowledge of the cat, sed and grep commands. The log file is time stamped and each day in the large log file begins with a "Session Start" string like so: ... (11 Replies)
Discussion started by: MrTangent
11 Replies

4. Shell Programming and Scripting

parsing large CDR XML file

Dear Freind in the file attached how parse the data to be like a normal table :D (3 Replies)
Discussion started by: saifsafaa
3 Replies

5. Red Hat

Help for capturing a URL from a line from large log file

Can someone please help me how do I find a URL from lines of log file and write all the output to a new file? For e.g - Log file has similar entries, 39.155.67.5 - - "GET /abc/login?service=http://161.120.36.39/CORPHR/TMA2007/default.asp HTTP/1.1" 401 3218 54.155.63.9 - - "GET... (2 Replies)
Discussion started by: rockf1bull
2 Replies

6. Shell Programming and Scripting

Help needed for parsing large XML with awk.

My XML structure looks like: <?xml version="1.0" encoding="UTF-8"?> <SearchRepository> <SearchItems> <SearchItem> ... </SearchItem> <SearchItem> ... ... (1 Reply)
Discussion started by: jasonjustice
1 Replies

7. UNIX for Dummies Questions & Answers

I need to isolate a date in a large log file

I wrote head -n1 example.log I grab the first line of the log, but I need to isolate just the date, which is 08/May/2012:09:52:52. I also need to find the reverse of this, which would be tail... http://i.imgur.com/Lp1eBD0.png Thanks in advance (4 Replies)
Discussion started by: spookydll
4 Replies

8. Shell Programming and Scripting

Log parsing

I have a directory with daily logs that have records like this: Date: 04/17/13 Time: 09:29:15 IP: 123.123.123.123 URL: usr/local/file1 and I want to only count how many times each file was accessed (e.g. file1 in that example above), and I want to also look in all the logs in the current... (3 Replies)
Discussion started by: Jaymz
3 Replies

9. Shell Programming and Scripting

Parsing large files in Solaris 11

I have a 1.2G file that contains no newline characters. This is essentially a log file with each entry being exactly 78bits long. The basic format is /DATE/USER/MISC/. The single uniform thing about the file is that that the 8 character is always ":" I worked with smaller files of the same... (8 Replies)
Discussion started by: os2mac
8 Replies

10. Shell Programming and Scripting

Parsing a subset of data from a large matrix

I do have a large matrix of the following format and it is tab delimited ch-ab1-20 ch-bb2-23 ch-ab1-34 ch-ab1-24 er-cc1-45 bv-cc1-78 ch-ab1-20 0 2 3 4 5 6 ch-bb2-23 3 0 5 ... (6 Replies)
Discussion started by: Kanja
6 Replies
MERGELOGS(1)						      General Commands Manual						      MERGELOGS(1)

NAME
mergelogs - merge and consolidate web server logs SYNOPSIS
mergelogs -p penlog [-c] [-d] [-j jitter] [-t seconds] server1:logfile1 [server2:logfile2 ...] EXAMPLES
mergelogs -p pen.log 10.0.0.1:access_log.1 10.0.0.2:access_log.2 mergelogs -p pen.log 10.0.18.6:access_log-10.0.18.6 10.0.18.8:access_log-10.0.18.8 DESCRIPTION
When pen is used to load balance web servers, the web server log file lists all accesses as coming from the host running pen. This makes it more difficult to analyze the log file. To solve this, pen creates its own log file, which contains the real client address, the time of the access, the target server address and the first few bytes of the requests. Mergelogs reads pen's log file and the log files of all load balanced web servers, compares each entry and creates a combined log file that looks as if the web server cluster were a single physical server. Client addresses are replaced with the real client addresses. In the event that no matching client address can be found in the pen log, the server address is used instead. This should never happen, and is meant as a debugging tool. A large number of these indicates that the server system date needs to be set, or that the jitter value is too small. You probably don't want to use this program. Penlog is a much more elegant and functional solution. OPTIONS
-c Do not cache pen log entries. The use of this option is not recommended, as it will make mergelogs search the entire pen log for every line in the web server logs. -d Debugging (repeat for more). -p penlog Log file from pen. -j jitter Jitter in seconds (default 600). This is the maximum variation in time stamps in the pen and web server log files. A smaller value will result in a smaller pen log cache and faster processing, at the risk of missed entries. -t seconds The difference in seconds between the time on the pen server and UTC. For example, this is 7200 (two hours) in Finland. server:logfile Web server address and name of log file. AUTHOR
Copyright (C) 2001-2003 Ulric Eriksson, <ulric@siag.nu>. SEE ALSO
pen(1), webresolve(1), penlog(1), penlogd(1) LOCAL MERGELOGS(1)
All times are GMT -4. The time now is 08:48 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy