05-28-2008
And show us the regex you are using. Simple greps would take a few seconds max unless your disks are very slow. It's reading the file linearly so you can't get much faster performance than that. If cat is too slow, there really isn't much hope in making it fast enough, other than replacing the disk, or managing the file in a different way (split into smaller chunks? Import into a DBMS?)
10 More Discussions You Might Find Interesting
1. UNIX for Dummies Questions & Answers
Okay, absolute newbie here...
I'm on a Mac trying to split an almost 2 Gig log file on a Unix box into manageable chunks for my web-based log analysis tool.
What do I need to do, what programs do I need to do it?
All and any help appreciated/needed :-)
Cheers (8 Replies)
Discussion started by: simmonet
8 Replies
2. Shell Programming and Scripting
Hi All,
Following is the sample file
and following is the op desired
that is the last entry of each unique first field is required.
My solution is as follows
However the original file has around a million entries and around a 100,000 uniques first fields, so this soln.... (6 Replies)
Discussion started by: gauravgoel
6 Replies
3. Shell Programming and Scripting
I have a very large (150 megs) IRC log file from 2000-2001 which I want to cut down to individual daily log files. I have a very basic knowledge of the cat, sed and grep commands. The log file is time stamped and each day in the large log file begins with a "Session Start" string like so:
... (11 Replies)
Discussion started by: MrTangent
11 Replies
4. Shell Programming and Scripting
Dear Freind
in the file attached how parse the data to be like a normal table
:D (3 Replies)
Discussion started by: saifsafaa
3 Replies
5. Red Hat
Can someone please help me how do I find a URL from lines of log file and write all the output to a new file?
For e.g - Log file has similar entries,
39.155.67.5 - - "GET /abc/login?service=http://161.120.36.39/CORPHR/TMA2007/default.asp HTTP/1.1" 401 3218
54.155.63.9 - - "GET... (2 Replies)
Discussion started by: rockf1bull
2 Replies
6. Shell Programming and Scripting
My XML structure looks like:
<?xml version="1.0" encoding="UTF-8"?>
<SearchRepository>
<SearchItems>
<SearchItem>
...
</SearchItem>
<SearchItem>
...
... (1 Reply)
Discussion started by: jasonjustice
1 Replies
7. UNIX for Dummies Questions & Answers
I wrote head -n1 example.log I grab the first line of the log, but I need to isolate just the date, which is 08/May/2012:09:52:52. I also need to find the reverse of this, which would be tail...
http://i.imgur.com/Lp1eBD0.png
Thanks in advance (4 Replies)
Discussion started by: spookydll
4 Replies
8. Shell Programming and Scripting
I have a directory with daily logs that have records like this:
Date: 04/17/13 Time: 09:29:15
IP: 123.123.123.123
URL: usr/local/file1
and I want to only count how many times each file was accessed (e.g. file1 in that example above), and I want to also look in all the logs in the current... (3 Replies)
Discussion started by: Jaymz
3 Replies
9. Shell Programming and Scripting
I have a 1.2G file that contains no newline characters. This is essentially a log file with each entry being exactly 78bits long. The basic format is /DATE/USER/MISC/. The single uniform thing about the file is that that the 8 character is always ":"
I worked with smaller files of the same... (8 Replies)
Discussion started by: os2mac
8 Replies
10. Shell Programming and Scripting
I do have a large matrix of the following format and it is tab delimited
ch-ab1-20 ch-bb2-23 ch-ab1-34 ch-ab1-24 er-cc1-45 bv-cc1-78
ch-ab1-20 0 2 3 4 5 6
ch-bb2-23 3 0 5 ... (6 Replies)
Discussion started by: Kanja
6 Replies
LEARN ABOUT DEBIAN
io_tryread
io_tryread(3) Library Functions Manual io_tryread(3)
NAME
io_tryread - read from a descriptor without blocking
SYNTAX
#include <io.h>
int io_tryread(int64 fd,char* buf,int64 len);
DESCRIPTION
io_tryread tries to read len bytes of data from descriptor fd into buf[0], buf[1], ..., buf[len-1]. (The effects are undefined if len is 0
or smaller.) There are several possible results:
o o_tryread returns an integer between 1 and len: This number of bytes was available for immediate reading; the bytes were read into the
beginning of buf. Note that this number can be, and often is, smaller than len; you must not assume that io_tryread always succeeds in
reading exactly len bytes.
o io_tryread returns 0: No bytes were read, because the descriptor is at end of file. For example, this descriptor has reached the end of
a disk file, or is reading an empty pipe that has been closed by all writers.
o io_tryread returns -1, setting errno to EAGAIN: No bytes were read, because the descriptor is not ready. For example, the descriptor is
reading an empty pipe that could still be written to.
o io_tryread returns -3, setting errno to something other than EAGAIN: No bytes were read, because the read attempt encountered a persis-
tent error, such as a serious disk failure (EIO), an unreachable network (ENETUNREACH), or an invalid descriptor number (EBADF).
io_tryread does not pause waiting for a descriptor that is not ready. If you want to pause, use io_waitread or io_wait.
You can make io_tryread faster and more efficient by making the socket non-blocking with io_nonblock().
SEE ALSO
io_nonblock(3), io_waitread(3), io_tryreadtimeout(3)
io_tryread(3)