Cutting a large log file in to smaller ones Post: 51310

Sponsored Content

Top Forums Shell Programming and Scripting Cutting a large log file in to smaller ones Post 51310 by MrTangent on Monday 17th of May 2004 06:39:06 AM

05-17-2004

Registered User

Cutting a large log file in to smaller ones

I have a very large (150 megs) IRC log file from 2000-2001 which I want to cut down to individual daily log files. I have a very basic knowledge of the cat, sed and grep commands. The log file is time stamped and each day in the large log file begins with a "Session Start" string like so:

Session Start: Tue Mar 21 03:54:00 2000

Any idea how I could take this large log file, and use sed/grep to break it up in to individual daily versions? Ideally the output would save in to a directory of my chosing in sequential format. Something like so: "resultingfile001.log" and "resultingfile002.log" and so on. I will manually rename the files afterwards.

Any and all help would be greatly appreciated!

MrTangent

View Public Profile for MrTangent

Find all posts by MrTangent

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Need help cutting a couple of fields from log file.

Data begins as such; Mar 16 03:27:05 afzpimdn01 named: denied update from .3983 for "nnn.nnn.in-addr.arpa" IN Mar 16 03:27:37 afzpimdn01 named: denied update from .4844 for "nnn.nnn.in-addr.arpa" IN Mar 16 03:27:56 afzpimdn01 named: denied update from .2650 for "nnn.nnn.in-addr.arpa" IN ...

2. UNIX for Dummies Questions & Answers

splitting the large file into smaller files

hi all im new to this forum..excuse me if anythng wrong. I have a file containing 600 MB data in that. when i do parse the data in perl program im getting out of memory error. so iam planning to split the file into smaller files and process one by one. can any one tell me what is the code...

3. Shell Programming and Scripting

Help with splitting a large text file into smaller ones

Hi Everyone, I am using a centos 5.2 server as an sflow log collector on my network. Currently I am using inmons free sflowtool to collect the packets sent by my switches. I have a bash script running on an infinate loop to stop and start the log collection at set intervals - currently one...

4. UNIX for Dummies Questions & Answers

multiple smaller files from one large file

I have a file with a simple list of ids. 750,000 rows. I have to break it down into multiple 50,000 row files to submit in a batch process.. Is there an easy script I could write to accomplish this task?

5. Shell Programming and Scripting

Split large file into smaller file

hi Guys i need some help here.. i have a file which has > 800,000 lines in it. I need to split this file into smaller files with 25000 lines each. please help thanks

6. Shell Programming and Scripting

Help needed - Split large file into smaller files based on pattern match

Help needed urgently please. I have a large file - a few hundred thousand lines. Sample CP START ACCOUNT 1234556 name 1 CP END ACCOUNT CP START ACCOUNT 2224444 name 1 CP END ACCOUNT CP START ACCOUNT 333344444 name 1 CP END ACCOUNT I need to split this file each time "CP START...

7. Shell Programming and Scripting

Sed: Splitting A large File into smaller files based on recursive Regular Expression match

I will simplify the explaination a bit, I need to parse through a 87m file - I have a single text file in the form of : <NAME>house........ SOMETEXT SOMETEXT SOMETEXT . . . . </script> MORETEXT MORETEXT . . .

8. UNIX for Dummies Questions & Answers

Split large file to smaller fastly

hi , I have a requirement input file: 1 1111111111111 108 1 1111111111111 109 1 1111111111111 109 1 1111111111111 110 1 1111111111111 111 1 1111111111111 111 1 1111111111111 111 1 1111111111111 112 1 1111111111111 112 1 1111111111111 112 The output should be,

9. Shell Programming and Scripting

Bash arrays: rebin/interpolate smaller array to large array

hello, i need a bit of help on how to do this effectively in bash without a lot of extra looping or massive switch/case i have a long array of M elements and a short array of N elements, so M > N always. M is not a multiple of N. for case 1, I want to stretch N to fit M arrayHuge H = (...

10. UNIX for Beginners Questions & Answers

Split large file into smaller files without disturbing the entry chunks

Dears, Need you help with the below file manipulation. I want to split the file into 8 smaller files but without cutting/disturbing the entries (meaning every small file should start with a entry and end with an empty line). It will be helpful if you can provide a one liner command for this...

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Need help cutting a couple of fields from log file.

Discussion started by: altamaha

2. UNIX for Dummies Questions & Answers

splitting the large file into smaller files

Discussion started by: vsnreddy

3. Shell Programming and Scripting

Help with splitting a large text file into smaller ones

Discussion started by: lord_butler

4. UNIX for Dummies Questions & Answers

multiple smaller files from one large file

Discussion started by: rtroscianecki

5. Shell Programming and Scripting

Split large file into smaller file

Discussion started by: sitaldip

6. Shell Programming and Scripting

Help needed - Split large file into smaller files based on pattern match

Discussion started by: frustrated1

7. Shell Programming and Scripting

Sed: Splitting A large File into smaller files based on recursive Regular Expression match

Discussion started by: sumguy

8. UNIX for Dummies Questions & Answers

Split large file to smaller fastly

Discussion started by: mechvijays

9. Shell Programming and Scripting

Bash arrays: rebin/interpolate smaller array to large array

Discussion started by: f77hack

10. UNIX for Beginners Questions & Answers

Split large file into smaller files without disturbing the entry chunks

Discussion started by: Kamesh G