Sponsored Content
Top Forums Shell Programming and Scripting Splitting a large file, split command will not do. Post 302382004 by jwillis0720 on Monday 21st of December 2009 06:02:34 PM
Old 12-21-2009
That would work fine if I wanted to split every instance of a line break into a separate file, but I want a certain number of instances to go into a separate file. I call this variable how many and its defined int he command line. This slight modification seems to work.
Code:
BEGIN{i=0; linecount=0}

{

if($0=="")

        {

          ++linecount;

        }

if(linecount != howmany)

        {

        print $0 >> i".test"

        }

        else

        {

        linecount = 0;

        ++i;

        }

}

 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Splitting a large log file

Okay, absolute newbie here... I'm on a Mac trying to split an almost 2 Gig log file on a Unix box into manageable chunks for my web-based log analysis tool. What do I need to do, what programs do I need to do it? All and any help appreciated/needed :-) Cheers (8 Replies)
Discussion started by: simmonet
8 Replies

2. Shell Programming and Scripting

Splitting large file into small files

Hi, I need to split a large file into small files based on a string. At different palces in the large I have the string ^Job. I need to split the file into different files starting from ^Job to the last character before the next ^Job. Also all the small files should be automatically named.... (4 Replies)
Discussion started by: dncs
4 Replies

3. UNIX for Dummies Questions & Answers

splitting the large file into smaller files

hi all im new to this forum..excuse me if anythng wrong. I have a file containing 600 MB data in that. when i do parse the data in perl program im getting out of memory error. so iam planning to split the file into smaller files and process one by one. can any one tell me what is the code... (1 Reply)
Discussion started by: vsnreddy
1 Replies

4. Shell Programming and Scripting

Help with splitting a large text file into smaller ones

Hi Everyone, I am using a centos 5.2 server as an sflow log collector on my network. Currently I am using inmons free sflowtool to collect the packets sent by my switches. I have a bash script running on an infinate loop to stop and start the log collection at set intervals - currently one... (2 Replies)
Discussion started by: lord_butler
2 Replies

5. Shell Programming and Scripting

awk - splitting 1 large file into multiple based on same key records

Hello gurus, I am new to "awk" and trying to break a large file having 4 million records into several output files each having half million but at the same time I want to keep the similar key records in the same output file, not to exist accross the files. e.g. my data is like: Row_Num,... (6 Replies)
Discussion started by: kam66
6 Replies

6. Shell Programming and Scripting

splitting a large text file into paragraphs

Hello all, newbie here. I've searched the forum and found many "how to split a text file" topics but none that are what I'm looking for. I have a large text file (~15 MB) in size. It contains a variable number of "paragraphs" (for lack of a better word) that are each of variable length. A... (3 Replies)
Discussion started by: lupin..the..3rd
3 Replies

7. Shell Programming and Scripting

Problem with splitting large file based on pattern

Hi Experts, I have to split huge file based on the pattern to create smaller files. The pattern which is expected in the file is: Master..... First... second.... second... third.. third... Master... First.. second... third... Master... First... second.. second.. second..... (2 Replies)
Discussion started by: saisanthi
2 Replies

8. Shell Programming and Scripting

Splitting large file and renaming based on field

I am trying to update an older program on a small cluster. It uses individual files to send jobs to each node. However the newer database comes as one large file, containing over 10,000 records. I therefore need to split this file. It looks like this: HMMER3/b NAME 1-cysPrx_C ACC ... (2 Replies)
Discussion started by: fozrun
2 Replies

9. Shell Programming and Scripting

Help with Splitting a Large XML file based on size AND tags

Hi All, This is my first post here. Hoping to share and gain knowledge from this great forum !!!! I've scanned this forum before posting my problem here, but I'm afraid I couldn't find any thread that addresses this exact problem. I'm trying to split a large XML file (with multiple tag... (7 Replies)
Discussion started by: Aviktheory11
7 Replies

10. Shell Programming and Scripting

Splitting a large file as per date

Hi, I need a suggestion for an issue in UNIX file. I have a log file in my system where data is appending everyday and as a consequence the file is increasing heavily everyday. Now I need a logic to split this file daily basis and remove the files more than 15 days. Request you to... (3 Replies)
Discussion started by: bhaski2012
3 Replies
CLIENT-LOCAL.CFG(5)						File Formats Manual					       CLIENT-LOCAL.CFG(5)

NAME
client-local.cfg - Local configuration settings for Xymon clients SYNOPSIS
~xymon/server/etc/client-local.cfg DESCRIPTION
The client-local.cfg file contains settings that are used by each Xymon client when it runs on a monitored host. It provides a convenient way of configuring clients from a central location without having to setup special configuration maintenance tools on all clients. The client-local.cfg file is currently used to configure what logfiles the client should fetch data from, to be used as the basis for the "msgs" status column; and to configure which files and directories are being monitored in the "files" status column. Note that there is a dependency between the client-local.cfg file and the hobbit-clients.cfg(5) file. When monitoring e.g. a logfile, you must first enter it into the client-local.cfg file, to trigger the Xymon client into reporting any data about the logfile. Next, you must configure hobbit-clients.cfg so the Xymon server knows what to look for in the file data sent by the client. So: client-local.cfg defines what raw data is collected by the client, and hobbit-clients.cfg defines how to analyze them. PROPAGATION TO CLIENTS
The client-local.cfg file resides on the Xymon server. When clients connect to the Xymon server to send in their client data, they will receive part of this file back from the Xymon server. The configuration received by the client is then used the next time the client runs. This method of propagating the configuration means that there is a delay of up to two poll cycles (i.e. 5-10 minutes) from a configuration change is entered into the client-local.cfg file, and until you see the result in the status messages reported by the client. FILE FORMAT
The file is divided into sections, delimited by "[name]" lines. A section name can be either an operating system identifier - linux, solaris, hp-ux, aix, freebsd, openbsd, netbsd, darwin - or a hostname. When deciding which section to send to a client, Xymon will first look for a section named after the hostname of the client; if such a section does not exist, it will look for a section named by the oper- ating system of the client. So you can configure special configurations for individual hosts, and have a default configuration for all other hosts of a certain type. Apart from the section delimiter, the file format is free-form, or rather it is defined by the tools that make use of the configuration. LOGFILE CONFIGURATION ENTRIES
A logfile configuration entry looks like this: log:/var/log/messages:10240 ignore MARK trigger Oops The log:FILENAME:SIZE line defines the filename of the log, and the maximum amount of data (in bytes) to send to the Xymon server. FILENAME is usually an explicit full-path filename on the client. If it is enclosed in backticks, it is a command which the Xymon client runs and each line of output from this command is then used as a filename. This allows scripting which files to monitor, e.g. if you have logfiles that are named with some sort of timestamp. The ignore PATTERN line (optional) defines lines in the logfile which are ignored entirely, i.e. they are stripped from the logfile data before sending it to the Xymon server. It is used to remove completely unwanted "noise" entries from the logdata processed by Xymon. "PAT- TERN" is a regular expression. The trigger PATTERN line (optional) is used only when there is more data in the log than the maximum size set in the "log:FILENAME:SIZE" line. The "trigger" pattern is then used to find particularly interesting lines in the logfile - these will always be sent to the Xymon server. After picking out the "trigger" lines, any remaining space up to the maximum size is filled in with the most recent entries from the logfile. "PATTERN" is a regular expression. COUNTING LOGENTRIES
A special type of log-handling is possible, where the number of lines matching a regular expressions are merely counted. This is linecount:FILENAME, followed by a number of lines of the form ID:PATTERN. E.g. linecount:/var/log/messages diskerrors:I/O error.*device.*hd badlogins:Failed login FILE CONFIGURATION ENTRIES
A file monitoring entry is used to watch the meta-data of a file: Owner, group, size, permissions, checksum etc. It looks like this: file:/var/log/messages[:HASH] The file:FILENAME line defines the filename of the file to monitor. As with the "log:" entries, a filename enclosed in backticks means a command which will generate the filenames dynamically. The optional [:HASH] setting defines what type of hash to compute for the file: md5, sha1 or rmd160. By default, no hash is calculated. NOTE: If you want to check multiple files using a wildcard, you must use a command to generate the filenames. Putting wildcards directly into the file: entry will not work. DIRECTORY CONFIGURATION ENTRIES
A directory monitoring entry is used to watch the size of a directory and any sub-directories. It looks like this: dir:DIRECTORYNAME The dir:DIRECTORYNAME line defines the filename of the file to monitor. As with the "log:" entries, a filename enclosed in backticks means a command which will generate the filenames dynamically. The Xymon client will run the du(1) command with the directoryname as parameter, and send the output back to the Xymon server. NOTE: If you want to check multiple directories using a wildcard, you must use a command to generate the directory names. Putting wildcards directly into the dir: entry will not work. E.g. use something like dir:`find /var/log -maxdepth 1 -type d` The "du" command used can be configured through the DU environment variable. On some systems, by default du reports data in disk blocks instead of KB (e.g. Solaris). So you may want to configure the Xymon client to use a du command which reports data in KB, e.g. by setting DU="du -k" in the hobbitclient.cfg file. NOTES
The ability of the Xymon client to calculate file hashes and monitor those can be used for file integrity validation on a small scale. How- ever, there is a significant processing overhead in calculating these every time the Xymon client runs, so this should not be considered a replacement for host-based intrusion detection systems such as Tripwire or AIDE. Use of the directory monitoring on directory structures with a large number of files and/or sub-directories can be quite ressource-inten- sive. SEE ALSO
hobbit-clients.cfg(5), hobbitd_client(8), hobbitd(8), xymon(7) Xymon Version 4.2.3: 4 Feb 2009 CLIENT-LOCAL.CFG(5)
All times are GMT -4. The time now is 12:14 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy