split command


 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users split command
# 1  
Old 01-09-2010
split command

./myapp | split -b 10m -d -a 1 - "myappLog"

here split command is reading the input from the output of myapp and it will write the text in to file where in each file size is 10MB and it will create upto 10 files.

I have observed split is flushing the data for every 4096 bytes. if my application suddenly crashes am not getting the latest messages in to the file.

Can i specify to split command to flush the data immediately once the input text is read?.

Please provide me the solution.

thanks in advance.
# 2  
Old 01-11-2010
The problem is not split. Your app needs to block & process signals, SIGSEGV, for example. When the program is going to bomb it should call fsync or fdatasync on the output stream to split, then re-raise signal it orginally got and let it proceed unblocked.

The same is true for fatal errors the program itself enounters.
# 3  
Old 01-11-2010
well..i know how to handle in signals. let me rewrite my statement.
let's say if the application safely exit if one condition met. Split command flushing data for every 4096 bytes. i want to see the logs when the application exits.

basically i used split command to put the restriction on the size of the out put files. i am calling split command in the infinite loop. this will be useful even if the max generation of the file limit reaches.
# 4  
Old 01-11-2010
The only way split will flush output is to get an EOF on stdin.

To my knowledge there is no way to force sync on a child process, especially when it is a command line utility. You will have to roll your own version of split. Have it look for something in the input stream that forces it to fflush() the output stream.

fflush(stdin) is undefined behavior in standard C. (C89 && C99)
# 5  
Old 01-12-2010
my objective is to put restriction on the size of the out put file.

for example
./myapp >> a.txt

let's say if my application runs for 1 year then the size of a.txt is huge in terms of billions.

I am looking for an approach to put the restriction on the size of the out put file. that's the reason i have gone for split. i wrote a wapper script over split command to run in infnate loop.
# 6  
Old 01-12-2010
When i first seen your post itself, it was interesting to me. I started a small research, but unsuccessful in that because of other reasons. But now got a good understanding, here is it, hope it will be useful for you.


Hope you understand the below code, if not, this is the explanation:
  • A perl application which writes a number every one second, and also handles SIGINT, by flusing and closing stdout, and exiting.

Code:
$| = 1;
$i = 0;

$SIG{'INT'} = 'int_handler';

sub int_handler { 
    print STDERR "signal caught...\n"; 
    
    close(stdout); 
    exit(0); 
}

while (1)  {
    print STDOUT ++$i;
    print STDERR $i;
    sleep (1);
}

I executed this program, like this:
Code:
perl write.pl | split -b 3 -d -a 1 - "myappLog" 
12

As expected, it started writing both in STDERR and also in STDOUT, so after some seconds i had given SIGINT like,
Code:
$ ps ax | grep write
 5314 pts/0    S+     0:00 perl write.pl

$ kill -INT 5314

So the following was the outcome,
Code:
$ perl write.pl | split -b 3 -d -a 1 - "myappLog" 
12345678signal caught...
$

Tested whether everything was given to split, and does split processed ?
Yes.

Code:
$ ls myappLog*
myappLog0  myappLog1  myappLog2
$ head myappLog*
==> myappLog0 <==
123
==> myappLog1 <==
456
==> myappLog2 <==
78

Program wrote only 8 characters, and all are handled by split ( and it does not wait for 4096 as you have mentioned ).

So the final thing, you have to handle the signal and flush the output.
Hope it helps.
# 7  
Old 01-12-2010
I have observed this behaviour on RHAS4.0

---------- Post updated at 12:44 PM ---------- Previous update was at 12:36 PM ----------

I have observed this problem on RHES 4.0.

---------- Post updated at 12:45 PM ---------- Previous update was at 12:44 PM ----------

I have observed this problem on RHES 4.0.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Help with Split Command

Hi All, I have a txt file which I would like to partition into 2 separate output files. I would like to partition the odd or even groups of 4 lines from the txt file. So I would like lines 1-4 to go to file1, and lines 5-8 to go to file2, and so on until the whole txt file is divided into two... (1 Reply)
Discussion started by: landrjos
1 Replies

2. UNIX for Beginners Questions & Answers

Urgent..!!Split command

Hi All, I want to split the file after size gets above 100kb. So I am using below command. split -b 100kb File.txt Test But after first file, my record is breaking as in middle of the record, size of file is getting above 100kb. So after splitting half record is in one file and half... (1 Reply)
Discussion started by: Amey Dixit
1 Replies

3. Shell Programming and Scripting

Want to split awk command

Hi, There is an awk command in script and it is running successfully. I want to split that command in 2 lines. I have tried using '\' but its not working.. Please suggest me the solution. (11 Replies)
Discussion started by: Sanket Dalvi
11 Replies

4. UNIX for Dummies Questions & Answers

Split command

Hi I have a sequence which looks like this # PH01000000 PH01000000G0240 P.he_genemodel_v1.0 CDS 120721 121773 . - . ID=PH01000000G0240.CDS;Parent=PH01000000G0240 PH01000001G0190 P.he_genemodel_v1.0 mRA 136867 137309 . - . ID=PH01000001G0190.mRNA;Parent=PH01000001G0190... (7 Replies)
Discussion started by: siya@
7 Replies

5. UNIX for Dummies Questions & Answers

change split with another command

hi all, i have problem with my script in unix ...i have script with split -d (--numeric-suffixes) in linux its working but in solaris machine the option -d isn't have so how to i can change split -d (this output) will same in output solaris can i change with awk and how do that thx before (2 Replies)
Discussion started by: zvtral
2 Replies

6. UNIX for Dummies Questions & Answers

filenames from split command

Is there an option or a way with the split command to rename the partitioned files with a counter. For example, can the files testaa, testab, testac be renamed to test1, test2, test3 from the split command without explicilty renaming files. Thanks, - CB (3 Replies)
Discussion started by: ChicagoBlues
3 Replies

7. UNIX for Advanced & Expert Users

Split Command in Perl

Hi, I have to split a line of the form 1232423#asdf#124324#54534#dcfg#wert#rrftt#4567 into an array in perl. I am using @fields; @fields=split('#',$line); if($fields eq "1") But this is not working. By using the syntax, the statements in "if" are never executed. Please help.... (9 Replies)
Discussion started by: rochitsharma
9 Replies

8. UNIX for Advanced & Expert Users

Split Command options

HI! All iam using Split command to split a large .txt file in to smaller files, The syntax iam using split -25000 Product.txt iam getting four output files but not in .txt format but in some other format , when i checked the properties the Type of the output files is Type can any... (7 Replies)
Discussion started by: mohdtausifsh
7 Replies

9. Shell Programming and Scripting

Split command

Can anyone tell me what this command will do? split -b$SPLITSIZE - $file1 < $file2 Will it split file1 or file2? Please explain. Malay (1 Reply)
Discussion started by: malaymaru
1 Replies

10. UNIX for Dummies Questions & Answers

Problem in split command

I want to split a file containing millions of records. I am issuing the command split -l 20000 filename which will split the file in 20K records each. It works fine except in some files, data after one particular field is lost( the field with space). Say the record is ... (4 Replies)
Discussion started by: superprogrammer
4 Replies
Login or Register to Ask a Question