Sponsored Content
Top Forums Shell Programming and Scripting script to splite large file to number of small files Post 302214821 by ahmed.gad on Tuesday 15th of July 2008 03:27:10 AM
Old 07-15-2008
the number of sections in the input file (large file) are 6,000,000 section, so what i need is splitting this file to 4 or 5 files with maintaining that the section start and end in each file.
Thanks,
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Splitting large file into small files

Hi, I need to split a large file into small files based on a string. At different palces in the large I have the string ^Job. I need to split the file into different files starting from ^Job to the last character before the next ^Job. Also all the small files should be automatically named.... (4 Replies)
Discussion started by: dncs
4 Replies

2. Shell Programming and Scripting

moving large number of files

I have a task to move more than 35000 files every two hours, from the same directory to another directory based on a file that has the list of filenames I tried the following logics (1) find . -name \*.dat > list for i in `cat list` do mv $i test/ done (2) cat list|xargs -i mv "{}"... (7 Replies)
Discussion started by: bryan
7 Replies

3. Shell Programming and Scripting

Split large file and add header and footer to each small files

I have one large file, after every 200 line i have to split the file and the add header and footer to each small file? It is possible to add different header and footer to each file? (7 Replies)
Discussion started by: ashish4422
7 Replies

4. Shell Programming and Scripting

Script to Compare a large number of files.

I have a large Filesystem on an AIX server and another one on a Red Hat box. I have syncd the two filesystems using rsysnc. What Im looking for is a script that would compare to the two filesystems to make sure the bits match up and the number of files match up. its around 2.8 million... (5 Replies)
Discussion started by: zippdawg2001
5 Replies

5. Shell Programming and Scripting

Concatenation of a large number of files

Hellow i have a large number of files that i want to concatenate to one. these files start with the word 'VOICE_' for example VOICE_0000000000 VOICE_1223o23u0 VOICE_934934927349 I use the following code: cat /ODS/prepaid/CDR_FLOW/MEDIATION/VOICE_* >> /ODS/prepaid/CDR_FLOW/WORK/VOICE ... (10 Replies)
Discussion started by: chriss_58
10 Replies

6. UNIX for Dummies Questions & Answers

Delete large number of files

Hi. I need to delete a large number of files listed in a txt file. There are over 90000 files in the list. Some of the directory names and some of the file names do have spaces in them. In the file, each line is a full path to a file: /path/to/the files/file1 /path/to/some other/files/file 2... (4 Replies)
Discussion started by: inakajin
4 Replies

7. Shell Programming and Scripting

Sftp large number of files

Want to sftp large number of files ... approx 150 files will come to server every minute. (AIX box) Also need make sure file has been sftped successfully... Please let me know : 1. What is the best / faster way to transfer files? 2. should I use batch option -b so that connectivity will be... (3 Replies)
Discussion started by: vegasluxor
3 Replies

8. Shell Programming and Scripting

Breaking large file into small files

Dear all, I have huge txt file with the input files for some setup_code. However for running my setup_code, I require txt files with maximum of 1000 input files Please help me in suggesting way to break down this big txt file to small txt file of 1000 entries only. thanks and Greetings, Emily (12 Replies)
Discussion started by: emily
12 Replies

9. Shell Programming and Scripting

Removing large number of temp files

Hi All, I am having a situation now to delete a huge number of temp files created during run times approx. 16700+ files. We have never imagined that we will get this this much big list of files during run time. It worked fine for lesser no of files in the list. But when list is huge we are... (7 Replies)
Discussion started by: mad man
7 Replies

10. UNIX for Beginners Questions & Answers

Split large file into 24 small files on one hour basis

I Have a large file with 24hrs log in the below format.i need to split the large file in to 24 small files on one hour based.i.e ex:from 09:55 to 10:55,10:55-11:55 can any one help me on this.! ... (20 Replies)
Discussion started by: Raghuram717
20 Replies
csplit(1)						      General Commands Manual							 csplit(1)

Name
       csplit - context split

Syntax
       csplit [ -s ] [ -k ] [ -f prefix ] file arg1 [ ...argn ]

Description
       The command reads file and separates it into n+1 sections, as defined by the arguments arg1...argn.  By default, the sections are placed in
       xx00...xxn (n may not be greater than 99).  The named file is sectioned in the following way:

       00:   From the start of file up to (but not including) the line referenced by arg1.

       01:   From the line referenced by arg1 up to the line referenced by arg2.
	     .
	     .
	     .

       n:    From the line referenced by argn to the end of file.

       If the file argument is a minus (-) then standard input is used. A minus is an ASCII octal 055.

Options
       -s		   Suppresses the printing of all character counts.  If the -s option is omitted, the command prints the character  counts
			   for each file created.

       -k		   Leaves  previously  created files intact.  If the -k option is omitted, automatically removes created files if an error
			   occurs.

       -fprefix 	   Names the created files prefix00...prefixn.	The default is xx00...xxn.

       The arguments (arg1...argn) to can be a combination of the following:

	     /rexp/[offset]	 A file is created for the section from the current line up to (but not including) the line containing the regular
				 expression  rexp.   The  current line becomes the line containing rexp.  The optional offset is plus (+) or minus
				 (-) the number of lines.  For example, /Page/-5.

	     %rexp%[offset]	 This argument is the same as /rexp/[offset], except that no file is created for the section.

	     lnno		 A file is created from the current line up to (but not including) lnno.  The current line becomes lnno.

	     {num}		 Repeat argument.  This argument may follow any of the above arguments.  If it follows a rexp argument, that argu-
				 ment  is  applied  num  more times.  If it follows lnno, the file will be split every lnno lines (num times) from
				 that point.

       Enclose all rexp type arguments that contain blanks or other characters meaningful to the Shell in the appropriate quotes.  Regular expres-
       sions should not contain embedded new-lines.  The command does not affect the original file; it is the user's responsibility to remove it.

Examples
       csplit -f cobol file /procedure division/ /par5./ /par16./
       This example creates four files, cobol00...cobol03.  After editing the files that created, they can be recombined as follows:
       cat cobol0[0-3] > file
       Note that this example overwrites the original file.

       csplit -k file  100  {99}
       This  example splits the file every 100 lines, up to 10,000 lines.  The -k option causes the created files to be retained if there are less
       than 10,000 lines; however, an error message would still be printed.

       csplit -k prog.c  '%main(%'  '/^}/+1'  {20}
       Assuming that follows the normal C coding convention of ending routines with a right brace (}) at the beginning of the line,  this  example
       creates a file containing each separate C routine (up to 21) in

Diagnostics
       The diagnostics are self explanatory except for the following:
       arg - out of range
       This message means that the given argument did not reference a line between the current position and the end of the file.

See Also
       ed(1), sh(1)

																	 csplit(1)
All times are GMT -4. The time now is 09:38 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy