split files into specified number of output files


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers split files into specified number of output files
# 1  
Old 05-13-2008
split files into specified number of output files

Hi everyone,

I have some large text files that I need to split into a specific number of files of equal size. As far as I know (and I don't really know that much Smilie) the split command only lets you specify the number of lines or bytes. The files are all of a different size, so the number of lines differ as well.
I use the following code to see how many lines each output file should be (1/50 of the original document):

y=`wc | awk '{ print $x/50}' | awk -F. '{if($2>=1){print $1+1}else{print $1}}'`
echo $y

The problem is that I can't use this variable $y as input for the split command like this:

split $x -d -l $y split/$x

It simply doesn't work. Does anyone know a way to pass the value of the $y variable into the split command? Or does anyone have a better way of solving the entire problem?

Thanks in advance!
# 2  
Old 05-13-2008
It's not really clear why you can't use the variable. The way it is now, I don't think it contains a useful value (what does the echo print for you?) but fixing the scripts so it works sounds like the best plan unless you can explain why it "simply doesn't work".

Assuming $x is the name of the file you are about to split, try something like this instead.

Code:
y=`wc -l < $x | awk '{ print 1+int($1/50) }'`

I put in the addition of one on the theory that this should do something vaguely useful even with files which are too small, with a slight bias towards making the last file in the series smaller than the others. (Maybe divide by 49 instead to compensate?)
# 3  
Old 05-13-2008
The echo gives me a number, which it's supposed to do.
The addition is used to round up the numbers: 231.2 becomes 232, but 231.0 stays 231.

I tried it again with the $x variable explicitly defined and then it worked. So the problem seems to be $x. (sorry, I didn't get much sleep last night Smilie) This changes my question:
If I want to use my script with the original text file as input (like this: sh test.sh < file.txt), how do I call on the original file inside the script?
I obviously thought that the $x in 'split $x -d -l $y split/$x' would be replaced with 'file.txt', but it's not.

Last edited by Migrainegirl; 05-13-2008 at 08:53 AM..
# 4  
Old 05-13-2008
The positional parameters are in $1 $2 $3 etc but if you are invoking the script with redirection, there is no positional parameter, and no file name, just standard input.

Code:
vnix$ ./myscript file.txt oops   #  inside myscript, $1 is "file.txt" and $2 is "oops"
vnix$ ./myscript   # inside myscript, no file name is known; input is from standard input (terminal)
vnix$ ./myscript <file.txt   # again, inside myscript, no file name is known; input is redirected to come from file.txt


Last edited by era; 05-13-2008 at 09:02 AM.. Reason: Add enlightening (?) example
# 5  
Old 05-13-2008
Thanks!! Smilie
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Automate splitting of files , scp files as each split completes and combine files on target server

i use the split command to split a one terabyte backup file into 10 chunks of 100 GB each. The files are split one after the other. While the files is being split, I will like to scp the files one after the other as soon as the previous one completes, from server A to Server B. Then on server B ,... (2 Replies)
Discussion started by: malaika
2 Replies

2. UNIX for Beginners Questions & Answers

Split Command Generating Incomplete Output Files

Hello All, May i please know how do i ensure my split command would NOT generate incomplete output files like below, the last lines in each file is missing some columns or last line is complete. split -b 50GB File File_ File_aa |551|70210203|xxxxxxx|12/22/2010 20:44:58|11/01/2010... (1 Reply)
Discussion started by: Ariean
1 Replies

3. Shell Programming and Scripting

Split a folder with huge number of files in n folders

We have a folder XYZ with large number of files (>350,000). how can i split the folder and create say 10 of them XYZ1 to XYZ10 with 35,000 files each. (doesnt matter which files go where). (12 Replies)
Discussion started by: AlokKumbhare
12 Replies

4. Shell Programming and Scripting

adding file extensions to split output files

Hello, I've searched this forum and others for a solution to my problem but nothing seems just right, I'm hoping I can get some help (seems like this should be easy, and I apologize if I've missed something on the forum): I have several large .fastq DNA sequence files (~20million reads,... (2 Replies)
Discussion started by: ljk
2 Replies

5. UNIX for Dummies Questions & Answers

Command to split the files based on the number of lines in it

Hello Friends, Can anyone help me for the below requirement. I am having a file called Input.txt. My requirement is first check the count that is wc -l input.txt If the result of the wc -l Input.txt is less than 10 then don't split the Input.txt file. Where as if Input.txt >= 10 the split... (12 Replies)
Discussion started by: malaya kumar
12 Replies

6. Shell Programming and Scripting

split a csv file into specified number of files (not lines)

hi, i really need it ...it's not simple to explain but as it's part of a crontab i can't split the file manually...and the file can change every day so the lines are not a good base. example: how to split 1 csv file in 15 files? thank you very much regards :b: (4 Replies)
Discussion started by: 7stars
4 Replies

7. UNIX for Dummies Questions & Answers

Split single file into n number of files

Hi, I am new to unix. we have a requirement here to split a single file into multiples files based on the number of people available for processing. So i tried my hand at writing some code as below. #!/bin/bash var1=`wc -l $filename` var2=$var1/$splitno split -l $var2 $1 Please help me... (6 Replies)
Discussion started by: quirkguy
6 Replies

8. Shell Programming and Scripting

Script to split files based on number of lines

I am getting a few gzip files into a folder by doing ftp to another server. Once I get them I move them to another location .But before that I need to make sure each gzip is not more than 5000 lines and split it up . The files I get are anywhere from 500 lines to 10000 lines in them and is in gzip... (4 Replies)
Discussion started by: gubbu
4 Replies

9. Shell Programming and Scripting

Split single file into multiple files based on the number in the column

Dear All, I would like to split a file of the following format into multiple files based on the number in the 6th column (numbers 1, 2, 3...): ATOM 1 N GLY A 1 -3.198 27.537 -5.958 1.00 0.00 N ATOM 2 CA GLY A 1 -2.199 28.399 -6.617 1.00 0.00 ... (3 Replies)
Discussion started by: tomasl
3 Replies

10. UNIX for Dummies Questions & Answers

split a file into a specified number of files

I have been googling on the 'split' unix command to see if it can split a large file into 'n' number of files. Can anyone spare an example or a code snippet? Thanks, - CB (2 Replies)
Discussion started by: ChicagoBlues
2 Replies
Login or Register to Ask a Question