Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Split file into multiple files using delimiter

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Split file into multiple files using delimiter
# 1  
Old 04-07-2013
Split file into multiple files using delimiter
Hi,

I have a file which has many URLs delimited by space. Now i want them to move to separate files each one holding 10 URLs per file.

Code:
http://3276.e-printphoto.co.uk/guardian http://abdera.apache.org/ http://abdera.apache.org/docs/api/index.html

I have used the below code to arrange each URLs in a line, but unable to write the output to a file(still URLs exist in the same line)

Code:
tr -s '[[:space:]]' '\n' < urls.txt

urls.txt - Source file which has multiple URLs

O/P
http://3276.e-printphoto.co.uk/guardian
http://abdera.apache.org/
http://abdera.apache.org/docs/api/index.html

Code:
echo `tr -s '[[:space:]]' '\n' < urls.txt` >> newline.txt
##This fails to write the each URL in a single line

Please help.

Regards,
Shunmugavel K
# 2  
Old 04-07-2013
You can achieve your objective if you pipe the output of tr into split.

Regards,
Alister
# 3  
Old 04-07-2013
Quote:
Originally Posted by alister
You can achieve your objective if you pipe the output of tr into split.

Regards,
Alister
I have done it, but same output. It doesn't work

Code:
tr -s '[[:space:]]' '\n' < urls.txt | split -l 1 urls.txt new

# 4  
Old 04-07-2013
You did it incorrectly. split needs to read from standard input (the pipe), so that it can see the result of tr's work, one url per line. Instead of the name of the file with the original data, you need to tell split to read -.

Regards,
Alister
Last edited by alister; 04-07-2013 at 05:39 PM..
This User Gave Thanks to alister For This Post:
vel4ever
# 5  
Old 04-07-2013
Quote:
Originally Posted by vel4ever
. . .
Code:
echo `tr -s '[[:space:]]' '\n' < urls.txt` >> newline.txt
##This fails to write the each URL in a single line

No, it does not. Your (unnecessary!) echo transforms the painfully achieved newlines to spaces. Why at all do you use echo?

Quote:
I have done it, but same output. It doesn't work
Code:
tr -s '[[:space:]]' '\n' < urls.txt | split -l 1 urls.txt new

That's true: it does not work. It can't! Remove the urls.txt new from the end, and split will happily create 3 files with one url in each. Set the l option to 10 for larger files.
This User Gave Thanks to RudiC For This Post:
vel4ever
# 6  
Old 04-07-2013
Here is an awk program that might also help you:
Code:
awk ' BEGIN {
        T = 1
        n = 1
        F = "URL" n
} {
        for ( i = 1; i <= NF; i++ )
        {
                if ( ++c > T )
                {
                        close(F)
                        ++n
                        F = "URL" n
                        c = 1
                }
                print $i > F
        }
} ' urls.txt

Note: Change variable T value as per your requirement depending on how many lines you want to split.
# 7  
Old 04-08-2013
Thanks alister and RodiC. It works as per the changes you have suggested.

---------- Post updated at 03:43 AM ---------- Previous update was at 03:41 AM ----------

Quote:
Originally Posted by Yoda
Here is an awk program that might also help you:
Code:
awk ' BEGIN {
        T = 1
        n = 1
        F = "URL" n
} {
        for ( i = 1; i <= NF; i++ )
        {
                if ( ++c > T )
                {
                        close(F)
                        ++n
                        F = "URL" n
                        c = 1
                }
                print $i > F
        }
} ' urls.txt

Note: Change variable T value as per your requirement depending on how many lines you want to split.
Thanks for the program. I don't have more knowledge on awk, but your program is simple for me to understand. I would love to learn it.
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Shell script to Split matrix file with delimiter into multiple files

I have a large semicolon delimited file with thousands of columns and many thousands of line. It looks like: ID1;ID2;ID3;ID4;A_1;B_1;C_1;A_2;B_2;C_2;A_3;B_3;C_3 AA;ax;ay;az;01;02;03;04;05;06;07;08;09 BB;bx;by;bz;03;05;33;44;15;26;27;08;09 I want to split this table in to multiple files: ... (1 Reply)
Discussion started by: trymega
1 Replies

2. Shell Programming and Scripting

Split files based on row delimiter count

I have a huge file (around 4-5 GB containing 20 million rows) which has text like: <EOFD>11<EOFD>22<EORD>2<EOFD>2222<EOFD>3333<EORD>3<EOFD>44<EOFD>55<EORD>66<EOFD>888<EOFD>9999<EORD> Actually above is an extracted file from a Sql Server with each field delimited by <EOFD> and each row ends... (8 Replies)
Discussion started by: amvip
8 Replies

3. Shell Programming and Scripting

split file into multiple files

Hi, I have a file of the following syntax that has around 120K records that are tab separated. input.txt abc def klm 20 76 . + . klm_mango unix_00000001; abc def klm 83 84 . + . klm_mango unix_0000103; abc def klm 415 439 . + . klm_mango unix_00001043; I am looking for an awk oneliner... (2 Replies)
Discussion started by: jacobs.smith
2 Replies

4. Shell Programming and Scripting

Help- counting delimiter in a huge file and split data into 2 files

I’m new to Linux script and not sure how to filter out bad records from huge flat files (over 1.3GB each). The delimiter is a semi colon “;” Here is the sample of 5 lines in the file: Name1;phone1;address1;city1;state1;zipcode1 Name2;phone2;address2;city2;state2;zipcode2;comment... (7 Replies)
Discussion started by: lv99
7 Replies

5. Shell Programming and Scripting

Split file into multiple files

Hi I have a file that has multiple sequences; the sequence name is the line starting with '>'. It looks like below: infile.txt: >HE_ER tttggtgccttgactcggattgggggacctcccttgggagatcaatcccctgtcctcctgctctttgctc cgtgaaaaggatccacctatgacctctagtcctcagacccaccagcccaaggaacatctcaccaatttca >M7B_Ho_sap... (2 Replies)
Discussion started by: jdhahbi
2 Replies

6. Shell Programming and Scripting

renaming files using split with a delimiter

I have a directory of files that I need to rename by splitting the first and second halves of the filenames using the delimiter "-O" and then renaming with the second half first, followed by two underscores and then the first half. For example, natfinal1995annvol1_14.pdf -O filenum-20639 will be... (2 Replies)
Discussion started by: swimulator
2 Replies

7. Shell Programming and Scripting

Split a file into multiple files

Hi, i have a file like this: 1|2|3|4|5| 1|2|8|4|6| Trailer1||||| 1|2|3| Trailer2||| 3|4|5|6| 3|4|5|7| 3|4|5|8| Trailer2||| I want to generate 3 files out of this based on the trailer record. Trailer record string can be different for each file or it may be same for one or two. No... (24 Replies)
Discussion started by: pparthji
24 Replies

8. UNIX for Dummies Questions & Answers

split a file into multiple files

Hi All, I have a file ABC.txt and I need to split this file on every 250 rows. And the file name should be ABC1.txt , ABC2.txt and so on. I tried with split command split -l 250 <filename> '<filename>' but the file name returned was ABC.txtaa ABC.txtab. Please... (8 Replies)
Discussion started by: kumar66
8 Replies

9. Shell Programming and Scripting

Split a file into multiple files

I have a file ehich has multiple create statements as create abc 123 one two create xyz 456 four five create nnn 666 six four I want to separte each create statement in seperate files (3 Replies)
Discussion started by: glamo_2312
3 Replies
Login or Register to Ask a Question