Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: split file by delimiter with csplit
Top Forums Shell Programming and Scripting split file by delimiter with csplit Post 302680029 by drl on Wednesday 1st of August 2012 06:00:35 AM
Old 08-01-2012
Hi.

Here is a demonstration of a gathering technique that might be useful here:
Code:
#!/usr/bin/env bash

# @(#) s1	Demonstrate gather of lines, split into groups, expand.

pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }
db() { ( printf " db, ";for _i;do printf "%s" "$_i";done;printf "\n" ) >&2 ; }
db() { : ; }
C=$HOME/bin/context && [ -f $C ] && $C awk tail split

FILE=${1-data1}

pl " Input data file $FILE:"
cat $FILE

# Remove debris.
rm -f xa*

pl " Script \"gather\":"
cat gather

# Bunch lines of each sequence together on a single line.
./gather $FILE |
tail --lines=+2 |
split --lines=2

pl " Files xa created by split:"
ls xa*

# Expand file to temporary, replace original.
for file in xa*
do
awk '
	{ gsub(/=/,"\n") ; printf("%s",$0) }
' $file > t1
mv t1 $file
done

pl " Sample: file xab expanded:"
cat xab

exit 0

producing
Code:
% ./s1

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 2.6.26-2-amd64, x86_64
Distribution        : Debian GNU/Linux 5.0.8 (lenny) 
bash GNU bash 3.2.39
awk GNU Awk 3.1.5
tail (GNU coreutils) 6.10
split (GNU coreutils) 6.10

-----
 Input data file data1:
>seq1
agtcagtc
agtcagtc
ag
>seq2
agtcagtcagtc
agtcagtcagtc
agtcagtc
>seq3
agtcagtcagtcagtc
agtcagtcagtcagtc
agtcagtcagtcagtc
>seq4
agtcagtcagtcagtcagtcagtc
>seq5
agtcagtc
>seq6
agtcagtcagtcagtcagtcagtcagtcagtcagtc
agtcagtcagtcagtcagtcagtcagtcagtcagtc
agtcagtcagtcagtcagtcagtc

-----
 Script "gather":
#!/usr/bin/env bash

# @(#) gather	Substitution of newline.

FILE=${1-data1}

awk '
BEGIN	{ RS = ">" }
	{ gsub(/\n/,"=") ; printf("%s%s\n", RS,$0) }
' $FILE

exit 0

-----
 Files xa created by split:
xaa  xab  xac

-----
 Sample: file xab expanded:
>seq3
agtcagtcagtcagtc
agtcagtcagtcagtc
agtcagtcagtcagtc
>seq4
agtcagtcagtcagtcagtcagtc

The short awk script in file gather collects lines belonging to a sequence into a super line. The newlines are replaced by some character not in the data, here I used "=".

Then the super lines are split into groups of 2.

Another awk script expands the super lines by replacing "=" with a real newline and re-writes the files.

Best wishes ... cheers, drl
 

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

split string with multibyte delimiter

Hi, I need to split a string, either using awk or cut or basic unix commands (no programming) , with a multibyte charectar as a delimeter. Ex: abcd-efgh-ijkl split by -efgh- to get two segments abcd & ijkl Is it possible? Thanks A.H.S (1 Reply)
Discussion started by: azmathshaikh
1 Replies

2. UNIX for Dummies Questions & Answers

Split files using Csplit

I have an excel file with more than 65K records... Since excel does not take more than 65K records i wan to split the file and send it as two excel files... Could some help me how to use the csplit by specifiying the no of records (7 Replies)
Discussion started by: savitha
7 Replies

3. UNIX for Dummies Questions & Answers

Split a file with no pattern -- Split, Csplit, Awk

I have gone through all the threads in the forum and tested out different things. I am trying to split a 3GB file into multiple files. Some files are even larger than this. For example: split -l 3000000 filename.txt This is very slow and it splits the file with 3 million records in each... (10 Replies)
Discussion started by: madhunk
10 Replies

4. Shell Programming and Scripting

How to split a string with no delimiter

Hi; I want to write a shell script that will split a string with no delimiter. Basically the script will read a line from a file. For example the line it read from the file contains: 99234523 These values are never the same but the length will always be 8. How do i split this... (8 Replies)
Discussion started by: saint34
8 Replies

5. Shell Programming and Scripting

Help- counting delimiter in a huge file and split data into 2 files

I’m new to Linux script and not sure how to filter out bad records from huge flat files (over 1.3GB each). The delimiter is a semi colon “;” Here is the sample of 5 lines in the file: Name1;phone1;address1;city1;state1;zipcode1 Name2;phone2;address2;city2;state2;zipcode2;comment... (7 Replies)
Discussion started by: lv99
7 Replies

6. Shell Programming and Scripting

Split file into multiple files using delimiter

Hi, I have a file which has many URLs delimited by space. Now i want them to move to separate files each one holding 10 URLs per file. http://3276.e-printphoto.co.uk/guardian http://abdera.apache.org/ http://abdera.apache.org/docs/api/index.html I have used the below code to arrange... (6 Replies)
Discussion started by: vel4ever
6 Replies

7. Shell Programming and Scripting

How to target certain delimiter to split text file?

Hi, all. I have an input file. I would like to generate 3 types of output files. Input: LG10_PM_map_19_LEnd_1000560 LG10_PM_map_6-1_27101856 LG10_PM_map_71_REnd_20597718 LG12_PM_map_5_chr_118419232 LG13_PM_map_121_24341052 LG14_PM_1a_456799 LG1_MM_scf_5a_opt_abc_9029993 ... (5 Replies)
Discussion started by: huiyee1
5 Replies

8. UNIX for Advanced & Expert Users

How to split large file with different record delimiter?

Hi, I have received a file which is 20 GB. We would like to split the file into 4 equal parts and process it to avoid memory issues. If the record delimiter is unix new line, I could use split command either with option l or b. The problem is that the line terminator is |##| How to use... (5 Replies)
Discussion started by: Ravi.K
5 Replies

9. UNIX for Beginners Questions & Answers

Shell script to Split matrix file with delimiter into multiple files

I have a large semicolon delimited file with thousands of columns and many thousands of line. It looks like: ID1;ID2;ID3;ID4;A_1;B_1;C_1;A_2;B_2;C_2;A_3;B_3;C_3 AA;ax;ay;az;01;02;03;04;05;06;07;08;09 BB;bx;by;bz;03;05;33;44;15;26;27;08;09 I want to split this table in to multiple files: ... (1 Reply)
Discussion started by: trymega
1 Replies

LEARN ABOUT LINUX

fmt

FMT(1)								   User Commands							    FMT(1)

NAME

       fmt - simple optimal text formatter

SYNOPSIS

       fmt [-WIDTH] [OPTION]... [FILE]...

DESCRIPTION

       Reformat each paragraph in the FILE(s), writing to standard output.  The option -WIDTH is an abbreviated form of --width=DIGITS.

       Mandatory arguments to long options are mandatory for short options too.

       -c, --crown-margin
	      preserve indentation of first two lines

       -p, --prefix=STRING
	      reformat only lines beginning with STRING, reattaching the prefix to reformatted lines

       -s, --split-only
	      split long lines, but do not refill

       -t, --tagged-paragraph
	      indentation of first line different from second

       -u, --uniform-spacing
	      one space between words, two after sentences

       -w, --width=WIDTH
	      maximum line width (default of 75 columns)

       --help display this help and exit

       --version
	      output version information and exit

       With no FILE, or when FILE is -, read standard input.

AUTHOR

       Written by Ross Paterson.

REPORTING BUGS

       Report fmt bugs to bug-coreutils@gnu.org
       GNU coreutils home page: <http://www.gnu.org/software/coreutils/>
       General help using GNU software: <http://www.gnu.org/gethelp/>
       Report fmt translation bugs to <http://translationproject.org/team/>

COPYRIGHT

       Copyright (C) 2010 Free Software Foundation, Inc.  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
       This is free software: you are free to change and redistribute it.  There is NO WARRANTY, to the extent permitted by law.

SEE ALSO

       The  full  documentation  for fmt is maintained as a Texinfo manual.  If the info and fmt programs are properly installed at your site, the
       command

	      info coreutils 'fmt invocation'

       should give you access to the complete manual.

GNU coreutils 8.5						   February 2011							    FMT(1)
All times are GMT -4. The time now is 02:33 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy