Here is a demonstration of a gathering technique that might be useful here:
Code:
#!/usr/bin/env bash
# @(#) s1 Demonstrate gather of lines, split into groups, expand.
pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }
db() { ( printf " db, ";for _i;do printf "%s" "$_i";done;printf "\n" ) >&2 ; }
db() { : ; }
C=$HOME/bin/context && [ -f $C ] && $C awk tail split
FILE=${1-data1}
pl " Input data file $FILE:"
cat $FILE
# Remove debris.
rm -f xa*
pl " Script \"gather\":"
cat gather
# Bunch lines of each sequence together on a single line.
./gather $FILE |
tail --lines=+2 |
split --lines=2
pl " Files xa created by split:"
ls xa*
# Expand file to temporary, replace original.
for file in xa*
do
awk '
{ gsub(/=/,"\n") ; printf("%s",$0) }
' $file > t1
mv t1 $file
done
pl " Sample: file xab expanded:"
cat xab
exit 0
producing
Code:
% ./s1
Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 2.6.26-2-amd64, x86_64
Distribution : Debian GNU/Linux 5.0.8 (lenny)
bash GNU bash 3.2.39
awk GNU Awk 3.1.5
tail (GNU coreutils) 6.10
split (GNU coreutils) 6.10
-----
Input data file data1:
>seq1
agtcagtc
agtcagtc
ag
>seq2
agtcagtcagtc
agtcagtcagtc
agtcagtc
>seq3
agtcagtcagtcagtc
agtcagtcagtcagtc
agtcagtcagtcagtc
>seq4
agtcagtcagtcagtcagtcagtc
>seq5
agtcagtc
>seq6
agtcagtcagtcagtcagtcagtcagtcagtcagtc
agtcagtcagtcagtcagtcagtcagtcagtcagtc
agtcagtcagtcagtcagtcagtc
-----
Script "gather":
#!/usr/bin/env bash
# @(#) gather Substitution of newline.
FILE=${1-data1}
awk '
BEGIN { RS = ">" }
{ gsub(/\n/,"=") ; printf("%s%s\n", RS,$0) }
' $FILE
exit 0
-----
Files xa created by split:
xaa xab xac
-----
Sample: file xab expanded:
>seq3
agtcagtcagtcagtc
agtcagtcagtcagtc
agtcagtcagtcagtc
>seq4
agtcagtcagtcagtcagtcagtc
The short awk script in file gather collects lines belonging to a sequence into a super line. The newlines are replaced by some character not in the data, here I used "=".
Then the super lines are split into groups of 2.
Another awk script expands the super lines by replacing "=" with a real newline and re-writes the files.
Hi,
I need to split a string, either using awk or cut or basic unix commands (no programming) , with a multibyte charectar as a delimeter.
Ex:
abcd-efgh-ijkl
split by -efgh- to get two segments abcd & ijkl
Is it possible?
Thanks
A.H.S (1 Reply)
I have an excel file with more than 65K records... Since excel does not take more than 65K records i wan to split the file and send it as two excel files... Could some help me how to use the csplit by specifiying the no of records (7 Replies)
I have gone through all the threads in the forum and tested out different things. I am trying to split a 3GB file into multiple files. Some files are even larger than this.
For example:
split -l 3000000 filename.txt
This is very slow and it splits the file with 3 million records in each... (10 Replies)
Hi;
I want to write a shell script that will split a string with no delimiter.
Basically the script will read a line from a file.
For example the line it read from the file contains:
99234523
These values are never the same but the length will always be 8.
How do i split this... (8 Replies)
I’m new to Linux script and not sure how to filter out bad records from huge flat files (over 1.3GB each). The delimiter is a semi colon “;”
Here is the sample of 5 lines in the file:
Name1;phone1;address1;city1;state1;zipcode1
Name2;phone2;address2;city2;state2;zipcode2;comment... (7 Replies)
Hi,
I have a file which has many URLs delimited by space. Now i want them to move to separate files each one holding 10 URLs per file.
http://3276.e-printphoto.co.uk/guardian http://abdera.apache.org/ http://abdera.apache.org/docs/api/index.html
I have used the below code to arrange... (6 Replies)
Hi, all.
I have an input file. I would like to generate 3 types of output files.
Input:
LG10_PM_map_19_LEnd_1000560
LG10_PM_map_6-1_27101856
LG10_PM_map_71_REnd_20597718
LG12_PM_map_5_chr_118419232
LG13_PM_map_121_24341052
LG14_PM_1a_456799
LG1_MM_scf_5a_opt_abc_9029993
... (5 Replies)
Hi,
I have received a file which is 20 GB. We would like to split the file into 4 equal parts and process it to avoid memory issues.
If the record delimiter is unix new line, I could use split command either with option l or b.
The problem is that the line terminator is |##|
How to use... (5 Replies)
I have a large semicolon delimited file with thousands of columns and many thousands of line. It looks like:
ID1;ID2;ID3;ID4;A_1;B_1;C_1;A_2;B_2;C_2;A_3;B_3;C_3
AA;ax;ay;az;01;02;03;04;05;06;07;08;09
BB;bx;by;bz;03;05;33;44;15;26;27;08;09
I want to split this table in to multiple files:
... (1 Reply)
Discussion started by: trymega
1 Replies
LEARN ABOUT PHP
cat
CAT(1) User Commands CAT(1)NAME
cat - concatenate files and print on the standard output
SYNOPSIS
cat [OPTION]... [FILE]...
DESCRIPTION
Concatenate FILE(s) to standard output.
With no FILE, or when FILE is -, read standard input.
-A, --show-all
equivalent to -vET
-b, --number-nonblank
number nonempty output lines, overrides -n
-e equivalent to -vE
-E, --show-ends
display $ at end of each line
-n, --number
number all output lines
-s, --squeeze-blank
suppress repeated empty output lines
-t equivalent to -vT
-T, --show-tabs
display TAB characters as ^I
-u (ignored)
-v, --show-nonprinting
use ^ and M- notation, except for LFD and TAB
--help display this help and exit
--version
output version information and exit
EXAMPLES
cat f - g
Output f's contents, then standard input, then g's contents.
cat Copy standard input to standard output.
AUTHOR
Written by Torbjorn Granlund and Richard M. Stallman.
REPORTING BUGS
GNU coreutils online help: <http://www.gnu.org/software/coreutils/>
Report cat translation bugs to <http://translationproject.org/team/>
COPYRIGHT
Copyright (C) 2017 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.
SEE ALSO tac(1)
Full documentation at: <http://www.gnu.org/software/coreutils/cat>
or available locally via: info '(coreutils) cat invocation'
GNU coreutils 8.28 January 2018 CAT(1)