Outputting sequences based on length with sed


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Outputting sequences based on length with sed
# 1  
Old 11-23-2018
Outputting sequences based on length with sed

I have this file:
Code:
>ID1
AA
>ID2
TTTTTT
>ID-3
AAAAAAAAA
>ID4
TTTTTTGGAGATCAGTAGCAGATGACAG-GGGGG-TGCACCCC

Add I am trying to use this script to output sequences longer than 15 characters:
Code:
sed -r '/^>/N;{/^.{,15}$/d}'

The desire output would be this:
Code:
>ID4
TTTTTTGGAGATCAGTAGCAGATGACAG-GGGGG-TGCACCCC

I am also trying to output the sequences that are shorter than 15 characters using the following script:
Code:
sed -rn '/^>/N;{/^.{,15}$/p}'

The desire output is:
Code:
>ID1
AA
>ID2
TTTTTT
>ID-3
AAAAAAAAA

How can I modify my scripts so I can generate the desire outputs?
Thanks in advance
# 2  
Old 11-23-2018
Why do you require that only sed be used?

If we were allowed to use awk I would find it much easier (assuming that the input file always contains pairs of lines and the line length you want to check is always just the second line in each set). (If my assumption is wrong, you need to much more clearly state what you are trying to do!)
# 3  
Old 11-23-2018
Quote:
Originally Posted by Xterra
Add I am trying to use this script to output sequences longer than 15 characters:
Code:
sed -r '/^>/N;{/^.{,15}$/d}'

The desire output would be this:
Code:
>ID4
TTTTTTGGAGATCAGTAGCAGATGACAG-GGGGG-TGCACCCC

Let us first examine why your script doesn't do what you want it to do. As a scientist you surely know that finding out why something doesn't work is as well a publishable result as is finding out why it does work. ;-))

Code:
sed -r '/^>/N;{/^.{,15}$/d}'

My first suggestion is to get into the habit of writing sed-scripts line by line and not crammed into one line. It is easier to understand what is done when this way. I will only repeat the script itself, minus the not necessary curly braces:

Code:
/^>/ {
     N
}
/^.{,15}$/ {
     d
}

So there is a rule for lines starting with ">" and it reads the next line. Since every second line starts with ">" we always end up with two lines - the ">"-line and its successor - in the pattern space. Then there is a rule for a pattern space with up to 15 characters and it says: delete.

First, the curly braces in "{,15}" need to be escaped: \{,15\}. Furthermore i am not sure if every (or, more specifically: your) sed-version understands "\{,15\}", maybe it needs to be "\{1,15\}". But this means "one to fifteen"! If you want to test if a string is longer than 15 you need to make that unconditional: "\{15\}".

The meaning of the multiplicators is:

Code:
\{n\}      # exactly n occurrences of the last expression
\{n,\}     # n or more occurrences of the last expression
\{m,n\}    # between m and n occurrences of the last expression
\{,n\}     # at most n occurrences of the last expression (basically the same as \{1,n\})

There is also another logical problem: when the first line starts with ">ID.." then after adding the next line to the patternspace the content is like this:

Code:
>ID1\nAA

Each of these characters ("\n" is a character too here) counts towards the fifteen threshold, in the example the pattern space is 7 characters long, not 2. I am not sure if this is what you want.

Since we "know" that the structure of your input is to have a ">"-line every second line (otherwise your automatic loading of exactly one line would be wrong too) we can drop the (always complicated and tricky) multiline-patternspaces and use the hold space:

- In case of a ">"-line we move it to the hold space and do nothing else.
- In case of a not-">"-line we test it and if it is long we output the hold space contents, then the line.

Remove the comments before trying it:
Code:
/^>/ {           # all lines starting with a ">"
     h                # put it in the hold space
     d                # and delete it (starts over automatically)
}
/^.\{15\}.*$/ {     # all lines with 15 or more characters
     x                # exchange hold space and pattern space (">"-line is in pattern space now)
     p                # print the pattern space
     x                # exchange hold space and pattern space (second line is in pattern space now)
     p
}

Quote:
Originally Posted by Xterra
I am also trying to output the sequences that are shorter than 15 characters using the following script:
You can do that by simply reversing the second rule:

Code:
/^>/ {
     h
     d
}
/^.\{15\}.*$/ ! {
     x
     p
     x
     p
}

I hope this helps.

bakunin

Last edited by bakunin; 11-23-2018 at 09:44 PM..
These 4 Users Gave Thanks to bakunin For This Post:
# 4  
Old 11-23-2018
Note that the following awk code produces both output files you want in one invocation (with pairs with 2nd lines longer than 15 characters in a file name long and pairs with 2nd lines shorter than 16 characters in a file named short):
Code:
awk '
/^>/ {	hdr = $0
	next
}
{	printf("%s\n%s\n", hdr, $0) > (length($0) > 15 ? "long" : "short")
}' file

If you want to try this on a Solaris/SunOS system, change awk to nawk or /usr/xpg4/bin/awk.
These 3 Users Gave Thanks to Don Cragun For This Post:
# 5  
Old 11-24-2018
Code:
grep -x -B1 '^[^>]\{,15\}'
grep -x -B1 '^[^>]\{15,\}'

These 2 Users Gave Thanks to nezabudka For This Post:
# 6  
Old 11-24-2018
Quote:
Originally Posted by nezabudka
Code:
grep -x -B1 '^[^>]\{,15\}'
grep -x -B1 '^[^>]\{15,\}'

Yes, true - but this works only with GNU-grep, not with a standard-(POSIX-) grep because "-B" is not a POSIX-switch.

It is a hard lesson one learns in administrating very diverse large datacenters that in order to avoid "bad surprises" one sticks as close to the standards as possible. Otherwise you have to develop not "a script" but one for AIX this version, one for AIX that version, one completely different for HP-UX, another one for Solaris, several dozens for the various Linuxes....

Not that following the standards rules out bad surprises anyway, mind you, which is another hard lesson....

I hope this helps.

bakunin
These 2 Users Gave Thanks to bakunin For This Post:
# 7  
Old 11-24-2018
Quote:
Originally Posted by bakunin
...
Remove the comments before trying it:
Code:
/^>/ {           # all lines starting with a ">"
     h                # put it in the hold space
     d                # and delete it (starts over automatically)
}
/^.\{15\}.*$/ {     # all lines with 15 or more characters
     x                # exchange hold space and pattern space (">"-line is in pattern space now)
     p                # print the pattern space
     x                # exchange hold space and pattern space (second line is in pattern space now)
     p
}

...

It might be worthwhile to mention that this code should be run with
- the -r flag removed
- the -n flag set
- the -fscriptname option added:


Code:
sed -nfscriptname datafile

This User Gave Thanks to RudiC For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

How to count the length of fasta sequences?

I could calculate the length of entire fasta sequences by following command, awk '/^>/{if (l!="") print l; print; l=0; next}{l+=length($0)}END{print l}' unique.fasta But, I need to calculate the length of a particular fasta sequence specified/listed in another txt file. The results to to be... (14 Replies)
Discussion started by: dineshkumarsrk
14 Replies

2. Shell Programming and Scripting

Print sequences from file2 based on match to, AND in same order as, file1

I have a list of IDs in file1 and a list of sequences in file2. I can print sequences from file2, but I'm asking for help in printing the sequences in the same order as the IDs appear in file1. file1: EN_comp12952_c0_seq3:367-1668 ES_comp17168_c1_seq6:1-864 EN_comp13395_c3_seq14:231-1088... (5 Replies)
Discussion started by: pathunkathunk
5 Replies

3. Shell Programming and Scripting

Eliminating sequences based on Distances

I have to remove sequences from a file based on the distance value. I am attaching the file containing the distances (Distance.xls) The second file looks something like this: Sequences.txt >Sample1 Freq 59 ggatatgatgatgaactggt >Sample1 Freq 54 ggatatgatgttgaactggt >Sample1 Freq 44... (2 Replies)
Discussion started by: Xterra
2 Replies

4. Shell Programming and Scripting

Selecting sequences based on scores

I have two files with thousands of sequences of different lengths. infile1 contains the actual sequences and infile2 the scores for each A, T, G and C in infile1. Something like this: infile1: >HZVJKYI01ECH5R TTGATGTGCCAGCTGCCGTTGGTGTGCCAA >HZVJKYI01AQWJ8 GGATATGATGATGAACTGGTTTGGCACACC... (4 Replies)
Discussion started by: Xterra
4 Replies

5. Shell Programming and Scripting

Extract length wise sequences from fastq file

I have a fastq file from small RNA sequencing with sequence lengths between 15 - 30. I wanted to filter sequence lengths between 21-25 and write to another fastq file. how can i do that? (4 Replies)
Discussion started by: empyrean
4 Replies

6. Shell Programming and Scripting

Extract sequences based on the list

Hi, I have a file with more than 28000 records and it looks like below.. >mm10_refflat_ABCD range=chr1:1234567-2345678 tgtgcacactacacatgactagtacatgactagac....so on >mm10_refflat_BCD range=chr1:3234567-4545678... tgtgcacactacacatgactagtatgtgcacactacacatgactagta . . . . . so on ... (2 Replies)
Discussion started by: Diya123
2 Replies

7. Shell Programming and Scripting

Trimming sequences based on Reference

My file looks something like this Wnat I need is to look for the Reference sequence (">Reference1") and based on the length of that sequence trim all the entries in that file. So, the rersulting file will contain all sequences with the same length, like this Thus, all sequences will keep... (5 Replies)
Discussion started by: Xterra
5 Replies

8. Shell Programming and Scripting

Deleting sequences based on character frequency

This is what I would like to accomplish, I have an input file (file A) that consist of thousands of sequence elements with the same number of characters (length), each headed by a free text header starting with the chevron ‘>' character followed by the ID (all different IDs with different lenghts)... (9 Replies)
Discussion started by: Xterra
9 Replies

9. Shell Programming and Scripting

Trimming sequences based on specific pattern

My files look like this And I need to cut the sequences at the last "A" found in the following 'pattern' -highlighted for easier identification, the pattern is the actual file is not highlighted. The expected result should look like this Thus, all the sequences would end with AGCCCTA... (2 Replies)
Discussion started by: Xterra
2 Replies

10. UNIX for Dummies Questions & Answers

Sed working on lines of small length and not large length

Hi , I have a peculiar case, where my sed command is working on a file which contains lines of small length. sed "s/XYZ:1/XYZ:3/g" abc.txt > xyz.txt when abc.txt contains lines of small length(currently around 80 chars) , this sed command is working fine. when abc.txt contains lines of... (3 Replies)
Discussion started by: thanuman
3 Replies
Login or Register to Ask a Question