Sponsored Content
Top Forums Shell Programming and Scripting Randomly selecting sequences and generating specific output files Post 302786183 by Don Cragun on Wednesday 27th of March 2013 08:45:07 AM
Old 03-27-2013
Quote:
Originally Posted by Scrutinizer
... ... ...
Quote:
Originally Posted by Don Cragun
In you output samples, the lines in each set of output files are all in the order in which they appeared in the input files. Is that a requirement for your output, or is it just a coincidence in the random numbers used for your example?

With truly random numbers, the output could contain more than one copy of some output lines. Is it a requirement that the output lines be unique?
Isn't that a matter of sampling with/without replacement?
Yes. My 2nd question was whether the output is sampling with or without replacement. And, Xterra has responded that each output set is to use sampling without replacement.

My 1st question was whether the random sequences 1, 3, 5 and 1, 5, 3 (and the six other orders of those three values) are to be treated as 8 distinct output sequences or they should all be normalized to the single sequence where the output lines are in the same order as they were in the input files. That question hasn't been answered yet, so I'm assuming they are to be treated as distinct sequences. (I expect to have an awk program with a shell wrapper to set options later today unless someone else comes up with working solution 1st.)

---------- Post updated at 05:45 ---------- Previous update was at 01:24 ----------

I believe this script does what was requested with considerable (although not complete) error checking. I use the Korn shell (and tested this script using it), but this should work with any POSIX conforming shell. If you are using a Solaris/SunOS system, use /usr/xpg4/bin/awk or nawk instead of awk:
Code:
#!/bin/ksh
# SYNOPSIS
#       rso [-l line_count] [-s sequence_count] fileA fileB
# DESCRIPTION
#       The rso utility "r"andomly "s"elects "sequence_count" (default 10)
#       sequences of "line_count" (default 3) corresponding lines from "fileA"
#       and "fileB" and "o"utputs those lines to files with names:
#               Outfile1-X and Outfile2-X
#       X is a sequence number ranging from 1 through "sequence_count".
#       Leading zeros will be added to the sequence number, if needed, to make
#       all output filenames be the same length.  Lines in Outfile1-* will be
#       from "fileA" and the corresponding lines in Outfile2-* will be the
#       corresponding lines from "fileB".
#
#       Even though the output lines selected will be randomly selected, no
#       input line will be output more than once in a given sequence.
#
#       The number of lines in "fileA" and "fileB" must be the same and the
#       number of lines in the files must be greater than or equal to
#       "line_count".

# Set defaults
ec=0    # Set error code (0 -> no error)
lc=3    # Set default line_count
sc=10   # Set default sequence_count
sn=$(basename $0)       # Save script name for diagnostics

# Process command line options
while getopts l:s: opt
do      case $opt in
        (l)     lc="$OPTARG";;
        (s)     sc="$OPTARG";;
        (?)     ec=1;;
        esac
done
shift $((OPTIND - 1))

# Verify # of operands
if [ $# -ne 2 ]
then    printf "%s: 2 operands are required; %d found\n" "$sn" $#
        ec=1
fi

# If we found errors or the awk script detects an error, print a usage message
if [ $ec -ne 0 ] || ! awk -v lc="$lc" -v sc="$sc" -v sn="$sn" '
# Verify that "line_count" and "sequence_count" are positive integer values
BEGIN { if(lc !~ /^[[:digit:]]+$/ || lc < 1) {
                printf("%s: line_count (%s) must be a positive integer\n",
                        sn, lc)
                ec = 1
                exit ec
        }
        if(sc !~ /^[[:digit:]]+$/ || sc < 1) {
                printf("%s: sequence_count (%s) must be a positive integer\n",
                        sn, sc)
                ec = 2
                exit ec
        }
}
# Save input file names for diagnostics
FNR == 1 {
        fn[++fc] = FILENAME
}
# Accumulate and count input lines from both input files
{       if(FNR == NR)   f1[++c1] = $0
        else            f2[++c2] = $0
}
END {   # If we got here due to an earlier detected error, get out now
        if(ec) exit ec
        # Verify that both files contain the same numbe of lines and that
        # line_count <= # of lines in the files
        if(c1 != c2) {
                printf("%s: lines in %s (%d) must equal lines in %s (%d).\n",
                        sn, fn[1], c1, fn[2], c2)
                exit 3
        }
        if(c1 < lc) {
                printf("%s: line_count(%d) must be <= # of lines in files(%d)\n",
                        sn, lc, c1)
                exit 4
        }
        # Produce output sequences
        for(i = 1; i <= sc; i++) {
                # Set output file names
                of1 = sprintf("Outfile1-%0*d", length(sc), i)
                of2 = sprintf("Outfile2-%0*d", length(sc), i)
                # Set random list of line_count line numbers to output for this
                # output sequence
                for(j = 1; j <= lc; j++) {
                        # Find line_count distinct line numbers
                        while((k = int(rand() * c1) + 1) in list) continue
                        list[k]
                }
                # Print corresponding pairs of lines from the input files into
                # the output files and delete the printed line number from the
                # list of selected random numbers.
                for(j in list) {
                        print f1[j] > of1
                        print f2[j] > of2
                        delete list[j]
                }
                # Close the output files created for this sequence
                close(of1)
                close(of2)
        }
}' "$1" "$2" >&2
then    printf "Usage: %s [-l line_count] [-s sequence_count] fileA fileB\n" \
                "$sn" >&2
        exit 1
fi

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Generating files of specific size

I've been working on getting a script to take size, dir name and file name variables from an input file and creating the same dir structure along with the file of specific size. An example of the input file: size/dirname/filename 2100/JAN_06/12345ABC.TCC 2354/FEB_06/24564XYZ.NOS... (2 Replies)
Discussion started by: nxd25
2 Replies

2. UNIX for Dummies Questions & Answers

randomly renaming files

I have a directory of files that look like filename 001.ext, filename 002.ext, etc. I'd like to rename the files with unique random numbered names, so that the original filenames are stripped and the files are given a new, random number name. I'm not super new to UNIX, but I don't often use it for... (2 Replies)
Discussion started by: platz
2 Replies

3. Shell Programming and Scripting

PERL - Selecting specific files based on 'date stamp' values

Hi, I've list of files in a directory, which have date stamp value in their names. ex: abc_data_20071102.csv, abc_data_20091221.csv, abc_data_20100110.csv, abc_data_20100222.csv, abc_data_20080620.csv,... etc., I need to select and process only files, within the given date... (4 Replies)
Discussion started by: ganapati
4 Replies

4. Shell Programming and Scripting

Trimming sequences based on specific pattern

My files look like this And I need to cut the sequences at the last "A" found in the following 'pattern' -highlighted for easier identification, the pattern is the actual file is not highlighted. The expected result should look like this Thus, all the sequences would end with AGCCCTA... (2 Replies)
Discussion started by: Xterra
2 Replies

5. Shell Programming and Scripting

Removing specific sequences from file

My file looks like this But I need to remove the entry with the identifier >Reference1 along with the entire sequence. Thus, I will end up having the following file Thanks in advance! (2 Replies)
Discussion started by: Xterra
2 Replies

6. UNIX for Dummies Questions & Answers

Selecting specific line using awk

Hi, I would like to get the specific line from the file taking specific coloumn as reference. Thanks and Regards (1 Reply)
Discussion started by: kkarthik_kaja
1 Replies

7. Shell Programming and Scripting

Selecting sequences based on scores

I have two files with thousands of sequences of different lengths. infile1 contains the actual sequences and infile2 the scores for each A, T, G and C in infile1. Something like this: infile1: >HZVJKYI01ECH5R TTGATGTGCCAGCTGCCGTTGGTGTGCCAA >HZVJKYI01AQWJ8 GGATATGATGATGAACTGGTTTGGCACACC... (4 Replies)
Discussion started by: Xterra
4 Replies

8. Shell Programming and Scripting

Help with selecting files from "diff" output

I have two directories Dir_A and Dir_A_Arc. Need help with a shell script. The shell script needs to take the path to these two directories as parameters $1 and $2. The script needs to check if any files exist in these directories and if either of the directories are empty then exit... (5 Replies)
Discussion started by: gaurav99
5 Replies

9. UNIX for Beginners Questions & Answers

Split Command Generating Incomplete Output Files

Hello All, May i please know how do i ensure my split command would NOT generate incomplete output files like below, the last lines in each file is missing some columns or last line is complete. split -b 50GB File File_ File_aa |551|70210203|xxxxxxx|12/22/2010 20:44:58|11/01/2010... (1 Reply)
Discussion started by: Ariean
1 Replies

10. UNIX for Beginners Questions & Answers

How to add specific bases at the beginning and ending of all the fasta sequences?

Hi, I have to add 7 bases of specific nucleotide at the beginning and ending of all the fasta sequences of a file. For example, I have a multi fasta file namely test.fasta as given below test.fasta >TalAA18_Xoo_CIAT_NZ_CP033194.1:_2936369-2939570:+1... (1 Reply)
Discussion started by: dineshkumarsrk
1 Replies
All times are GMT -4. The time now is 07:29 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy