Randomly selecting sequences and generating specific output files | Unix Linux Forums | Shell Programming and Scripting

  Go Back    


Shell Programming and Scripting Post questions about KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and OTHER shell scripts and shell scripting languages here.

Randomly selecting sequences and generating specific output files

Shell Programming and Scripting


Closed Thread    
 
Thread Tools Search this Thread Display Modes
    #1  
Old 03-26-2013
Xterra Xterra is offline
Registered User
 
Join Date: Jun 2010
Last Activity: 18 March 2014, 9:30 AM EDT
Posts: 214
Thanks: 66
Thanked 0 Times in 0 Posts
Randomly selecting sequences and generating specific output files

I have two files containing hundreds of different sequences with the same Identifiers (ID-001, ID-002, etc.,), something like this:
Infile1:

Code:
ID-001 ATGGGAGCGGGGGCGTCTGCCTTGAGGGGAGAGAAGCTAGATACA 
ID-002 ATGGGAGCGGGGGCGTCTGTTTTGAGGGGAGAGAAGCTAGATACA 
ID-003 ATGGGAGCGAAGGCGTCTGTTTTGAGGGGAGAGAAGCTAGATACA 
ID-004 ATTTGAGCGGGGGCGTCTGTTTTGAGGGGAGAGAAGCTAGATACA 
ID-005 ATGGGAGCGGGGGCGTCTGTTTTGAGGGGAGAGAAGCTAGATAGT

Infile2:

Code:
ID-001 ATGGGAGCGGGGGCGTCTGCCTTGAGGGG 
ID-002 ATGGGAGCGGGGGCGTCTGTTTTGAGGGG 
ID-003 ATGGGAGCGAAGGCGTCTGTTTTGAGGGG 
ID-004 ATTTGAGCGGGGGCGTCTGTTTTGAGGGG 
ID-005 ATGGGAGCGGGGGCGTCTGTTTTGAGGGG

I need an awk script that can randomly select a number of sequences, let say 3, with the same IDs from both input files and generate two different outfiles, something like this:
Outfile1-1:

Code:
ID-001 ATGGGAGCGGGGGCGTCTGCCTTGAGGGGAGAGAAGCTAGATACA 
ID-003 ATGGGAGCGAAGGCGTCTGTTTTGAGGGGAGAGAAGCTAGATACA 
ID-005 ATGGGAGCGGGGGCGTCTGTTTTGAGGGGAGAGAAGCTAGATAGT

Oufile2-1:

Code:
ID-001 ATGGGAGCGGGGGCGTCTGCCTTGAGGGG 
ID-003 ATGGGAGCGAAGGCGTCTGTTTTGAGGGG 
ID-005 ATGGGAGCGGGGGCGTCTGTTTTGAGGGG

I need to repeat this process a number of times, let say 10 times, and generate the corresponding Outfiles (Outfile1-2 and Outfile2-2; Outfile1-3 and Outfile2-3; Outfile1-4 and Outfile2-4; etc.,). Thus, the second pair of files will look like this:
Outfile1-2:

Code:
ID-003 ATGGGAGCGAAGGCGTCTGTTTTGAGGGGAGAGAAGCTAGATACA 
ID-004 ATTTGAGCGGGGGCGTCTGTTTTGAGGGGAGAGAAGCTAGATACA 
ID-005 ATGGGAGCGGGGGCGTCTGTTTTGAGGGGAGAGAAGCTAGATAGT

Outfile2-2:

Code:
ID-003 ATGGGAGCGAAGGCGTCTGTTTTGAGGGG 
ID-004 ATTTGAGCGGGGGCGTCTGTTTTGAGGGG 
ID-005 ATGGGAGCGGGGGCGTCTGTTTTGAGGGG

I really do not know how to do the randomization, and therefore, any help will be greatly appreciated.

Last edited by Scrutinizer; 03-27-2013 at 01:37 AM.. Reason: code tags instead of quote tags and format removal
Sponsored Links
    #2  
Old 03-26-2013
Don Cragun's Avatar
Don Cragun Don Cragun is offline Forum Staff  
Moderator
 
Join Date: Jul 2012
Last Activity: 19 April 2014, 12:23 AM EDT
Location: San Jose, CA, USA
Posts: 3,467
Thanks: 141
Thanked 1,197 Times in 1,015 Posts
In you output samples, the lines in each set of output files are all in the order in which they appeared in the input files. Is that a requirement for your output, or is it just a coincidence in the random numbers used for your example?

With truly random numbers, the output could contain more than one copy of some output lines. Is it a requirement that the output lines be unique?

Last edited by Don Cragun; 03-26-2013 at 07:35 PM.. Reason: added another question
Sponsored Links
    #3  
Old 03-26-2013
Xterra Xterra is offline
Registered User
 
Join Date: Jun 2010
Last Activity: 18 March 2014, 9:30 AM EDT
Posts: 214
Thanks: 66
Thanked 0 Times in 0 Posts
Don,
The sequences should be unique.
Thanks,
X
    #4  
Old 03-27-2013
hanson44 hanson44 is offline
Registered User
 
Join Date: Mar 2013
Last Activity: 12 May 2013, 11:33 PM EDT
Posts: 858
Thanks: 18
Thanked 180 Times in 177 Posts
Is your heart set on awk? Would you consider a shell script?
Sponsored Links
    #5  
Old 03-27-2013
Scrutinizer's Avatar
Scrutinizer Scrutinizer is online now Forum Staff  
Moderator
 
Join Date: Nov 2008
Last Activity: 19 April 2014, 5:12 AM EDT
Location: Amsterdam
Posts: 8,874
Thanks: 230
Thanked 2,141 Times in 1,922 Posts
Here is a start in awk you could try. The first file is read twice solely to determine the number of records..


Code:
awk -v s=3 -v iter=1 '
  NR==FNR { 
    next 
  }
  FNR==1 {
    if(!set) {
      srand();
      n=NR-1
      for(i=1; i<=s; i++) {
        line=0
        while(!line || line in A) line=int(rand()*n)+1
        A[line]
      }
      set=1
    }
    close(f)
    f=FILENAME ".out" iter
  } 
  FNR in A {
    print > f
  }
' infile1 infile1 infile2

You could embed it in a shell loop that increases the iter variable ( -v iter="$loopvar" )


--

Quote:
Originally Posted by Don Cragun View Post
In you output samples, the lines in each set of output files are all in the order in which they appeared in the input files. Is that a requirement for your output, or is it just a coincidence in the random numbers used for your example?

With truly random numbers, the output could contain more than one copy of some output lines. Is it a requirement that the output lines be unique?
Isn't that a matter of sampling with/without replacement?

Last edited by Scrutinizer; 03-28-2013 at 10:13 AM.. Reason: Added parentheses to rand and stand, to make it work in gawk
Sponsored Links
    #6  
Old 03-27-2013
Don Cragun's Avatar
Don Cragun Don Cragun is offline Forum Staff  
Moderator
 
Join Date: Jul 2012
Last Activity: 19 April 2014, 12:23 AM EDT
Location: San Jose, CA, USA
Posts: 3,467
Thanks: 141
Thanked 1,197 Times in 1,015 Posts
Quote:
Originally Posted by Scrutinizer View Post
... ... ...
Quote:
Originally Posted by Don Cragun
In you output samples, the lines in each set of output files are all in the order in which they appeared in the input files. Is that a requirement for your output, or is it just a coincidence in the random numbers used for your example?

With truly random numbers, the output could contain more than one copy of some output lines. Is it a requirement that the output lines be unique?
Isn't that a matter of sampling with/without replacement?
Yes. My 2nd question was whether the output is sampling with or without replacement. And, Xterra has responded that each output set is to use sampling without replacement.

My 1st question was whether the random sequences 1, 3, 5 and 1, 5, 3 (and the six other orders of those three values) are to be treated as 8 distinct output sequences or they should all be normalized to the single sequence where the output lines are in the same order as they were in the input files. That question hasn't been answered yet, so I'm assuming they are to be treated as distinct sequences. (I expect to have an awk program with a shell wrapper to set options later today unless someone else comes up with working solution 1st.)

---------- Post updated at 05:45 ---------- Previous update was at 01:24 ----------

I believe this script does what was requested with considerable (although not complete) error checking. I use the Korn shell (and tested this script using it), but this should work with any POSIX conforming shell. If you are using a Solaris/SunOS system, use /usr/xpg4/bin/awk or nawk instead of awk :

Code:
#!/bin/ksh
# SYNOPSIS
#       rso [-l line_count] [-s sequence_count] fileA fileB
# DESCRIPTION
#       The rso utility "r"andomly "s"elects "sequence_count" (default 10)
#       sequences of "line_count" (default 3) corresponding lines from "fileA"
#       and "fileB" and "o"utputs those lines to files with names:
#               Outfile1-X and Outfile2-X
#       X is a sequence number ranging from 1 through "sequence_count".
#       Leading zeros will be added to the sequence number, if needed, to make
#       all output filenames be the same length.  Lines in Outfile1-* will be
#       from "fileA" and the corresponding lines in Outfile2-* will be the
#       corresponding lines from "fileB".
#
#       Even though the output lines selected will be randomly selected, no
#       input line will be output more than once in a given sequence.
#
#       The number of lines in "fileA" and "fileB" must be the same and the
#       number of lines in the files must be greater than or equal to
#       "line_count".

# Set defaults
ec=0    # Set error code (0 -> no error)
lc=3    # Set default line_count
sc=10   # Set default sequence_count
sn=$(basename $0)       # Save script name for diagnostics

# Process command line options
while getopts l:s: opt
do      case $opt in
        (l)     lc="$OPTARG";;
        (s)     sc="$OPTARG";;
        (?)     ec=1;;
        esac
done
shift $((OPTIND - 1))

# Verify # of operands
if [ $# -ne 2 ]
then    printf "%s: 2 operands are required; %d found\n" "$sn" $#
        ec=1
fi

# If we found errors or the awk script detects an error, print a usage message
if [ $ec -ne 0 ] || ! awk -v lc="$lc" -v sc="$sc" -v sn="$sn" '
# Verify that "line_count" and "sequence_count" are positive integer values
BEGIN { if(lc !~ /^[[:digit:]]+$/ || lc < 1) {
                printf("%s: line_count (%s) must be a positive integer\n",
                        sn, lc)
                ec = 1
                exit ec
        }
        if(sc !~ /^[[:digit:]]+$/ || sc < 1) {
                printf("%s: sequence_count (%s) must be a positive integer\n",
                        sn, sc)
                ec = 2
                exit ec
        }
}
# Save input file names for diagnostics
FNR == 1 {
        fn[++fc] = FILENAME
}
# Accumulate and count input lines from both input files
{       if(FNR == NR)   f1[++c1] = $0
        else            f2[++c2] = $0
}
END {   # If we got here due to an earlier detected error, get out now
        if(ec) exit ec
        # Verify that both files contain the same numbe of lines and that
        # line_count <= # of lines in the files
        if(c1 != c2) {
                printf("%s: lines in %s (%d) must equal lines in %s (%d).\n",
                        sn, fn[1], c1, fn[2], c2)
                exit 3
        }
        if(c1 < lc) {
                printf("%s: line_count(%d) must be <= # of lines in files(%d)\n",
                        sn, lc, c1)
                exit 4
        }
        # Produce output sequences
        for(i = 1; i <= sc; i++) {
                # Set output file names
                of1 = sprintf("Outfile1-%0*d", length(sc), i)
                of2 = sprintf("Outfile2-%0*d", length(sc), i)
                # Set random list of line_count line numbers to output for this
                # output sequence
                for(j = 1; j <= lc; j++) {
                        # Find line_count distinct line numbers
                        while((k = int(rand() * c1) + 1) in list) continue
                        list[k]
                }
                # Print corresponding pairs of lines from the input files into
                # the output files and delete the printed line number from the
                # list of selected random numbers.
                for(j in list) {
                        print f1[j] > of1
                        print f2[j] > of2
                        delete list[j]
                }
                # Close the output files created for this sequence
                close(of1)
                close(of2)
        }
}' "$1" "$2" >&2
then    printf "Usage: %s [-l line_count] [-s sequence_count] fileA fileB\n" \
                "$sn" >&2
        exit 1
fi

Sponsored Links
    #7  
Old 03-27-2013
Xterra Xterra is offline
Registered User
 
Join Date: Jun 2010
Last Activity: 18 March 2014, 9:30 AM EDT
Posts: 214
Thanks: 66
Thanked 0 Times in 0 Posts
Scrutinizer,
I am afraid I do not understand your script. I get some error when trying to run the script.
Am I missing something?
X

Code:
awk -v s=3 -v iter=1 '
  NR==FNR { 
    next 
  }
  FNR==1 {
    if(!set) {
      srand;
      n=NR-1
      for(i=1; i<=s; i++) {
        line=0
        while(!line || line in A) line=int(rand*n)+1
        A[line]
      }
      set=1
    }
    close(f)
    f=FILENAME ".out" iter
  } 
  FNR in A {
    print > f
  }
' infile1.txt infile1.txt infile2.txt
awk: cmd. line:7: srand;
awk: cmd. line:7: ^ syntax error
awk: cmd. line:11: while(!line || line in A) line=int(rand*n)+1
awk: cmd. line:11: ^ syntax error
awk: cmd. line:11: while(!line || line in A) line=int(rand*n)+1
awk: cmd. line:11:

---------- Post updated at 05:25 PM ---------- Previous update was at 05:24 PM ----------

Hanson44,
Sheel scripting will be ok too.
Thanks!
X
Sponsored Links
Closed Thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Removing specific sequences from file Xterra Shell Programming and Scripting 2 06-24-2010 04:12 PM
Trimming sequences based on specific pattern Xterra Shell Programming and Scripting 2 06-23-2010 06:49 PM
PERL - Selecting specific files based on 'date stamp' values ganapati Shell Programming and Scripting 4 02-26-2010 01:07 AM
randomly renaming files platz UNIX for Dummies Questions & Answers 2 05-12-2009 01:52 AM
Generating files of specific size nxd25 Shell Programming and Scripting 2 06-27-2006 11:06 AM



All times are GMT -4. The time now is 05:25 AM.