Sponsored Content
Top Forums Shell Programming and Scripting Randomly selecting sequences and generating specific output files Post 302786141 by Scrutinizer on Wednesday 27th of March 2013 02:22:17 AM
Old 03-27-2013
Here is a start in awk you could try. The first file is read twice solely to determine the number of records..

Code:
awk -v s=3 -v iter=1 '
  NR==FNR { 
    next 
  }
  FNR==1 {
    if(!set) {
      srand();
      n=NR-1
      for(i=1; i<=s; i++) {
        line=0
        while(!line || line in A) line=int(rand()*n)+1
        A[line]
      }
      set=1
    }
    close(f)
    f=FILENAME ".out" iter
  } 
  FNR in A {
    print > f
  }
' infile1 infile1 infile2

You could embed it in a shell loop that increases the iter variable ( -v iter="$loopvar" )


--

Quote:
Originally Posted by Don Cragun
In you output samples, the lines in each set of output files are all in the order in which they appeared in the input files. Is that a requirement for your output, or is it just a coincidence in the random numbers used for your example?

With truly random numbers, the output could contain more than one copy of some output lines. Is it a requirement that the output lines be unique?
Isn't that a matter of sampling with/without replacement?

Last edited by Scrutinizer; 03-28-2013 at 11:13 AM.. Reason: Added parentheses to rand and stand, to make it work in gawk
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Generating files of specific size

I've been working on getting a script to take size, dir name and file name variables from an input file and creating the same dir structure along with the file of specific size. An example of the input file: size/dirname/filename 2100/JAN_06/12345ABC.TCC 2354/FEB_06/24564XYZ.NOS... (2 Replies)
Discussion started by: nxd25
2 Replies

2. UNIX for Dummies Questions & Answers

randomly renaming files

I have a directory of files that look like filename 001.ext, filename 002.ext, etc. I'd like to rename the files with unique random numbered names, so that the original filenames are stripped and the files are given a new, random number name. I'm not super new to UNIX, but I don't often use it for... (2 Replies)
Discussion started by: platz
2 Replies

3. Shell Programming and Scripting

PERL - Selecting specific files based on 'date stamp' values

Hi, I've list of files in a directory, which have date stamp value in their names. ex: abc_data_20071102.csv, abc_data_20091221.csv, abc_data_20100110.csv, abc_data_20100222.csv, abc_data_20080620.csv,... etc., I need to select and process only files, within the given date... (4 Replies)
Discussion started by: ganapati
4 Replies

4. Shell Programming and Scripting

Trimming sequences based on specific pattern

My files look like this And I need to cut the sequences at the last "A" found in the following 'pattern' -highlighted for easier identification, the pattern is the actual file is not highlighted. The expected result should look like this Thus, all the sequences would end with AGCCCTA... (2 Replies)
Discussion started by: Xterra
2 Replies

5. Shell Programming and Scripting

Removing specific sequences from file

My file looks like this But I need to remove the entry with the identifier >Reference1 along with the entire sequence. Thus, I will end up having the following file Thanks in advance! (2 Replies)
Discussion started by: Xterra
2 Replies

6. UNIX for Dummies Questions & Answers

Selecting specific line using awk

Hi, I would like to get the specific line from the file taking specific coloumn as reference. Thanks and Regards (1 Reply)
Discussion started by: kkarthik_kaja
1 Replies

7. Shell Programming and Scripting

Selecting sequences based on scores

I have two files with thousands of sequences of different lengths. infile1 contains the actual sequences and infile2 the scores for each A, T, G and C in infile1. Something like this: infile1: >HZVJKYI01ECH5R TTGATGTGCCAGCTGCCGTTGGTGTGCCAA >HZVJKYI01AQWJ8 GGATATGATGATGAACTGGTTTGGCACACC... (4 Replies)
Discussion started by: Xterra
4 Replies

8. Shell Programming and Scripting

Help with selecting files from "diff" output

I have two directories Dir_A and Dir_A_Arc. Need help with a shell script. The shell script needs to take the path to these two directories as parameters $1 and $2. The script needs to check if any files exist in these directories and if either of the directories are empty then exit... (5 Replies)
Discussion started by: gaurav99
5 Replies

9. UNIX for Beginners Questions & Answers

Split Command Generating Incomplete Output Files

Hello All, May i please know how do i ensure my split command would NOT generate incomplete output files like below, the last lines in each file is missing some columns or last line is complete. split -b 50GB File File_ File_aa |551|70210203|xxxxxxx|12/22/2010 20:44:58|11/01/2010... (1 Reply)
Discussion started by: Ariean
1 Replies

10. UNIX for Beginners Questions & Answers

How to add specific bases at the beginning and ending of all the fasta sequences?

Hi, I have to add 7 bases of specific nucleotide at the beginning and ending of all the fasta sequences of a file. For example, I have a multi fasta file namely test.fasta as given below test.fasta >TalAA18_Xoo_CIAT_NZ_CP033194.1:_2936369-2939570:+1... (1 Reply)
Discussion started by: dineshkumarsrk
1 Replies
slack(8)						      System Manager's Manual							  slack(8)

NAME
slack - Sysadmin's lazy autoconfiguration kit SYNOPSIS
slack [option ...] [role ...] DESCRIPTION
slack is a master command which coordinates the activities of its backends, which variously: o determine the list of roles to be installed on this server o create a local cached copy of the role files from the central repository o merge file trees from subroles into a single, unified tree o install files onto the local filesystem o run scripts before and after installation Options you give to slack will be generally passed along to the backends where relevant. OPTIONS
-h, --help Print a usage statement. --version Print the version and exit. -v, --verbose Increase verbosity. Can be specified multiple times. --quiet Don't be verbose (Overrides previous uses of --verbose). -C, --config FILE Use the specfied FILE for configuration instead of the default, /etc/slack.conf. -s, --source DIR Source directory for slack files -e, --rsh COMMAND Remote shell for rsync -c, --cache DIR Local cache directory for slack files -t, --stage DIR Local staging directory for slack files -r, --root DIR Root destination for slack files --no-sync Skip the slack-sync step (useful if you're pushing stuff into the CACHE outside slack). --no-files Don't install any files in ROOT, but tell rsync to print what it would do. --no-scripts Don't run scripts -n, --dry-run Same as --no-files --no-scripts (CACHE, STAGE will still be updated) --role-list Role list for slack-getroles(8). -b, --backup Make backups of existing files in ROOT that are overwritten. This option defaults to on if it is not set to 0 in a config file or disabled with --nobackup on the command line. --backup-dir Put backups from the --backup option into this directory. -H, --hostname HOST Pretend to be running on HOST, instead of the name given by gethostname(2). --preview MODE Do a diff of scripts and files before running them. MODE can be one of 'simple' or 'prompt' (See PREVIEW MODES, below). --diff PROG Use this diff program for previews. --sleep TIME Randomly sleep between 1 and TIME seconds before starting operations. Useful in crontabs. PREVIEW MODES
Preview functionality is new in slack 0.14.0. I haven't quite worked out how things will work, so this usage is somewhat subject to change in future versions. I thought I would try it this way and see how people like it. In 'simple' mode, after syncing and staging the files directory, slack will present a diff of the files and scripts. In this mode, slack will not run the preinstall or fixfiles scripts, and because of this, it may provide some false output about permissions changes to files. In 'prompt' mode, after syncing and staging the files directory, slack will diff the script directory. If there are differences, slack will present them to you and ask you if you want to continue. If you say no, it will exit. If you say yes, it will stage the scripts directory, run the preinstall and fixfiles scripts, and then diff the files in the stage with those in the root. If there are differences, slack will present them to you and ask you if you want to continue. If you say no, it will exit. If you say yes, it will install the files and run the postinstall script. So, the 'simple' mode is easy to use, and will be accurate if you don't use fixfiles. The 'prompt' mode will be accurate if you use fix- files, but requires some interaction. Why can't we just have one mode that works with fixfiles and requires no interaction? Well, that would require slack to understand what your free-form fixfiles executable was going to do, which would either require some kind of universe simulator or would require you to write your fixfiles in a less free-form way, which would make slack less like slack. EXAMPLES
To install all the roles configured in the role list for a server: slack To install a specific role: slack rolename To test a new role before checking in the changes: slack --source user@workstation:/home/user/.../slack rolename To avoid killing your master server when calling from cron: slack --sleep 3600 FILES
/etc/slack.conf SEE ALSO
slack.conf(5), rsync(1) Administrative commands 2004-10-22 slack(8)
All times are GMT -4. The time now is 03:04 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy