How to randomly select lines from a text file Post: 302720475

Sponsored Content

Top Forums UNIX for Dummies Questions & Answers How to randomly select lines from a text file Post 302720475 by drl on Wednesday 24th of October 2012 09:00:25 AM

10-24-2012

Registered User

Hi.

There are a number of commonly-available utilities to do this. Here is a demonstration of two:

Code:

#!/usr/bin/env bash

# @(#) s1	Demonstrate random selection of lines with rl, shuf.

pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }
db() { ( printf " db, ";for _i;do printf "%s" "$_i";done;printf "\n" ) >&2 ; }
db() { : ; }
C=$HOME/bin/context && [ -f $C ] && $C rl shuf

pl " Lines selected from:"
cat data0

# Prepare data file from single line of words.
tr ' ' '\n' < data0 > data1

pl " Results from shuf, 1:"
shuf -n 3 data1

pl " Results from shuf, 2:"
shuf -n 3 data1

pl " Results from rl, 1:"
rl -c 3 data1

pl " Results from rl, 2:"
rl -c 3 data1

exit 0

producing:

Code:

% ./s1

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 2.6.26-2-amd64, x86_64
Distribution        : Debian GNU/Linux 5.0.8 (lenny) 
bash GNU bash 3.2.39
rl 0.2.7
shuf (GNU coreutils) 6.10

-----
 Lines selected from:
foo bar baz qux quux corge grault garble warg fred plugh xyzzy thud

-----
 Results from shuf, 1:
quux
thud
qux

-----
 Results from shuf, 2:
thud
xyzzy
quux

-----
 Results from rl, 1:
corge
qux
grault

-----
 Results from rl, 2:
thud
corge
warg

You may need to install these from your distribution repository. See man pages for details.

See also Algorithm::Numerical::Sample - search.cpan.org if a perl module is desirable.

Best wishes ... cheers, drl

This User Gave Thanks to drl For This Post:

drl

View Public Profile for drl

Find all posts by drl

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

how to select a value randomly

on my desktop i am using the kde rotating desktop image option. this rotates images randomly every half hour. now, i would like to write an html file which will have an inline frame with some text, maybe system messages, or my friends live journal thati read alot, or unix.com! however, i dont want...

2. AIX

Randomly appearing control characters in text files

Hi, From some time, we have noticed that our ascii files have started corrupting due to the presence of some random control characters (^@, ^M, ^H, ^D). The characters appear randomly on any file after the process that creates the file finishes. If we rerun the process, the files re creates...

3. UNIX for Dummies Questions & Answers

Select only certain lines from file and mantain formatting

I want to take the below data, and have it output to file only the STMC#/(IP address) and the "there are X number of updates to install" lines for each machine. I know it's easy, but Im a beginner in BASH stuff, my solution would probably take way too many lines to do something easy.Thanks! ...

4. Shell Programming and Scripting

Select some lines from a txt file and create a new file with awk

Hi there, I have a text file with several colums separated by "|;#" I need to search the file extracting all columns starting with the value of "1" or "2" saving in a separate file just the first 7 columns of each row maching the criteria, with replacement of the saparators in the nearly created...

5. Shell Programming and Scripting

Select lines in which column have value greater than some percent of total file lines

i have a file in following format 1 32 3 4 6 4 4 45 1 45 4 61 54 66 4 5 65 51 56 65 1 12 32 85 now here the total number of lines are 8(they vary each time) Now i want to select only those lines in which the values...

6. Shell Programming and Scripting

randomly shuffle two text files the same way

What I have are two text files that I need to shuffle randomly, but I need the two files to be randomly shuffled the same way. I have heard of shuf but I do not know how to use it for two files. Maybe there is also an easy/simple awk command I do not know about that could handle this problem. ...

7. Shell Programming and Scripting

Get 20% of lines in File randomly

Hello, This is my code: nb_lignes=`wc -l $1 | cut -d " " -f1` for i in $(seq $nb_lignes) do m=`head $1 -n $i | tail -1` //command done Please how can i change it to get Get 20% of lines in File randomly to apply "command" on each line ? 20% or 40% or 60 % (it's a parameter) Thank you.

8. Shell Programming and Scripting

Select lines from a file based on a criteria

Hi I need to select lines from a txt file, I have got a line starting with ZMIO:MSISDN= and after a few line I have another line starting with 'MOBILE STATION ISDN NUMBER' and another one starting with 'VLR-ADDRESS' I need to copy these three lines as three different columns in a separate...

9. Shell Programming and Scripting

How to select lines randomly without replacement in UNIX?

Dear Folks I have one column of 15000 lines and want to select randomly 5000 of them in five different times without replacement. I am aware that command 'shuf' and 'sort -R' could select randomly those lines but I am not sure how could I avoid the replacement of selection line. Does anyone have...

10. Shell Programming and Scripting

Match text to lines in a file, iterate backwards until text or text substring matches, print to file

hi all, trying this using shell/bash with sed/awk/grep I have two files, one containing one column, the other containing multiple columns (comma delimited). file1.txt abc12345 def12345 ghi54321 ... file2.txt abc1,text1,texta abc,text2,textb def123,text3,textc gh,text4,textd...

LEARN ABOUT LINUX

shuf

SHUF(1) 							   User Commands							   SHUF(1)

NAME

       shuf - generate random permutations

SYNOPSIS

       shuf [OPTION]... [FILE]
       shuf -e [OPTION]... [ARG]...
       shuf -i LO-HI [OPTION]...

DESCRIPTION

       Write a random permutation of the input lines to standard output.

       Mandatory arguments to long options are mandatory for short options too.

       -e, --echo
	      treat each ARG as an input line

       -i, --input-range=LO-HI
	      treat each number LO through HI as an input line

       -n, --head-count=COUNT
	      output at most COUNT lines

       -o, --output=FILE
	      write result to FILE instead of standard output

       --random-source=FILE
	      get random bytes from FILE

       -z, --zero-terminated
	      end lines with 0 byte, not newline

       --help display this help and exit

       --version
	      output version information and exit

       With no FILE, or when FILE is -, read standard input.

AUTHOR

       Written by Paul Eggert.

REPORTING BUGS

       Report shuf bugs to bug-coreutils@gnu.org
       GNU coreutils home page: <http://www.gnu.org/software/coreutils/>
       General help using GNU software: <http://www.gnu.org/gethelp/>
       Report shuf translation bugs to <http://translationproject.org/team/>

COPYRIGHT

       Copyright (C) 2010 Free Software Foundation, Inc.  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
       This is free software: you are free to change and redistribute it.  There is NO WARRANTY, to the extent permitted by law.

SEE ALSO

       The  full documentation for shuf is maintained as a Texinfo manual.  If the info and shuf programs are properly installed at your site, the
       command

	      info coreutils 'shuf invocation'

       should give you access to the complete manual.

GNU coreutils 8.5						   February 2011							   SHUF(1)

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

how to select a value randomly

Discussion started by: norsk hedensk

2. AIX

Randomly appearing control characters in text files

Discussion started by: aakashahuja

3. UNIX for Dummies Questions & Answers

Select only certain lines from file and mantain formatting

Discussion started by: glev2005

4. Shell Programming and Scripting

Select some lines from a txt file and create a new file with awk

Discussion started by: capnino