"How to randomly select lines from a text file"

Post #302720231 by gary_w on Tuesday 23rd of October 2012 04:05:11 PM

For the fun of it here's another way that does not use awk although the awk version will be more efficient. This has the overhead of creating the pipeline repeatedly which should be avoided for good practice. Also I believe the ksh RANDOM built-in has a limit of 32767 that must be considered if the file is large.
Code:
$ cat x
##
## x nbr_of_lines_wanted  filename
##
#!/bin/ksh

iterations=$1
file="$2"

((lines_avail=$(wc -l < "$file")+1))

while (( $iterations > 0 )); do
  head -$((${RANDOM} % $lines_avail)) "$file" | tail -1
  (( iterations=$iterations - 1 ))
done

exit 0

This is actually a good example of how a seemingly simple solution for a small file can end up burning you on performance and system limitations should you need to run it on a much larger file
or a system that may see increased load in the future.
Typically when you see a long command line or pipeline like this being done a large number of times (especially a user-enterable number of times) it should
be a red flag warning that there will most likely be a more efficient way of structuring the program.

Last edited by gary_w; 10-23-2012 at 06:11 PM..
This User Gave Thanks to gary_w For This Post:
 
Test Your Knowledge in Computers #277
Difficulty: Easy
Solaris is a version of UNIX System V Release 4 (SVR4), jointly developed by Sun and AT&T, and was licensed by Sun from DEC to replace SunOS.
True or False?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

how to select a value randomly

on my desktop i am using the kde rotating desktop image option. this rotates images randomly every half hour. now, i would like to write an html file which will have an inline frame with some text, maybe system messages, or my friends live journal thati read alot, or unix.com! however, i dont want... (1 Reply)
Discussion started by: norsk hedensk
1 Replies

2. AIX

Randomly appearing control characters in text files

Hi, From some time, we have noticed that our ascii files have started corrupting due to the presence of some random control characters (^@, ^M, ^H, ^D). The characters appear randomly on any file after the process that creates the file finishes. If we rerun the process, the files re creates... (0 Replies)
Discussion started by: aakashahuja
0 Replies

3. UNIX for Dummies Questions & Answers

Select only certain lines from file and mantain formatting

I want to take the below data, and have it output to file only the STMC#/(IP address) and the "there are X number of updates to install" lines for each machine. I know it's easy, but Im a beginner in BASH stuff, my solution would probably take way too many lines to do something easy.Thanks! ... (5 Replies)
Discussion started by: glev2005
5 Replies

4. Shell Programming and Scripting

Select some lines from a txt file and create a new file with awk

Hi there, I have a text file with several colums separated by "|;#" I need to search the file extracting all columns starting with the value of "1" or "2" saving in a separate file just the first 7 columns of each row maching the criteria, with replacement of the saparators in the nearly created... (4 Replies)
Discussion started by: capnino
4 Replies

5. Shell Programming and Scripting

Select lines in which column have value greater than some percent of total file lines

i have a file in following format 1 32 3 4 6 4 4 45 1 45 4 61 54 66 4 5 65 51 56 65 1 12 32 85 now here the total number of lines are 8(they vary each time) Now i want to select only those lines in which the values... (6 Replies)
Discussion started by: vaibhavkorde
6 Replies

6. Shell Programming and Scripting

randomly shuffle two text files the same way

What I have are two text files that I need to shuffle randomly, but I need the two files to be randomly shuffled the same way. I have heard of shuf but I do not know how to use it for two files. Maybe there is also an easy/simple awk command I do not know about that could handle this problem. ... (3 Replies)
Discussion started by: adrunknarwhal
3 Replies

7. Shell Programming and Scripting

Get 20% of lines in File randomly

Hello, This is my code: nb_lignes=`wc -l $1 | cut -d " " -f1` for i in $(seq $nb_lignes) do m=`head $1 -n $i | tail -1` //command done Please how can i change it to get Get 20% of lines in File randomly to apply "command" on each line ? 20% or 40% or 60 % (it's a parameter) Thank you. (15 Replies)
Discussion started by: chercheur857
15 Replies

8. Shell Programming and Scripting

Select lines from a file based on a criteria

Hi I need to select lines from a txt file, I have got a line starting with ZMIO:MSISDN= and after a few line I have another line starting with 'MOBILE STATION ISDN NUMBER' and another one starting with 'VLR-ADDRESS' I need to copy these three lines as three different columns in a separate... (3 Replies)
Discussion started by: Tlcm sam
3 Replies

9. Shell Programming and Scripting

How to select lines randomly without replacement in UNIX?

Dear Folks I have one column of 15000 lines and want to select randomly 5000 of them in five different times without replacement. I am aware that command 'shuf' and 'sort -R' could select randomly those lines but I am not sure how could I avoid the replacement of selection line. Does anyone have... (10 Replies)
Discussion started by: sajmar
10 Replies

10. Shell Programming and Scripting

Match text to lines in a file, iterate backwards until text or text substring matches, print to file

hi all, trying this using shell/bash with sed/awk/grep I have two files, one containing one column, the other containing multiple columns (comma delimited). file1.txt abc12345 def12345 ghi54321 ... file2.txt abc1,text1,texta abc,text2,textb def123,text3,textc gh,text4,textd... (6 Replies)
Discussion started by: shogun1970
6 Replies
XScreenSaver(1) 					      General Commands Manual						   XScreenSaver(1)

NAME
flag - draws a waving flag, containing text or an image SYNOPSIS
flag [-display host:display.screen] [-foreground color] [-background color] [-window] [-root] [-mono] [-install] [-visual visual] [-ncolors integer] [-delay microseconds] [-cycles integer] [-size integer] [-text string] [-font font] [-bitmap xbm-file] DESCRIPTION
The flag program draws a waving flag that contains text or a bitmap. OPTIONS
flag accepts the following options: -window Draw on a newly-created window. This is the default. -root Draw on the root window. -mono If on a color display, pretend we're on a monochrome display. -install Install a private colormap for the window. -visual visual Specify which visual to use. Legal values are the name of a visual class, or the id number (decimal or hex) of a specific visual. -ncolors integer How many colors should be used (if possible). Default 200. -cycles integer -count integer -size integer How large the pixels in the flag should be, from 1 to 8. If this is a negative number, the pixel size is chosen randomly from the range 1 to -size. Default -7. -text text The text to display in the flag. Multiple lines of text are allowed; the lines will be displayed centered atop one another. Default: none. If the text is the magic string "(default)", then the text used will be the local machine name; a newline; and the local OS version. -bitmap xbm-file The bitmap to display in the flag; this must be an XBM file (color XPMs are not allowed.) Default: none. If the bitmap is the magic string "(default)", then the bitmap used will be a charming little picture of J. R. "Bob" Dobbs. If neither -text nor -bitmap are specified, then either the builtin text or the builtin bitmap will be chosen randomly. -font font The font in which to draw the text; the default is "-*-helvetica-bold-r-*-240-*". ENVIRONMENT
DISPLAY to get the default host and display number. XENVIRONMENT to get the name of a resource file that overrides the global resources stored in the RESOURCE_MANAGER property. SEE ALSO
X(1), xscreensaver(1), xlock(1) COPYRIGHT
Copyright (C) 1996 Charles Vidal. Permission to use, copy, modify, and distribute this software and its documentation for any purpose and without fee is hereby granted, pro- vided that the above copyright notice appear in all copies and that both that copyright notice and this permission notice appear in sup- porting documentation. AUTHOR
Charles Vidal <vidalc@univ-mlv.fr>, 1996. Ability to run standalone or with xscreensaver, and the -text and -bitmap options, added by Jamie Zawinski <jwz@jwz.org>, 24-May-97. X Version 11 24-May-97 XScreenSaver(1)

Featured Tech Videos

All times are GMT -4. The time now is 05:38 AM.
Unix & Linux Forums Content Copyright 1993-2019. All Rights Reserved.
Privacy Policy