Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Quick way to select many records from a large file
Top Forums Shell Programming and Scripting Quick way to select many records from a large file Post 302942338 by sea on Monday 27th of April 2015 07:22:23 PM
Old 04-27-2015
Hello and welcome to the forum.

Please use code tags for code block examples, as required by the forum rules.

Not sure on the script, as i dont know csh or ksh.
In bash i'd do:
Code:
while read line;do
   awk -v key="$line" {/^key/} {print $0} records.txt
done<key.txt

Then execute it:
Code:
$SHELL pick_records.s > output.txt

Note that i find "pick_" quite irritating, since you dont pick (as in: select a single entry) anything of something, but run through everything (every line) found.

Half a million (or just 200'000) lines/entries do need their time to be done.
You could figure out a difference by adding time in front of executing the script.
Code:
time $SHELL pick-records.sh

Hope this helps (hth)
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Using a variable to select records with awk

As part of a bigger task, I had to read thru a file and separate records into various batches based on a field. Specifically, separate records based on the value in the batch field as defined below. The batch field left-justified numbers. The datafile is here > cat infile 12345 1 John Smith ... (5 Replies)
Discussion started by: joeyg
5 Replies

2. UNIX for Dummies Questions & Answers

Quick question about finding a large file

what is the correct command for finding the largest file and displaying it without any error information? I can find it, but how do I display it in the same command? (6 Replies)
Discussion started by: raidkridley
6 Replies

3. Shell Programming and Scripting

Extract data from large file 80+ million records

Hello, I have got one file with more than 120+ million records(35 GB in size). I have to extract some relevant data from file based on some parameter and generate other output file. What will be the besat and fastest way to extract the ne file. sample file format :--... (2 Replies)
Discussion started by: learner16s
2 Replies

4. Shell Programming and Scripting

select records from one file based on a second file

Hi all: I have two files: file1: 74 DS 9871 199009871 1 1990 4 1 165200 Sc pr de te sa ox 1.0 1.0 13.0000 35.7560 5.950 3.0 3.0 13.0100 35.7550 5.970 ** 74 DS 99004 74DS99004 6738 1990 4 1 165200 Eb pr de te sa ox 1.0 1.0 13.0000 ... (7 Replies)
Discussion started by: rleal
7 Replies

5. Shell Programming and Scripting

How to Pick Random records from a large file

Hi, I have a huge file say with 2000000 records. The file has 42 fields. I would like to pick randomly 1000 records from this huge file. Can anyone help me how to do this? (1 Reply)
Discussion started by: ajithshankar@ho
1 Replies

6. Shell Programming and Scripting

Automatically select records from several files and then run a C executable file inside the script

Dear list its my first post and i would like to greet everyone What i would like to do is select records 7 and 11 from each files in a folder then run an executable inside the script for the selected parameters. The file format is something like this 7 100 200 7 100 250 7 100 300 ... (1 Reply)
Discussion started by: Gtolis
1 Replies

7. Shell Programming and Scripting

awk - splitting 1 large file into multiple based on same key records

Hello gurus, I am new to "awk" and trying to break a large file having 4 million records into several output files each having half million but at the same time I want to keep the similar key records in the same output file, not to exist accross the files. e.g. my data is like: Row_Num,... (6 Replies)
Discussion started by: kam66
6 Replies

8. Shell Programming and Scripting

Block of records to select from a file

Hello: I am new to shell script programming. Now I would like to select specific records block from a file. For example, current file "xyz.txt" is containing 1million records and want to select the block of records from line number 50000 to 100000 and save into a file. Can anyone suggest me how... (3 Replies)
Discussion started by: nvkuriseti
3 Replies

9. Shell Programming and Scripting

Split a large file in n records and skip a particular record

Hello All, I have a large file, more than 50,000 lines, and I want to split it in even 5000 records. Which I can do using sed '1d;$d;' <filename> | awk 'NR%5000==1{x="F"++i;}{print > x}'Now I need to add one more condition that is not to break the file at 5000th record if the 5000th record... (20 Replies)
Discussion started by: ibmtech
20 Replies

10. Shell Programming and Scripting

Select records and fields

Hi All I would like to modify a file like this: >antax gioq21 tris notes abcdefghij klmnopqrs >betax gion32 ter notes2 tuvzabcdef ahgskslsooin this: >tris abcdefghij klmnopqrs >ter tuvzabcdef ahgskslsoo So, I would like to remove the first two fields(and output field 3) in record... (4 Replies)
Discussion started by: giuliangiuseppe
4 Replies

LEARN ABOUT SUSE

gappletviewer

GAPPLETVIEWER(1)							GNU							  GAPPLETVIEWER(1)

NAME

       gappletviewer - Load and runs an applet

SYNOPSIS

       appletviewer [OPTION]... URL...

       appletviewer [OPTION]... -code CODE

       appletviewer [OPTION]... -plugin INPUT,OUTPUT

DESCRIPTION

       The appletviewer tool loads and runs an applet.

       Use the first form to test applets specified by tag.  The URL should resolve to an HTML document from which the appletviewer will extract
       applet tags.  The APPLET, EMBED and OBJECT tags are supported.  If a given document contains multiple applet tags, all the applets will be
       loaded, with each applet appearing in its own window.  Likewise, when multiple URLs are specified, each applet tag instance is given its
       own window.  If a given document contains no recognized tags the appletviewer does nothing.

	       appletviewer http://www.gnu.org/software/classpath/

       Use the second form to test an applet in development.  This form allows applet tag attributes to be supplied on the command line.  Only one
       applet may be specified using the -code option.	The -code option overrides the URL form -- any URLs specified will be ignored.

	       appletviewer -code Test.class -param datafile,data.txt

       gcjwebplugin uses the third form to communicate with the appletviewer through named pipes.

OPTIONS

       URL OPTIONS

       -debug
	   This option is not yet implemented but is provided for compatibility.

       -encoding CHARSET
	   Use this option to specify an alternate character encoding for the specified HTML page.

       APPLET TAG OPTIONS

       -code CODE
	   Use the -code option to specify the value of the applet tag CODE attribute.

       -codebase CODEBASE
	   Use the -codebase option to specify the value of the applet tag CODEBASE attribute.

       -archive ARCHIVE
	   Use the -archive option to specify the value of the applet tag ARCHIVE attribute.

       -width WIDTH
	   Use the -width option to specify the value of the applet tag WIDTH attribute.

       -height HEIGHT
	   Use the -height option to specify the value of the applet tag HEIGHT attribute.

       -param NAME,VALUE
	   Use the -param option to specify values for the NAME and VALUE attributes of an applet PARAM tag.

       PLUGIN OPTION

       -plugin INPUT,OUTPUT
	   gcjwebplugin uses the -plugin option to specify the named pipe the appletviewer should use for receiving commands (INPUT) and the one
	   it should use for sending commands to gcjwebplugin (OUTPUT).

       DEBUGGING OPTION

       -verbose
	   Use the -verbose option to have the appletviewer print debugging messages.

       STANDARD OPTIONS

       -help
	   Use the -help option to have the appletviewer print a usage message, then exit.

       -version
	   Use the -version option to have the appletviewer print its version, then exit.

       -JOPTION
	   Use the -J option to pass OPTION to the virtual machine that will run the appletviewer.  Unlike other options, there must not be a
	   space between the -J and OPTION.

BUGS

SEE ALSO

AUTHOR

0.98								    2010-07-05							  GAPPLETVIEWER(1)
All times are GMT -4. The time now is 09:18 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy