Sponsored Content
Top Forums Shell Programming and Scripting Quick way to select many records from a large file Post 302942389 by zenongz on Tuesday 28th of April 2015 12:06:26 PM
Old 04-28-2015
Hi vgersh99 and sea,

Thanks for your solutions.

I tried vgersh99's one line awk command. It worked great and solved my problem in seconds. AmazingSmilie. Thanks again.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Using a variable to select records with awk

As part of a bigger task, I had to read thru a file and separate records into various batches based on a field. Specifically, separate records based on the value in the batch field as defined below. The batch field left-justified numbers. The datafile is here > cat infile 12345 1 John Smith ... (5 Replies)
Discussion started by: joeyg
5 Replies

2. UNIX for Dummies Questions & Answers

Quick question about finding a large file

what is the correct command for finding the largest file and displaying it without any error information? I can find it, but how do I display it in the same command? (6 Replies)
Discussion started by: raidkridley
6 Replies

3. Shell Programming and Scripting

Extract data from large file 80+ million records

Hello, I have got one file with more than 120+ million records(35 GB in size). I have to extract some relevant data from file based on some parameter and generate other output file. What will be the besat and fastest way to extract the ne file. sample file format :--... (2 Replies)
Discussion started by: learner16s
2 Replies

4. Shell Programming and Scripting

select records from one file based on a second file

Hi all: I have two files: file1: 74 DS 9871 199009871 1 1990 4 1 165200 Sc pr de te sa ox 1.0 1.0 13.0000 35.7560 5.950 3.0 3.0 13.0100 35.7550 5.970 ** 74 DS 99004 74DS99004 6738 1990 4 1 165200 Eb pr de te sa ox 1.0 1.0 13.0000 ... (7 Replies)
Discussion started by: rleal
7 Replies

5. Shell Programming and Scripting

How to Pick Random records from a large file

Hi, I have a huge file say with 2000000 records. The file has 42 fields. I would like to pick randomly 1000 records from this huge file. Can anyone help me how to do this? (1 Reply)
Discussion started by: ajithshankar@ho
1 Replies

6. Shell Programming and Scripting

Automatically select records from several files and then run a C executable file inside the script

Dear list its my first post and i would like to greet everyone What i would like to do is select records 7 and 11 from each files in a folder then run an executable inside the script for the selected parameters. The file format is something like this 7 100 200 7 100 250 7 100 300 ... (1 Reply)
Discussion started by: Gtolis
1 Replies

7. Shell Programming and Scripting

awk - splitting 1 large file into multiple based on same key records

Hello gurus, I am new to "awk" and trying to break a large file having 4 million records into several output files each having half million but at the same time I want to keep the similar key records in the same output file, not to exist accross the files. e.g. my data is like: Row_Num,... (6 Replies)
Discussion started by: kam66
6 Replies

8. Shell Programming and Scripting

Block of records to select from a file

Hello: I am new to shell script programming. Now I would like to select specific records block from a file. For example, current file "xyz.txt" is containing 1million records and want to select the block of records from line number 50000 to 100000 and save into a file. Can anyone suggest me how... (3 Replies)
Discussion started by: nvkuriseti
3 Replies

9. Shell Programming and Scripting

Split a large file in n records and skip a particular record

Hello All, I have a large file, more than 50,000 lines, and I want to split it in even 5000 records. Which I can do using sed '1d;$d;' <filename> | awk 'NR%5000==1{x="F"++i;}{print > x}'Now I need to add one more condition that is not to break the file at 5000th record if the 5000th record... (20 Replies)
Discussion started by: ibmtech
20 Replies

10. Shell Programming and Scripting

Select records and fields

Hi All I would like to modify a file like this: >antax gioq21 tris notes abcdefghij klmnopqrs >betax gion32 ter notes2 tuvzabcdef ahgskslsooin this: >tris abcdefghij klmnopqrs >ter tuvzabcdef ahgskslsoo So, I would like to remove the first two fields(and output field 3) in record... (4 Replies)
Discussion started by: giuliangiuseppe
4 Replies
FITCIRCLE(l)															      FITCIRCLE(l)

NAME
fitcircle - find mean position and pole of best-fit great [or small] circle to points on a sphere. SYNOPSIS
fitcircle [ xyfile ] -Lnorm [ -H[nrec] ] [ -S ] [ -V ] [ -: ] [ -bi[s][n] ] DESCRIPTION
fitcircle reads lon,lat [or lat,lon] values from the first two columns on standard input [or xyfile]. These are converted to cartesian three-vectors on the unit sphere. Then two locations are found: the mean of the input positions, and the pole to the great circle which best fits the input positions. The user may choose one or both of two possible solutions to this problem. The first is called -L1 and the second is called -L2. When the data are closely grouped along a great circle both solutions are similar. If the data have large dispersion, the pole to the great circle will be less well determined than the mean. Compare both solutions as a qualitative check. The -L1 solution is so called because it approximates the minimization of the sum of absolute values of cosines of angular distances. This solution finds the mean position as the Fisher average of the data, and the pole position as the Fisher average of the cross-products between the mean and the data. Averaging cross-products gives weight to points in proportion to their distance from the mean, analogous to the "leverage" of distant points in linear regression in the plane. The -L2 solution is so called because it approximates the minimization of the sum of squares of cosines of angular distances. It creates a 3 by 3 matrix of sums of squares of components of the data vectors. The eigenvectors of this matrix give the mean and pole locations. This method may be more subject to roundoff errors when there are thousands of data. The pole is given by the eigenvector corresponding to the smallest eigenvalue; it is the least-well represented factor in the data and is not easily estimated by either method. -L Specify the desired norm as 1 or 2, or use -L or -L3 to see both solutions. OPTIONS
xyfile ASCII [or binary, see -b] file containing lon,lat [lat,lon] values in the first 2 columns. If no file is specified, fitcircle will read from standard input. -H Input file(s) has Header record(s). Number of header records can be changed by editing your .gmtdefaults file. If used, GMT default is 1 header record. -S Attempt to fit a small circle instead of a great circle. The pole will be constrained to lie on the great circle connecting the pole of the best-fit great circle and the mean location of the data. -V Selects verbose mode, which will send progress reports to stderr [Default runs "silently"]. -: Toggles between (longitude,latitude) and (latitude,longitude) input/output. [Default is (longitude,latitude)]. Applies to geo- graphic coordinates only. -bi Selects binary input. Append s for single precision [Default is double]. Append n for the number of columns in the binary file(s). [Default is 2 input columns]. EXAMPLES
Suppose you have lon,lat,grav data along a twisty ship track in the file ship.xyg. You want to project this data onto a great circle and resample it in distance, in order to filter it or check its spectrum. Try: fitcircle ship.xyg -L2 project ship.xyg -Cox/oy -Tpx/py -S -pz | sample1d -S-100 -I1 > output.pg Here, ox/oy is the lon/lat of the mean from fitcircle, and px/py is the lon/lat of the pole. The file output.pg has distance, gravity data sampled every 1 km along the great circle which best fits ship.xyg SEE ALSO
gmt(1gmt), project(1gmt), sample1d(1gmt) 1 Jan 2004 FITCIRCLE(l)
All times are GMT -4. The time now is 08:16 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy