Shell Programming and Scripting

Hi,

I have a file with values like this
1 11
2 11
3 44
4 55
5 66 (an representative of what I have).
I want to split this file into smaller files based on column 1 values (values within a range).

The issue that I am facing is that the file is really big, and takes too long to run.

Is there anyway that I can do this-

1. given a initial range, export the values, save the last line number.
2. start the next search from where the previous search finished.

I tried playing with NR values but turned up useless.

Or may be SED is an alternative?

HELP!!

Thanks a lot in advance

Hi,
Please elaborate the query. Also provide with more sample input/output of exactly what you want.

Hi,

Thanks for responding. Expanding on the query, say I have a file with 2 columns, the second column has many numbers which does not concern this search and export. The first column has linearly increasing values, say from 1.000 to 1000000.000. I want to export this big file into smaller files with each file containing a thousand column 1 values (it is not exactly 1000 every time, it is all values between 1 and 1000, 1000 and 2000). There are decimals so between 1 to 1000 there may be more than a thousand values. I used

Code:

awk -vmyvarA=$a -vmyvarB=$b '($1 >= myvarA && $1 <= myvarB) { print $0; }'

to split into smaller files.

As you can see, this is very inefficient if I loop it over all values I need. It will search from the start for every iteration. What I wanted to do was to do the first search, save the last matching line number and continue the next search from that line.

I hope it is clear now.

If you want to split the files, you can make use of split command.
I still didn't get what you really want! What do you mean by export?

--ahamed

I dont think split will work because I am splitting one big file into smaller files, but the number of lines in the smaller file is not same every time. I meant export as writing the output to a new text file.

I am splitting one big file into smaller ones.
Each small file has values between 'a' and 'b'.
Though the difference between a and b are fixed every time, the number of entries within this range is not.
So, when I run between 'a' and 'b' for the second iteration, awk will begin to look from the first line-> I want to stop this from happening as I have to iterate a large number of times.
So, I thought it would be more efficient if I save the last matching NR and resume the search from where I left off ( since column 1 increases linearly).

Hi,
First try

Code:

 cat -n filename

. Then try the for loop in awk if you can give me the output to cat -n i will try but it won't be a generic solution only problem based solution.

Hi,

cat -n is too large to post here, I have 42273547 lines in the file.
A small snippet of what I have is-

Code:

40556271        1319211119.897235 0.0047939
40556272        1319211119.897236 0.0000012
40557277        1319211121.492435 0.0000250
40557278        1319211121.492436 0.0000010

What I want is to create smaller files containing values in between two column 1 values.

As you can see, say I split into smaller files for every 1000 column 1 values, doing a exhaustive search 1000 times would be wasteful. Once I reach the last 'b' value, I want to store it and resume the search from where I left off (and it would also be inline with my search)
Thanks!