Shell Programming and Scripting

The dataset I'm working on is about 450G, with about 7000 colums and 30,000,000 rows.
I want to extract about 2000 columns from the original file to form a new file.
I have the list of number of the columns I need, but don't know how to extract them.
Thanks!

Is the file delimited (like with commas between field) or fixed record length?

happypoker,
You can use gnu awk and use array for the desired columns to store and print the data.

Sorry I didn't make it clear.
Yes, the file is tab delimited.

If you can post a sample data it may be helpful to determine the actual code.

You want some columns, all rows, not keyed by some columns, some values? I'd say 'cut' but I have found cut disappointing in speed in some situations. No chance the rows or columns are fixed length? I'd go with C, mmap() and a 64 bit compile, but awk sounds like a good fit: field oriented and delimiter configurable.

Tools like Ab Initio would divide the file into N byte blocks and process it in parallel. You could do the same, if you write a script that seeks and only goes so far, but it gets tricky ensuring the row containing byte N+1 is processed by the first instance, as it must be ignored by the second instance, since it might be (and probably is) partial and belongs to the prior instance. Or not, you get to decide who owns that row. My thought is that every instance goes N+1 bytes, ignoring the first line if not the first instance, and beyond N+1 to get the end of the last line.

Welcome to big data.

Thank you! I'm able to get the list column # into an array, but still don't know how to print those corresponding columns out.

The data is something like:

Code:

000 A B C D 2.22 3.4 1.03 .....
001 B F S D 3.2 2.1 3.2 2.3 .....
....
...
132 F S F X 2.3 3.4 5.3 2.1 ....

It has about 6600 columns and 30,000,000 lines. It's too big to put the needed columns in some array, I have to print the needed columns line by line.