Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

How to open large datafie in awk?

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting How to open large datafie in awk?
# 1  
Old 03-08-2013
How to open large datafie in awk?
I just tried

Code:
awk '{print}' all.plo 
awk: cannot open all.plo (Value too large for defined data type)

Code:
awk '{print $8"-"$7"-"$6,$9,$4,$5,$12,$15}' all.plo 
awk: cannot open all.plo (Value too large for defined data type)

Code:
datafile size is 4.8GB

any other provision ? only
Code:
cat

works

Code:
FS is space as well as tab(mixed)

# 2  
Old 03-08-2013
this is usual case with large files and can be worked around with split, which in turn makes it possible to use the full data

Code:
#split -b 900m filename prefix
and then
#awk '{print $8"-"$7"-"$6,$9,$4,$5,$12,$15}'  prefix*

should help.
# 3  
Old 03-08-2013
What is your OS and version? And what is your version of awk? Are you using a 32-bit version?
# 4  
Old 03-08-2013
Certainly a 32-bit awk program; should be upgraded/patched for "largefile" support.
Perhaps the shell has "largefile" support?
Then it helps to open the file in the shell:
Code:
awk '{print $8"-"$7"-"$6,$9,$4,$5,$12,$15}' < all.plo

This User Gave Thanks to MadeInGermany For This Post:
Scrutinizer
# 5  
Old 03-08-2013
Its Ubuntu 12.04 LTS 32-bit and awk is 1.2 default

I tried #4 solution its working Thank you all
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Reversing large data set with awk?

Hello! I have quite a bit of map data that I have to edit. I originally had a DOS script that would reverse x1, y1 coordinates in order to change the direction of a particular segment in a map file. It worked wonderfully and all was well, but my bossman told me that there is a boatload of nodes... (9 Replies)
Discussion started by: Mothra
9 Replies

2. Shell Programming and Scripting

Process multiple large files with awk

Hi there, I'm camor and I'm trying to process huge files with bash scripting and awk. I've got a dataset folder with 10 files (16 millions of row each one - 600MB), and I've got a sorted file with all keys inside. For example: a sample_1 200 a.b sample_2 10 a sample_3 10 a sample_1 10 a... (4 Replies)
Discussion started by: camor
4 Replies

3. Shell Programming and Scripting

awk output yields error: awk:can't open job_name (Autosys)

Good evening, Im newbie at unix specially with awk From an scheduler program called Autosys i want to extract some data reading an inputfile that comprises jobs names, then formating the output to columns for example 1. This is the inputfile: $ more MapaRep.txt ds_extra_nikira_usuarios... (18 Replies)
Discussion started by: alexcol
18 Replies

4. Shell Programming and Scripting

awk input large file

Hi...Does anyone know how to input huge file about 25 GB to awk if single file then this works awk '{print}' <hugefile suppose if have to use something like this awk FNR==NR{x=$0;next}{print $0,x}' hugefile1 hugefile2 then how to redirect ? and is there any provision to assign memory... (12 Replies)
Discussion started by: Akshay Hegde
12 Replies

5. UNIX for Dummies Questions & Answers

Find common numbers from two very large files using awk or the like

I've got two files that each contain a 16-digit number in positions 1-16. The first file has 63,120 entries all sorted numerically. The second file has 142,479 entries, also sorted numerically. I want to read through each file and output the entries that appear in both. So far I've had no... (13 Replies)
Discussion started by: Scottie1954
13 Replies

6. Shell Programming and Scripting

sed and awk not working on a large record file

Hi All, I have a very large single record file. abc;date||bcd;efg|......... pqr;stu||record_count;date when i do wc -l on this file it gives me "0" records, coz of missing line feed. my problem is there is an extra pipe that is coming at the end of this record like... (6 Replies)
Discussion started by: Gurkamal83
6 Replies

7. Shell Programming and Scripting

Help needed for parsing large XML with awk.

My XML structure looks like: <?xml version="1.0" encoding="UTF-8"?> <SearchRepository> <SearchItems> <SearchItem> ... </SearchItem> <SearchItem> ... ... (1 Reply)
Discussion started by: jasonjustice
1 Replies

8. Programming

Is there a system call other than 'open' for opening very large files?

Dear all, Inside a C program, I want to open a very big file (about 12 GB) in order to read its content. Here is the code: /* argv contains the path to the file. */ inputFileDescriptor = open(argv, O_RDONLY); if (inputFileDescriptor < 0) { ... (6 Replies)
Discussion started by: dariyoosh
6 Replies

9. Shell Programming and Scripting

AWK Shell Program to Split Large Files

Hi, I need some help creating a tidy shell program with awk or other language that will split large length files efficiently. Here is an example dump: <A001_MAIL.DAT> 0001 Ronald McDonald 01 H81 0002 Elmo St. Elmo 02 H82 0003 Cookie Monster 01 H81 0004 Oscar ... (16 Replies)
Discussion started by: mkastin
16 Replies

10. Shell Programming and Scripting

Reading large file, awk and cut

Hello all, I have 2 files, the first (indexFile1) contains start offset and length for each record inside the second file. The second file can be very large, each actual record start offset and length is defined by the entry in indexFile1. Since there are no records separators wc-l returns 0 for... (1 Reply)
Discussion started by: gio001
1 Replies
Login or Register to Ask a Question