awk to parse huge files Post: 302853197

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Compare 2 huge files wrt to a key using awk

Hi Folks, I need to compare two very huge file ( i.e the files would contain a minimum of 70k records each) using awk or sed. The comparison needs to be done with respect to a 'key'. For example : File1 ********** 1234|TONY|Y75634|20/07/2008 1235|TINA|XCVB56|30/07/2009...

2. Shell Programming and Scripting

Comparing two huge files

Hi, I have two files file A and File B. File A is a error file and File B is source file. In the error file. First line is the actual error and second line gives the information about the record (client ID) that throws error. I need to compare the first field (which doesnt start with '//') of...

3. UNIX for Advanced & Expert Users

Huge files manipulation

Hi , i need a fast way to delete duplicates entrys from very huge files ( >2 Gbs ) , these files are in plain text. I tried all the usual methods ( awk / sort /uniq / sed /grep .. ) but it always ended with the same result (memory core dump) In using HP-UX large servers. Any advice will...

4. Shell Programming and Scripting

Compare 2 folders to find several missing files among huge amounts of files.

Hi, all: I've got two folders, say, "folder1" and "folder2". Under each, there are thousands of files. It's quite obvious that there are some files missing in each. I just would like to find them. I believe this can be done by "diff" command. However, if I change the above question a...

5. Shell Programming and Scripting

awk script to parse results from TWO files

I am trying to parse two files and get data that does not match in one of the columns ( column 3 in my case ) Data for two files are as follows A.txt ===== abc 10 5 0 1 16 xyz 16 1 1 0 18 efg 30 8 0 2 40 ijk 22 2 0 1 25 B.txt ===== abc...

6. Shell Programming and Scripting

AWK failing to parse on certain files

Dear Unix Gurus, need your expertise to help troubleshoot a certain problem i'm having. I crated a shell script which will ftp get 1 crash log from multiple servers (listed in a text file). Each log will then be parsed by calling an awk script. The problem is, for certain log its parsing...

7. Shell Programming and Scripting

How to parse a huge 600MB zipped file?

I'm new to Unix, trying to parse a huge 600MB zipped file... I need to bzcat this file once and do some calculations (word count) on the lines based on certain criteria (see script) the correct result/output should be: column1=6 column2=4 the problem is that I'm getting column2=0 (see...

8. Shell Programming and Scripting

awk does not work well with huge data?

Dear all , I found that if we work with thousands line of data, awk does not work perfectly. It will cut hundreds line (others are deleted) and works only on the remain data. I used this command : awk '$1==1{$1="Si"}{print>FILENAME}' coba.xyz to change value of first column whose value is 1...

9. Shell Programming and Scripting

awk Parse And Create Multiple Files Based on Field Value

Hello: I am working parsing a large input file which will be broken down into multiples based on the second field in the file, in this case: STORE. The idea is to create each file with the corresponding store number, for example: Report_$STORENUM_$DATETIMESTAMP , and obtaining the...

10. Shell Programming and Scripting

Parse input of two files to be the same in awk

I have two files that I am going to use diff to find the differences but need to parse them before I do that. I have include the format of each file1 and file2 with the desired output of each (the first 5 fields in each file). The first file has a "chr" before the # that needs to be removed. I...

LEARN ABOUT REDHAT

nice

NICE(2) 						     Linux Programmer's Manual							   NICE(2)

NAME

       nice - change process priority

SYNOPSIS

       #include <unistd.h>

       int nice(int inc);

DESCRIPTION

       nice adds inc to the nice value for the calling pid.  (A large nice value means a low priority.)  Only the superuser may specify a negative
       increment, or priority increase.

RETURN VALUE

       On success, zero is returned.  On error, -1 is returned, and errno is set appropriately.

ERRORS

       EPERM  A non-super user attempts to do a priority increase by supplying a negative inc.

CONFORMING TO

       SVr4, SVID EXT, AT&T, X/OPEN, BSD 4.3. However, the Linux and glibc (earlier than glibc 2.2.4) return  value  is  nonstandard,  see  below.
       SVr4 documents an additional EINVAL error code.

NOTES

       Note  that  the routine is documented in SUSv2 to return the new nice value, while the Linux syscall and (g)libc (earlier than glibc 2.2.4)
       routines return 0 on success.  The new nice value can be found using getpriority(2).  Note that an implementation in which nice returns the
       new  nice  value  can  legitimately  return -1.	To reliably detect an error, set errno to 0 before the call, and check its value when nice
       returns -1.

SEE ALSO

       nice(1), getpriority(2), setpriority(2), fork(2), renice(8)

Linux								    2001-06-04								   NICE(2)

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Compare 2 huge files wrt to a key using awk

Discussion started by: Ranjani

2. Shell Programming and Scripting

Comparing two huge files

Discussion started by: kmkbuddy_1983

3. UNIX for Advanced & Expert Users

Huge files manipulation

Discussion started by: Klashxx

4. Shell Programming and Scripting

Compare 2 folders to find several missing files among huge amounts of files.

Discussion started by: jiapei100