UNIX for Advanced & Expert Users

Hi.

I have a huge file (350 million lines).
I need to delete all lines in it that:

1. Begin with a pipe character -- '|'
2. Or have less than 5 pipe characters in the line

Have been searching for some SED/AWK help for this (which is faster btw on such a ginormous file?) but only found this, but this works with a number not a pipe character.


Much appreciate any pointers.

Does not matter - the time is due to I/O. sed reads the file once as does awk.

Code:

awk -F '|'  ' /^|/ || NF<5 {next} {print } ' bigfile > newbigfile

Thanks Jim. Doesn't work. The "newbigfile" is totally blank.


---------- Post updated at 07:58 AM ---------- Previous update was at 07:28 AM ----------

Found it. The first | has to be escaped by a back slash. So the command should be:

Code:

awk -F '|'  ' /^\|/ || NF<5 {next} {print } ' bigfile > newbigfile

Try this

Code:

awk -F "|" '{if( NF>5 && $1 != "") print}' inputfile

Hi Sorry to intrupt in this thread.
i am new to this form..
can any one please let me know how to create new thread?

Go to the relevant forum and click on New Thread on top left of the forum.

Thanks tene