Search Results

i don't have such scenario's. my trouble is to ignore commas within double quotes.

Hi

Description of input file I have:
-------------------------
1) CSV with double quotes for string fields.
2) Some string fields have Comma as part of field value.
3) Have Duplicate lines...

Hi Corona,

cat file.in | awk -F"," '!($1$2$3$4$5$6$7$8 in a){a[$1$2$3$4$5$6$7$8];print $0}' | cut -d, -f1-150 > file.out

I use the above code, ie., using cut within awk. Do you think its...

Hi

file.in and file.out are in csv format. the code I have now is,

cat file.in | awk -F"," '!($1$2$3$4$5$6$7$8 in a){a[$1$2$3$4$5$6$7$8];print $0}' > file.out


Here, I am printing entire...

Thank you everyone.

I am using awk '!($0 in a){a[$0];print}' which I found efficient out of all options.

sort: write failed: /tmp/sortmO2esr: No space left on device
during the last sort execution.

I am trying the split and sort, I will let you know once it is done. Meanwhile, I have a doubt, why can't we implement something like below so that it will not take much space ..
for line in `cat...

Sorry not the mention .. "thanks for your prompt replies".

Now I am running ..
perl -ne'print unless $_{$_}++' result > result_new2

I am getting the error messge - Segmentation fault

awk: cmd. line:1: (FILENAME=result FNR=6094197) fatal: assoc_lookup: bucket->ahname_str: can't allocate 423 bytes of memory (Cannot allocate memory)

---------- Post updated at 09:21 AM ----------...

Hi All,

I have a very huge file (4GB) which has duplicate lines. I want to delete duplicate lines leaving unique lines. Sort, uniq, awk '!x[$0]++' are not working as its running out of buffer...