05-26-2011
Quote:
Originally Posted by
DGPickett
How's swap space? (Still should work, stall or report error not quiet exit.)
I'm not sure, do you happen to know how to check swap space or tmp space?
The sort solution is much more robust than in-memory duplicate detection. If duplictes are not fatal, sort -u and compare file sizes.
Yes, the concept of the sort occured to me some more.
The script I have determines duplicates by a concatenation of specific columns in the file (the key) and uses another column ( a datetime) to determine which record to keep. (the greatest datetime) This is why I use awk.
I'm thinking now and thanks to your post ! , that maybe a can sort the file with the key columns with the greatest datetime on top and then just keep the first one.
I'm not sure how the sort command would look like though.
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
I have a piece of script that needs to pull every element in column 2 from a file of varying size. Rather than writing something to read the file and then pass the data line by line to my script, I want to use awk.
However, what is the correct syntax to get awk to call a function (named via:
... (7 Replies)
Discussion started by: quaddriver
7 Replies
2. UNIX for Dummies Questions & Answers
Hey guys, need some help. Running AIX Version 5.2 and one of our cron jobs is writing errors to a log file. Any ideas on the following error message.
Error: Internal system error: Unable to initialize standard output file
I'm guessing more info might be needed, so let me know.
Thanks (2 Replies)
Discussion started by: firkus
2 Replies
3. Shell Programming and Scripting
:)Hi,
I am working on perl-cgi script which i wrote on unix server, and now i want to run it from windows.
Have put DNS entry, sybase and apache is running...
But still I am getting 500 Internal Server Error!!!
what could be the reason? (2 Replies)
Discussion started by: darshakraut
2 Replies
4. UNIX for Advanced & Expert Users
Hi,
I need to redirect internal internet requests to a auth client site siting on the gateway. Currently users that are authenticated to access the internet have there mac address listed in the FORWARD chain. All other users need to be redirected to a internal site for authentication.
Can... (1 Reply)
Discussion started by: mshindo
1 Replies
5. AIX
Dear experts,
please inform when i am trying to use dbx to debug my code,
iam getting following error:
internal error: expected member attribute 's', 'p', 'r' or 'b', found... (1 Reply)
Discussion started by: vin_pll
1 Replies
6. Shell Programming and Scripting
Hi,
I need to use a function in the find command to do some process on the file. I'm trying:
funcname(){ ... }
...
find ./ -name "*" -exec funcname {} \;
But somehow this is not working. I don't want to have a separate script for whatever processing the function does. I want to have... (1 Reply)
Discussion started by: victorcheung
1 Replies
7. Shell Programming and Scripting
Hello,
When I run my awk script with input files greater than 300 MB I always get this error:
awk: Internal software error in the tostring function on TS1101?05044400?.0085498227?0?.0011041461?.0034752266?.00397045?0?0?0?0?0?0?11/02/10?09/23/10???10?no??0??no?sct_det3_10_20110516_143936.txt
... (0 Replies)
Discussion started by: script_op2a
0 Replies
8. Shell Programming and Scripting
Hello,
I am running a bash script under linux which first defines an CA-array like
j=0
num1=120.00
num2=10.00
until
do
CA='echo $num1 + $j*$num2'
j=$
done
within the later awk section of this same script I want to read data from a file. If the value of the second column is... (3 Replies)
Discussion started by: MotAah
3 Replies
9. Shell Programming and Scripting
Hi to all,
I have an AWK code that looks like this:
awk 'NR==FNR
{
#code to process file1
next
}
{
#code to process file2
a=myfunction(x)
}
END {
#Some code
b=myfunction(x)
... (3 Replies)
Discussion started by: Ophiuchus
3 Replies
10. HP-UX
==================================================================================
SEGV_MAPERR: Address not mapped to object
fffffffffffffff0 (Memory location which caused fault)
------------------ C-STACK ----------------------
(0) 0x40000000007622f0 CTrcStack2 + 0x1d0 at dptstack_mt.c:228... (6 Replies)
Discussion started by: sidharthmellam
6 Replies