03-30-2010
You should just run it and find out if performance and resource consumption is acceptable. What does and doesn't have performance issues depends on the person, situation, priority of the job, the hardware, etc.
Only thing that can be stated by looking at the awk/sort code, is that it will require something on the order of 2 x 10 x avg_line_length megabytes of memory for a 10 million line data set, since both awk and sort will require full copies of the data in memory (worst case scenario). If the average line length is 50 characters, it could approach 1 gigabyte or ram required.
Regards,
Alister
10 More Discussions You Might Find Interesting
1. UNIX for Dummies Questions & Answers
Hi Gurus,
I am new to UNIX(HP). my requirmnet is File needs to needs to be split into multiple files dependa on one key value.
example
1 abc 333 us
2 bbc 444 ch
5 nnn 333 kk
7 hhh 555 ll
3 hhh 333 hh
now the requirment is line with 333 should be copied into test1.txt and... (14 Replies)
Discussion started by: arund_01
14 Replies
2. Shell Programming and Scripting
Hello,
I have a text file like this:
NONE FILE_Rename
frompath: /log_audit/AIX/log/current/AIXAFTPP.log
NONE FILE_Unlink
filename /audit/tempfile.14041142
NONE FILE_Rename ... (8 Replies)
Discussion started by: carloskl
8 Replies
3. Shell Programming and Scripting
Hi ,
Need help in concating two lines based on certain character, for example my file has the messages :
:57A:qweqweww
:58A:qeqewqeqe
-}$
{1:fffff2232323}{2:123123dasds}{4:
:20:121323232323232
:21:sdsadasdasddadad
if the line ends with "-}$" or if a line starts with "{1:" then it... (5 Replies)
Discussion started by: ulin
5 Replies
4. Shell Programming and Scripting
I have file which contains gene lines something like this
Transcript Name GO
POPTR_0016s06290.1 98654
POPTR_2158s00200.1 11324
POPTR_0004s22390.1 12897
POPTR_0001s11490.1
POPTR_0016s13950.1 14532
POPTR_0015s05840.1 13455
POPTR_0013s06470.1 12344... (6 Replies)
Discussion started by: shen
6 Replies
5. Shell Programming and Scripting
I would like to print string between two sequent digits and concatenate it into one single line.
input.txt
99 cord, rope, strand, twine,
twist, 100 strand, twine, twist,
cord, rope 101 strand, twine,
twist, twine, twist, cord, rope
105 cord, rope ,twi ... (8 Replies)
Discussion started by: sdf
8 Replies
6. Shell Programming and Scripting
Hello,
I have the following example data file:
Rv.Global_Sk,1077.160523,D,16/09/2011
Rv.Global_Sk,1077.08098,D,17/09/2011
Rv.Global_Sk,1077.001445,D,18/09/2011
Rv.Global_Sk,1072.660733,D,19/09/2011
Rv.Global_Sk,1070.381557,D,20/09/2011
Rv.Global_Sk,1071.971747,D,21/09/2011... (4 Replies)
Discussion started by: csierra
4 Replies
7. Shell Programming and Scripting
Hi. I'd appreciate if I can get some direction in this issue to get me going.
Datafile1:
-About 4000 records, I have to update field#4 in selected records based on a match in the key field (Field#1).
-Field #1 is the key field (servername) . # of Fields may vary
# comment
server1 bbb ccc... (2 Replies)
Discussion started by: RascalHoudi
2 Replies
8. UNIX for Dummies Questions & Answers
I have a file with line 4 : F SITE SPA_M2 SPA_M3 SPA_M4
and a line 237 with: BV_N4 VbdGO_PW Rs_NW_STI Rc_N+OD
need a awk liner to concat the two lines so that line 2 sits next to line1 and looks like:
F SITE SPA_M2 SPA_M3 SPA_M4 BV_N4 VbdGO_PW ... (8 Replies)
Discussion started by: awkaddict
8 Replies
9. Shell Programming and Scripting
Hi All,
I have a XML file which is looks like as below. <<please see the attachment >>
<?xml version="1.0" encoding="UTF-8"?>
<esites>
<esite>
<name>XXX.com</name>
<storeId>10001</storeId>
<module>
... (4 Replies)
Discussion started by: Rajeev_hbk
4 Replies
10. Shell Programming and Scripting
Hello experts
I have a file with paragraphs begining with a keeping date and ending with "END":
20120301 num num
John num num A keepnum1 num num
kathrin num num A keepnum1 num num
kathrin num num B keepnum2 num num
Pete num num A keepnum1 num num
Jacob num... (2 Replies)
Discussion started by: phaethon
2 Replies
SORT(1) General Commands Manual SORT(1)
NAME
sort - sort a file of ASCII lines
SYNOPSIS
sort [-bcdfimnru] [-tc] [-o name] [+pos1] [-pos2] file ...
OPTIONS
-b Skip leading blanks when making comparisons
-c Check to see if a file is sorted
-d Dictionary order: ignore punctuation
-f Fold upper case onto lower case
-i Ignore nonASCII characters
-m Merge presorted files
-n Numeric sort order
-o Next argument is output file
-r Reverse the sort order
-t Following character is field separator
-u Unique mode (delete duplicate lines)
EXAMPLES
sort -nr file # Sort keys numerically, reversed
sort +2 -4 file # Sort using fields 2 and 3 as key
sort +2 -t: -o out # Field separator is :
sort +.3 -.6 # Characters 3 through 5 form the key
DESCRIPTION
Sort sorts one or more files. If no files are specified, stdin is sorted. Output is written on standard output, unless -o is specified.
The options +pos1 -pos2 use only fields pos1 up to but not including pos2 as the sort key, where a field is a string of characters delim-
ited by spaces and tabs, unless a different field delimiter is specified with -t. Both pos1 and pos2 have the form m.n where m tells the
number of fields and n tells the number of characters. Either m or n may be omitted.
SEE ALSO
comm(1), grep(1), uniq(1).
SORT(1)