06-02-2009
Huge Files to be Joined on Ux instead of ORACLE
we have one file (11 Million) line that is being matched with (10 Billion) line.
the proof of concept we are trying , is to join them on Unix :
All files are delimited and they have composite keys..
could unix be faster than Oracle in This regards..
Please advice
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hi All,
I'm using the following script to produce a result:
#!/bin/sh
awk ' $0 ~ /\+\+\+\+\+\+\+\+\+\+\+\+\+\+\+\+Interface\+\+\+\+\+\+\+\+\+\+\+\+\+\+\+\+\+\+\+\+\+/ {
match_str="YES"; line_cnt=0; next; }
{
if((line_cnt < 5) && ( match_str=="YES"))
{
print $0;
line_cnt += 1;
}
else... (0 Replies)
Discussion started by: ntgobinath
0 Replies
2. Shell Programming and Scripting
Hi,
I have two files file A and File B. File A is a error file and File B is source file. In the error file. First line is the actual error and second line gives the information about the record (client ID) that throws error. I need to compare the first field (which doesnt start with '//') of... (11 Replies)
Discussion started by: kmkbuddy_1983
11 Replies
3. UNIX for Dummies Questions & Answers
Hi,
As per my requirement, I need to take difference between two big files(around 6.5 GB) and get the difference to a output file without any line numbers or '<' or '>' in front of each new line.
As DIFF command wont work for big files, i tried to use BDIFF instead.
I am getting incorrect... (13 Replies)
Discussion started by: pyaranoid
13 Replies
4. UNIX for Advanced & Expert Users
Hi , i need a fast way to delete duplicates entrys from very huge files ( >2 Gbs ) , these files are in plain text.
I tried all the usual methods ( awk / sort /uniq / sed /grep .. ) but it always ended with the same result (memory core dump)
In using HP-UX large servers.
Any advice will... (8 Replies)
Discussion started by: Klashxx
8 Replies
5. Shell Programming and Scripting
I have a file with 20 million records. I need to read each record and process it.
Which will be faster? Perl, Shell or awk?
and what is the best method to read huge files line by line? (3 Replies)
Discussion started by: tene
3 Replies
6. Shell Programming and Scripting
Hi, all:
I've got two folders, say, "folder1" and "folder2".
Under each, there are thousands of files.
It's quite obvious that there are some files missing in each. I just would like to find them. I believe this can be done by "diff" command.
However, if I change the above question a... (1 Reply)
Discussion started by: jiapei100
1 Replies
7. Shell Programming and Scripting
I have this 2 files:
k5login
sanwar@systems.nyfix.com
jjamnik@systems.nyfix.com
nisha@SYSTEMS.NYFIX.COM
rdpena@SYSTEMS.NYFIX.COM
service/backups-ora@SYSTEMS.NYFIX.COM
ivanr@SYSTEMS.NYFIX.COM
nasapova@SYSTEMS.NYFIX.COM
tpulay@SYSTEMS.NYFIX.COM
rsueno@SYSTEMS.NYFIX.COM... (11 Replies)
Discussion started by: linuxgeek
11 Replies
8. UNIX for Dummies Questions & Answers
i have something like this
abc 123 3234 1234 * qqoiki * abc 4533 34 1234 * lloiki *
i want to make it two lines i,e.,abc 123 3234 1234 * qqoiki *
abc 4533 34 1234 * lloiki * how to do that ? (13 Replies)
Discussion started by: anurupa777
13 Replies
9. Shell Programming and Scripting
Hi all,
I need help on getting difference between 2 .csv files.
I have 2 large . csv files which has equal number of columns. I nned to compare them and get output in new file which will have difference olny.
E.g.
File1.csv
Name, Date, age,number
Sakshi, 16-12-2011, 22, 56
Akash,... (10 Replies)
Discussion started by: Dimple
10 Replies
10. Shell Programming and Scripting
Hi Friends !!
I am facing a hash total issue while performing over a set of files of huge volume:
Command used:
tail -n +2 <File_Name> |nawk -F"|" -v '%.2f' qq='"' '{gsub(qq,"");sa+=($156<0)?-$156:$156}END{print sa}' OFMT='%.5f'
Pipe delimited file and 156 column is for hash totalling.... (14 Replies)
Discussion started by: Ravichander
14 Replies
LEARN ABOUT NETBSD
orders
ORDERS(7) BSD Miscellaneous Information Manual ORDERS(7)
NAME
orders -- orders of magnitude
DESCRIPTION
The following table lists common multiples of bytes.
Name Prefix Power of 2 Power of 10
Kilobyte kB 2^10 10^3
Megabyte MB 2^20 10^6
Gigabyte GB 2^30 10^9
Terabyte TB 2^40 10^12
Petabyte PB 2^50 10^15
Exabyte EB 2^60 10^18
Zettabyte ZB 2^70 10^21
Yottabyte YB 2^80 10^24
The following table lists common bit rates as a power of ten.
Name Prefix Bit per second Byte per second
Bit per second bit/s 1 0.125
Byte per second B/s 8 1
Kilobit per second kbit/s 10^3 125
Kilobyte per second kB/s 8 * 10^3 1000
Megabit per second Mbit/s 10^6 125000
Megabyte per second MB/s 8 * 10^6 1000000
Gigabit per second Gbit/s 10^9 125000000
Gigabyte per second GB/s 8 * 10^9 1000000000
Terabit per second Tbit/s 10^12 125000000000
Terabyte per second TB/s 8 * 10^12 1000000000000
The following table lists common orders of magnitude as a power of ten.
Name Order Prefix Symbol Decimal
Septillionth 10^-24 yocto y 0.000000000000000000000001
Sextillionth 10^-21 zepto z 0.000000000000000000001
Quintillionth 10^-18 atto a 0.000000000000000001
Quadrillionth 10^-15 femto f 0.000000000000001
Trillionth 10^-12 pico p 0.000000000001
Billionth 10^-9 nano n 0.000000001
Millionth 10^-6 micro mu 0.000001
Thousandth 10^-3 milli m 0.001
Hundredth 10^-2 centi c 0.01
Tenth 10^-1 deci d 0.1
One 10^0 - - 1
Ten 10^1 deca da 10
Hundred 10^2 hecto h 100
Thousand 10^3 kilo k 1000
Million 10^6 mega M 1000000
Billion 10^9 giga G 1000000000
Trillion 10^12 tera T 1000000000000
Quadrillion 10^15 peta P 1000000000000000
Quintillion 10^18 exa E 1000000000000000000
Sextillion 10^21 zetta Z 1000000000000000000000
Septillion 10^24 yotta Y 1000000000000000000000000
SEE ALSO
units(1), strsuftoll(3), number(6)
STANDARDS
There have been various attempts to standardize the set of binary prefixes. Organizations such as International Electrotechnical Commission
(IEC) have proposed new prefixes such as ``kibi'', ``mebi'', ``gibi'', and ``yobi'', but the adoption has been slow at best.
BSD
August 6, 2011 BSD