Sponsored Content
Special Forums UNIX and Linux Applications High Performance Computing Huge Files to be Joined on Ux instead of ORACLE Post 302321915 by magedfawzy on Tuesday 2nd of June 2009 11:03:33 AM
Old 06-02-2009
MySQL Huge Files to be Joined on Ux instead of ORACLE

we have one file (11 Million) line that is being matched with (10 Billion) line.

the proof of concept we are trying , is to join them on Unix :

All files are delimited and they have composite keys..

could unix be faster than Oracle in This regards..

Please advice
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Script needs to be modified - Each 5 Rows to be joined in single line with comma (,)

Hi All, I'm using the following script to produce a result: #!/bin/sh awk ' $0 ~ /\+\+\+\+\+\+\+\+\+\+\+\+\+\+\+\+Interface\+\+\+\+\+\+\+\+\+\+\+\+\+\+\+\+\+\+\+\+\+/ { match_str="YES"; line_cnt=0; next; } { if((line_cnt < 5) && ( match_str=="YES")) { print $0; line_cnt += 1; } else... (0 Replies)
Discussion started by: ntgobinath
0 Replies

2. Shell Programming and Scripting

Comparing two huge files

Hi, I have two files file A and File B. File A is a error file and File B is source file. In the error file. First line is the actual error and second line gives the information about the record (client ID) that throws error. I need to compare the first field (which doesnt start with '//') of... (11 Replies)
Discussion started by: kmkbuddy_1983
11 Replies

3. UNIX for Dummies Questions & Answers

Difference between two huge files

Hi, As per my requirement, I need to take difference between two big files(around 6.5 GB) and get the difference to a output file without any line numbers or '<' or '>' in front of each new line. As DIFF command wont work for big files, i tried to use BDIFF instead. I am getting incorrect... (13 Replies)
Discussion started by: pyaranoid
13 Replies

4. UNIX for Advanced & Expert Users

Huge files manipulation

Hi , i need a fast way to delete duplicates entrys from very huge files ( >2 Gbs ) , these files are in plain text. I tried all the usual methods ( awk / sort /uniq / sed /grep .. ) but it always ended with the same result (memory core dump) In using HP-UX large servers. Any advice will... (8 Replies)
Discussion started by: Klashxx
8 Replies

5. Shell Programming and Scripting

Best Stratergy to process Huge files

I have a file with 20 million records. I need to read each record and process it. Which will be faster? Perl, Shell or awk? and what is the best method to read huge files line by line? (3 Replies)
Discussion started by: tene
3 Replies

6. Shell Programming and Scripting

Compare 2 folders to find several missing files among huge amounts of files.

Hi, all: I've got two folders, say, "folder1" and "folder2". Under each, there are thousands of files. It's quite obvious that there are some files missing in each. I just would like to find them. I believe this can be done by "diff" command. However, if I change the above question a... (1 Reply)
Discussion started by: jiapei100
1 Replies

7. Shell Programming and Scripting

Comparing 2 huge text files

I have this 2 files: k5login sanwar@systems.nyfix.com jjamnik@systems.nyfix.com nisha@SYSTEMS.NYFIX.COM rdpena@SYSTEMS.NYFIX.COM service/backups-ora@SYSTEMS.NYFIX.COM ivanr@SYSTEMS.NYFIX.COM nasapova@SYSTEMS.NYFIX.COM tpulay@SYSTEMS.NYFIX.COM rsueno@SYSTEMS.NYFIX.COM... (11 Replies)
Discussion started by: linuxgeek
11 Replies

8. UNIX for Dummies Questions & Answers

How to seperate two lines that are joined?

i have something like this abc 123 3234 1234 * qqoiki * abc 4533 34 1234 * lloiki * i want to make it two lines i,e.,abc 123 3234 1234 * qqoiki * abc 4533 34 1234 * lloiki * how to do that ? (13 Replies)
Discussion started by: anurupa777
13 Replies

9. Shell Programming and Scripting

Difference between two huge .csv files

Hi all, I need help on getting difference between 2 .csv files. I have 2 large . csv files which has equal number of columns. I nned to compare them and get output in new file which will have difference olny. E.g. File1.csv Name, Date, age,number Sakshi, 16-12-2011, 22, 56 Akash,... (10 Replies)
Discussion started by: Dimple
10 Replies

10. Shell Programming and Scripting

Aggregation of Huge files

Hi Friends !! I am facing a hash total issue while performing over a set of files of huge volume: Command used: tail -n +2 <File_Name> |nawk -F"|" -v '%.2f' qq='"' '{gsub(qq,"");sa+=($156<0)?-$156:$156}END{print sa}' OFMT='%.5f' Pipe delimited file and 156 column is for hash totalling.... (14 Replies)
Discussion started by: Ravichander
14 Replies
JOIN(1) 							   User Commands							   JOIN(1)

NAME
join - join lines of two files on a common field SYNOPSIS
join [OPTION]... FILE1 FILE2 DESCRIPTION
For each pair of input lines with identical join fields, write a line to standard output. The default join field is the first, delimited by whitespace. When FILE1 or FILE2 (not both) is -, read standard input. -a FILENUM print unpairable lines coming from file FILENUM, where FILENUM is 1 or 2, corresponding to FILE1 or FILE2 -e EMPTY replace missing input fields with EMPTY -i, --ignore-case ignore differences in case when comparing fields -j FIELD equivalent to `-1 FIELD -2 FIELD' -o FORMAT obey FORMAT while constructing output line -t CHAR use CHAR as input and output field separator -v FILENUM like -a FILENUM, but suppress joined output lines -1 FIELD join on this FIELD of file 1 -2 FIELD join on this FIELD of file 2 --check-order check that the input is correctly sorted, even if all input lines are pairable --nocheck-order do not check that the input is correctly sorted --header treat the first line in each file as field headers, print them without trying to pair them --help display this help and exit --version output version information and exit Unless -t CHAR is given, leading blanks separate fields and are ignored, else fields are separated by CHAR. Any FIELD is a field number counted from 1. FORMAT is one or more comma or blank separated specifications, each being `FILENUM.FIELD' or `0'. Default FORMAT outputs the join field, the remaining fields from FILE1, the remaining fields from FILE2, all separated by CHAR. Important: FILE1 and FILE2 must be sorted on the join fields. E.g., use ` sort -k 1b,1 ' if `join' has no options, or use ` join -t '' ' if `sort' has no options. Note, comparisons honor the rules specified by `LC_COLLATE'. If the input is not sorted and some lines cannot be joined, a warning message will be given. AUTHOR
Written by Mike Haertel. REPORTING BUGS
Report join bugs to bug-coreutils@gnu.org GNU coreutils home page: <http://www.gnu.org/software/coreutils/> General help using GNU software: <http://www.gnu.org/gethelp/> Report join translation bugs to <http://translationproject.org/team/> COPYRIGHT
Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. SEE ALSO
comm(1), uniq(1) The full documentation for join is maintained as a Texinfo manual. If the info and join programs are properly installed at your site, the command info coreutils 'join invocation' should give you access to the complete manual. GNU coreutils 8.5 February 2011 JOIN(1)
All times are GMT -4. The time now is 01:58 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy