05-22-2006
Challenging Compare and validate question -- plus speed.
I have a tab delimited HUGE file (13 million records) with Detail, Metadata and Summary records.
Sample File looks like this
M BESTWESTERN 4 ACTIVITY_CNT_L12 A 3
M AIRTRAN 4 ACTIVITY_CNT_L12 A 3
D BESTWESTERN FIRSTNAME LASTNAME 209 N SANBORN AVE
D BESTWESTERN FIRSTNAME LASTNAME 6997 COUNTY ROAD D
D AIRTRAN FIRSTNAME LASTNAME 6997 COUNTY ROAD D
S BESTWESTERN 2
S AIRTRAN 2
I have split the file into three different files.
Metadata file
Detail file
Summary file
The challenge is to check if the information in Metadata records exist in the Detail record file. The names are not constant and WILL change with every incoming file.
1) The script needs to dynamically check the column in the Metadata record file that contains, for example 'BESTWESTERN' and 'AIRTRAN' and make sure that it also exists in the detail record file.
This is a huge file and need to know the fastest way to process it.
What is the best way to approach this dynamically changing file?
Please advice...
Thank You,
Madhu
10 More Discussions You Might Find Interesting
1. UNIX for Advanced & Expert Users
I am in desperate need of an answer to this question. I have looked everywhere (even the man pages) and found very little.
Solaris has the concept of "plumbing" a network interface. What does this mean?
I would be really greatful to whoever could help me answer this question. I am so... (1 Reply)
Discussion started by: Sparticus007
1 Replies
2. Filesystems, Disks and Memory
Hello,
I have a Supermicro server with a P4SCI mother board running Debian Sarge 3.1. This is the "dmidecode" output related to RAM info:
RAM speed information is incomplete.. "Current Speed: Unknown", is there anyway/soft to get the speed of installed RAM modules? thanks!!
Regards :)... (0 Replies)
Discussion started by: Santi
0 Replies
3. Shell Programming and Scripting
Hi,
I have a script xyz.ksh which accpets two parameters the format of first one is :X_TABLENAME_Y and second one is a digit. I can extract a table name from that parameter and store it in a variable var_tblnm, so if i pass a parameter X_TABLE1_Y the value in var_tblenm is "TABLE1"
now i have... (1 Reply)
Discussion started by: hcdiss
1 Replies
4. Shell Programming and Scripting
Hi all,
How do I compare contents of entire two files except for the first line is each of them? I am sure first lines from both my files are going to be different so I want to ignore them.
Is there a easier way than creating temporary files by cutting out the first line and then comparing... (1 Reply)
Discussion started by: jakSun8
1 Replies
5. Shell Programming and Scripting
Hello All,
I am new to this forum, and the reason I came here is to seek solution from the experts.
I have written following wrapper script, it was running fine from past couple of months, until last week.
When one of the function in the script which suppose to login through ssh to the... (2 Replies)
Discussion started by: tajdar
2 Replies
6. Filesystems, Disks and Memory
I analysed disk performance with blktrace and get some data:
read:
8,3 4 2141 2.882115217 3342 Q R 195732187 + 32
8,3 4 2142 2.882116411 3342 G R 195732187 + 32
8,3 4 2144 2.882117647 3342 I R 195732187 + 32
8,3 4 2145 ... (1 Reply)
Discussion started by: W.C.C
1 Replies
7. Shell Programming and Scripting
I'm writing a bash shell script to 'help' me post to susepaste (I can NEVER remember the time options).
Here's the code:
#!/bin/bash
##########
#
# Project : personal script.
# Started : Wed Aug 03, 2011
# Author : Habitual
# Description : susepaste c-li script with user... (5 Replies)
Discussion started by: Habitual
5 Replies
8. Shell Programming and Scripting
Hi,
My input file contains
1,2
2,4
3,6
4,9
9,10
My expected output is
1,10
2,10
3,6
4,1
9,10 (6 Replies)
Discussion started by: pandeesh
6 Replies
9. Programming
Hi,
On a hosted linux environment which I have very little control over, I have a PHP script that takes in X number of floats, performs Y number of simple recursive arithmetic calculations, and produces some output for display to the user.
When I first created the script, 'X' and 'Y' were... (4 Replies)
Discussion started by: MoreCowbell
4 Replies
10. Shell Programming and Scripting
The below awk verifies the count and order of each text file in the directory. The script does execute and produce output, however the order of the headers are not compared to key. The portion in bold is supposed to do that. If the order of the headers in each text file is the same as key, then... (0 Replies)
Discussion started by: cmccabe
0 Replies
LEARN ABOUT DEBIAN
colorgccrc
COLORGCCRC(5) File Formats Manual COLORGCCRC(5)
NAME
colorgccrc - configuration file for colorgcc
DESCRIPTION
A colorgccrc configuration file is used to configure the highlighting of the compiler output from colorgcc.
SYNTAX
Each line consists of a keyword designating a configuration variable. The keyword is followed by `:' and then one or several values
(depending on the keyword). Lines beginning with a hash mark `#' are comments.
CONFIGURATION VARIABLES
g++ | gcc | c++ | cc | g77 | gcj | gnat | gpc
Specifies the paths to the compilers. Takes one value; a path to the compiler.
nocolor
Specifies what terminal types colorization should be disabled on. Takes one or several values, separated by whitespace.
srcColor
Specifies the highlighting attributes source-code should be given. Takes one or several color attributes. See the section COLOR
ATTRIBUTES for more information.
introColor
Specifies the highlighting attributes for normal compiler output. Takes one or several color attributes. See the section COLOR
ATTRIBUTES for more information.
warningFileNameColor | errorFileNameColor
Specifies the highlighting attributes for the filename in a warning or an error, respectively. Takes one or several color
attributes. See the section COLOR ATTRIBUTES for more information.
warningNumberColor | errorNumberColor
Specifies the highlighting attributes for the line-number in a warning or an error, respectively. Takes one or several color
attributes. See the section COLOR ATTRIBUTES for more information.
warningMessageColor | errorMessageColor
Specifies the highlighting attributes for the message-text in a warning or an error, respectively. Takes one or several color
attributes. See the section COLOR ATTRIBUTES for more information.
COLOR ATTRIBUTES
The following attributes are valid for highlighting.
clear, reset
bold, underline, underscore, blink, reverse, concealed
black, red, green, yellow, blue, magenta, cyan, white
on_black, on_red, on_green, on_yellow, on_blue, on_magenta, on_cyan, on_white
SEE ALSO
gcc(1), colorgcc(1)
HISTORY
Jan 15 2003: Initial version of this manual-page.
REPORTING BUGS
Report bugs to <jmoyers@geeks.com>
AUTHORS
Jamie Moyers <jmoyers@geeks.com> is the author of colorgcc.
This manual page was written by Joe Wreschnig <piman@sacredchao.net>, and modified by David Weinehall <tao@debian.org>, for the Debian
GNU/Linux system (but may be used by others).
COPYRIGHT
Copyright (C) 2003 Jamie Moyers
This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICU-
LAR PURPOSE.
Jan 15, 2003 COLORGCCRC(5)