Sponsored Content
Top Forums Shell Programming and Scripting Challenging Compare and validate question -- plus speed. Post 302074287 by madhunk on Monday 22nd of May 2006 12:14:16 PM
Old 05-22-2006
Challenging Compare and validate question -- plus speed.

I have a tab delimited HUGE file (13 million records) with Detail, Metadata and Summary records.

Sample File looks like this

M BESTWESTERN 4 ACTIVITY_CNT_L12 A 3
M AIRTRAN 4 ACTIVITY_CNT_L12 A 3
D BESTWESTERN FIRSTNAME LASTNAME 209 N SANBORN AVE
D BESTWESTERN FIRSTNAME LASTNAME 6997 COUNTY ROAD D
D AIRTRAN FIRSTNAME LASTNAME 6997 COUNTY ROAD D
S BESTWESTERN 2
S AIRTRAN 2

I have split the file into three different files.

Metadata file
Detail file
Summary file

The challenge is to check if the information in Metadata records exist in the Detail record file. The names are not constant and WILL change with every incoming file.

1) The script needs to dynamically check the column in the Metadata record file that contains, for example 'BESTWESTERN' and 'AIRTRAN' and make sure that it also exists in the detail record file.

This is a huge file and need to know the fastest way to process it.

What is the best way to approach this dynamically changing file?
Please advice...

Thank You,
Madhu
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Very Challenging Question! Need help bad!

I am in desperate need of an answer to this question. I have looked everywhere (even the man pages) and found very little. Solaris has the concept of "plumbing" a network interface. What does this mean? I would be really greatful to whoever could help me answer this question. I am so... (1 Reply)
Discussion started by: Sparticus007
1 Replies

2. Filesystems, Disks and Memory

dmidecode, RAM speed = "Current Speed: Unknown"

Hello, I have a Supermicro server with a P4SCI mother board running Debian Sarge 3.1. This is the "dmidecode" output related to RAM info: RAM speed information is incomplete.. "Current Speed: Unknown", is there anyway/soft to get the speed of installed RAM modules? thanks!! Regards :)... (0 Replies)
Discussion started by: Santi
0 Replies

3. Shell Programming and Scripting

Challenging!! Help needed

Hi, I have a script xyz.ksh which accpets two parameters the format of first one is :X_TABLENAME_Y and second one is a digit. I can extract a table name from that parameter and store it in a variable var_tblnm, so if i pass a parameter X_TABLE1_Y the value in var_tblenm is "TABLE1" now i have... (1 Reply)
Discussion started by: hcdiss
1 Replies

4. Shell Programming and Scripting

Compare files question

Hi all, How do I compare contents of entire two files except for the first line is each of them? I am sure first lines from both my files are going to be different so I want to ignore them. Is there a easier way than creating temporary files by cutting out the first line and then comparing... (1 Reply)
Discussion started by: jakSun8
1 Replies

5. Shell Programming and Scripting

Need help with this challenging code....

Hello All, I am new to this forum, and the reason I came here is to seek solution from the experts. I have written following wrapper script, it was running fine from past couple of months, until last week. When one of the function in the script which suppose to login through ssh to the... (2 Replies)
Discussion started by: tajdar
2 Replies

6. Filesystems, Disks and Memory

data from blktrace: read speed V.S. write speed

I analysed disk performance with blktrace and get some data: read: 8,3 4 2141 2.882115217 3342 Q R 195732187 + 32 8,3 4 2142 2.882116411 3342 G R 195732187 + 32 8,3 4 2144 2.882117647 3342 I R 195732187 + 32 8,3 4 2145 ... (1 Reply)
Discussion started by: W.C.C
1 Replies

7. Shell Programming and Scripting

Another validate input Question.

I'm writing a bash shell script to 'help' me post to susepaste (I can NEVER remember the time options). Here's the code: #!/bin/bash ########## # # Project : personal script. # Started : Wed Aug 03, 2011 # Author : Habitual # Description : susepaste c-li script with user... (5 Replies)
Discussion started by: Habitual
5 Replies

8. Shell Programming and Scripting

Challenging scenario

Hi, My input file contains 1,2 2,4 3,6 4,9 9,10 My expected output is 1,10 2,10 3,6 4,1 9,10 (6 Replies)
Discussion started by: pandeesh
6 Replies

9. Programming

Generic speed question (PHP vs.)

Hi, On a hosted linux environment which I have very little control over, I have a PHP script that takes in X number of floats, performs Y number of simple recursive arithmetic calculations, and produces some output for display to the user. When I first created the script, 'X' and 'Y' were... (4 Replies)
Discussion started by: MoreCowbell
4 Replies

10. Shell Programming and Scripting

awk to compare files and validate order of headers

The below awk verifies the count and order of each text file in the directory. The script does execute and produce output, however the order of the headers are not compared to key. The portion in bold is supposed to do that. If the order of the headers in each text file is the same as key, then... (0 Replies)
Discussion started by: cmccabe
0 Replies
shells(4)							   File Formats 							 shells(4)

NAME
shells - shell database SYNOPSIS
/etc/shells DESCRIPTION
The shells file contains a list of the shells on the system. Applications use this file to determine whether a shell is valid. See getuser- shell(3C). For each shell a single line should be present, consisting of the shell's path, relative to root. A hash mark (#) indicates the beginning of a comment; subsequent characters up to the end of the line are not interpreted by the routines which search the file. Blank lines are also ignored. The following default shells are used by utilities: /bin/bash, /bin/csh, /bin/jsh, /bin/ksh, /bin/pfcsh, /bin/pfksh, /bin/pfsh, /bin/sh, /bin/tcsh, /bin/zsh, /sbin/jsh, /sbin/sh, /usr/bin/bash, /usr/bin/csh, /usr/bin/jsh, /usr/bin/ksh, /usr/bin/pfcsh, /usr/bin/pfksh, /usr/bin/pfsh, and /usr/bin/sh, /usr/bin/tcsh, /usr/bin/zsh. Note that /etc/shells overrides the default list. Invalid shells in /etc/shells may cause unexpected behavior (such as being unable to log in by way of ftp(1)). FILES
/etc/shells lists shells on system SEE ALSO
vipw(1B), ftpd(1M), sendmail(1M), getusershell(3C), aliases(4) SunOS 5.10 4 Jun 2001 shells(4)
All times are GMT -4. The time now is 06:39 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy