Sponsored Content
Full Discussion: Unix File Validation! Help
Top Forums Shell Programming and Scripting Unix File Validation! Help Post 302250739 by minnuverma on Friday 24th of October 2008 07:05:15 AM
Old 10-24-2008
Joey,

I have tried your logic. It works but involves huges amount of processing & time while processing millions of records.

#!bin/ksh
date
delimiter_char=","
last_field_num=6
file_name=2.txt
col_count=$last_field_num
while read line; do
act_count=`echo $line | tr ${delimiter_char} "\n" | wc -l | awk '{print $1}'`
if [ $col_count -ne $act_count ]; then
echo $line >> ${file_name%.txt}.bad
else
echo $line >> ${file_name%.txt}.good
fi
done < ${file_name}
---------------------------------------------------------------

Avis, Your logic works fine, But i couldn't use it either. Coz i couldn't make UDF out of it. Say i have many files, Number of columns vary

Yours is good at performance
grep -e "^[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*$" 2.txt > 3.good &
pid_good=$!
grep -v "^[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*$" 2.txt > 3.bad &
pid_bad=$!
wait $pid_good
wait $pid_bad

Note: Worked on a CSV File. Any unicode character might appear in my data except the delimiter. ;-)


Any other ideas??????
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Time Validation in UNIX?

I am very new to scripting in UNIX and in need of help. I am creating a program that will check a file that has a target time in the form of HH:MM:SS before another program can begin executing. The file with the target time will only have that target time in it and nothing else. Is there any way... (1 Reply)
Discussion started by: mosammey
1 Replies

2. UNIX for Dummies Questions & Answers

How Can I Do Time Validation in UNIX

I am very new to scripting in UNIX and in need of help. I am creating a program that will check a file that has a target time in the form of HH:MM:SS before another program can begin executing. The file with the target time will only have that target time in it and nothing else. Is there any... (4 Replies)
Discussion started by: mosammey
4 Replies

3. Shell Programming and Scripting

UNIX script Validation

Hi, I have a UNIX script which has two parts: 1. It connects to a database and refreshes a materialized view 2. It then connects to another database and inserts refresh statistics to a table The script works, but I'm not too good at UNIX validation. Currently, if the first part of the job... (1 Reply)
Discussion started by: matchey
1 Replies

4. UNIX for Dummies Questions & Answers

validation required in unix

is there any way to check null data against some of the column in file My file have such structure 1,,4,SUMISHO ,SMG110880 ,1,12,SUMISHO CAPITAL MANAGEMENT (SINGAPORE) PTE LTD ,ACCT01,20080531,2008,5,30,20080630,1,1,TXGRP ,CGST ,1,74,5.18,74,0,5.18... (2 Replies)
Discussion started by: u263066
2 Replies

5. UNIX for Dummies Questions & Answers

Unix File Validation! Help

Hi All, I got a file with 3 fields delimited by hyphen "-". I have to validate & cleanse the data before i begine the processing Requirements 1. No record should contain more than 2 delimiters 2. No record should even contain less than 2 delimiters 3. Any records that matches rule 1 &... (1 Reply)
Discussion started by: minnuverma
1 Replies

6. UNIX for Advanced & Expert Users

Unix File Validation! Help

Hi All, I got a file with 3 fields delimited by hyphen "-". I have to validate & cleanse the data before i begine the processing Requirements 1. No record should contain more than 2 delimiters 2. No record should even contain less than 2 delimiters 3. Any records that matches rule 1 &... (3 Replies)
Discussion started by: minnuverma
3 Replies

7. Shell Programming and Scripting

Date Validation in unix

I have a script which is take date as parameter sh abc.sh <2010-02-01> #!/sh/bin my_date=$1 #Here i want to two diffrent dates ## 3 Days before ##date14query=$mydate - 4 (it will be 2010-01-28) ##date24query=$mydate +4 (it will be 2010-01-05) #Please Help (3 Replies)
Discussion started by: pritish.sas
3 Replies

8. UNIX for Dummies Questions & Answers

Unix date validation

Dears, I am working on a batch that processes file with name containing date prefix eg., 20101222_file.dat. The logic is to process files in order. Eg., 20101225 must be processed only after 20101222. Ok first glance it looked simple, it use a variable to check this date value as number and... (2 Replies)
Discussion started by: naraink
2 Replies

9. Shell Programming and Scripting

Help With UNIX Shell Scripting For Data Validation

Hi All, I am completely new to Unix Shell Scripting. I m validating(Basic File Validation) an .HHT file in TIBCO. After that i need to do Data Validation through UNIX Shell scripting. Rules in DataValidation: 1.) Need to Check Every field wheather it is a Char or Number?(Fields are... (1 Reply)
Discussion started by: Chaitanya K
1 Replies

10. Shell Programming and Scripting

PAN card validation using UNIX

Please help me to validate PAN card using unix ---------- Post updated at 03:06 AM ---------- Previous update was at 02:21 AM ---------- 1) The first three letters are sequence of alphabets from AAA to zzz 2) The fourth character informs about the type of holder of the Card. Each assesse... (1 Reply)
Discussion started by: suryanarayana
1 Replies
COMP_ERR(1)						       MySQL Database System						       COMP_ERR(1)

NAME
comp_err - compile MySQL error message file SYNOPSIS
comp_err [options] DESCRIPTION
comp_err creates the errmsg.sys file that is used by mysqld to determine the error messages to display for different error codes. comp_err normally is run automatically when MySQL is built. It compiles the errmsg.sys file from the plaintext file located at sql/share/errmsg.txt in MySQL source distributions. comp_err also generates mysqld_error.h, mysqld_ername.h, and sql_state.h header files. For more information about how error messages are defined, see the MySQL Internals Manual[1]. Invoke comp_err like this: shell> comp_err [options] comp_err supports the following options. o --help, -? Display a help message and exit. o --charset=path, -C path The character set directory. The default is ../sql/share/charsets. o --debug=debug_options, -# debug_options Write a debugging log. A typical debug_options string is d:t:O,file_name. The default is d:t:O,/tmp/comp_err.trace. o --debug-info, -T Print some debugging information when the program exits. o --header_file=file_name, -H file_name The name of the error header file. The default is mysqld_error.h. o --in_file=file_name, -F file_name The name of the input file. The default is ../sql/share/errmsg.txt. o --name_file=file_name, -N file_name The name of the error name file. The default is mysqld_ername.h. o --out_dir=path, -D path The name of the output base directory. The default is ../sql/share/. o --out_file=file_name, -O file_name The name of the output file. The default is errmsg.sys. o --statefile=file_name, -S file_name The name for the SQLSTATE header file. The default is sql_state.h. o --version, -V Display version information and exit. COPYRIGHT
Copyright (C) 1997, 2014, Oracle and/or its affiliates. All rights reserved. This documentation is free software; you can redistribute it and/or modify it only under the terms of the GNU General Public License as published by the Free Software Foundation; version 2 of the License. This documentation is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with the program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA or see http://www.gnu.org/licenses/. NOTES
1. MySQL Internals Manual http://dev.mysql.com/doc/internals/en SEE ALSO
For more information, please refer to the MySQL Reference Manual, which may already be installed locally and which is also available online at http://dev.mysql.com/doc/. AUTHOR
Oracle Corporation (http://dev.mysql.com/). MySQL 5.5 01/30/2014 COMP_ERR(1)
All times are GMT -4. The time now is 03:38 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy