10-24-2008
Joey,
I have tried your logic. It works but involves huges amount of processing & time while processing millions of records.
#!bin/ksh
date
delimiter_char=","
last_field_num=6
file_name=2.txt
col_count=$last_field_num
while read line; do
act_count=`echo $line | tr ${delimiter_char} "\n" | wc -l | awk '{print $1}'`
if [ $col_count -ne $act_count ]; then
echo $line >> ${file_name%.txt}.bad
else
echo $line >> ${file_name%.txt}.good
fi
done < ${file_name}
---------------------------------------------------------------
Avis, Your logic works fine, But i couldn't use it either. Coz i couldn't make UDF out of it. Say i have many files, Number of columns vary
Yours is good at performance
grep -e "^[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*$" 2.txt > 3.good &
pid_good=$!
grep -v "^[^,]*,[^,]*,[^,]*,[^,]*,[^,]*,[^,]*$" 2.txt > 3.bad &
pid_bad=$!
wait $pid_good
wait $pid_bad
Note: Worked on a CSV File. Any unicode character might appear in my data except the delimiter. ;-)
Any other ideas??????
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
I am very new to scripting in UNIX and in need of help. I am creating a program that will check a file that has a target time in the form of HH:MM:SS before another program can begin executing. The file with the target time will only have that target time in it and nothing else.
Is there any way... (1 Reply)
Discussion started by: mosammey
1 Replies
2. UNIX for Dummies Questions & Answers
I am very new to scripting in UNIX and in need of help. I am creating a program that will check a file that has a target time in the form of HH:MM:SS before another program can begin executing. The file with the target time will only have that target time in it and nothing else.
Is there any... (4 Replies)
Discussion started by: mosammey
4 Replies
3. Shell Programming and Scripting
Hi,
I have a UNIX script which has two parts:
1. It connects to a database and refreshes a materialized view
2. It then connects to another database and inserts refresh statistics to a table
The script works, but I'm not too good at UNIX validation. Currently, if the first part of the job... (1 Reply)
Discussion started by: matchey
1 Replies
4. UNIX for Dummies Questions & Answers
is there any way to check null data against some of the column in file
My file have such structure
1,,4,SUMISHO ,SMG110880 ,1,12,SUMISHO CAPITAL MANAGEMENT (SINGAPORE) PTE LTD ,ACCT01,20080531,2008,5,30,20080630,1,1,TXGRP ,CGST ,1,74,5.18,74,0,5.18... (2 Replies)
Discussion started by: u263066
2 Replies
5. UNIX for Dummies Questions & Answers
Hi All,
I got a file with 3 fields delimited by hyphen "-". I have to validate & cleanse the data before i begine the processing
Requirements
1. No record should contain more than 2 delimiters
2. No record should even contain less than 2 delimiters
3. Any records that matches rule 1 &... (1 Reply)
Discussion started by: minnuverma
1 Replies
6. UNIX for Advanced & Expert Users
Hi All,
I got a file with 3 fields delimited by hyphen "-". I have to validate & cleanse the data before i begine the processing
Requirements
1. No record should contain more than 2 delimiters
2. No record should even contain less than 2 delimiters
3. Any records that matches rule 1 &... (3 Replies)
Discussion started by: minnuverma
3 Replies
7. Shell Programming and Scripting
I have a script which is take date as parameter
sh abc.sh <2010-02-01>
#!/sh/bin
my_date=$1
#Here i want to two diffrent dates
## 3 Days before
##date14query=$mydate - 4 (it will be 2010-01-28)
##date24query=$mydate +4 (it will be 2010-01-05)
#Please Help (3 Replies)
Discussion started by: pritish.sas
3 Replies
8. UNIX for Dummies Questions & Answers
Dears,
I am working on a batch that processes file with name containing date prefix eg., 20101222_file.dat.
The logic is to process files in order. Eg., 20101225 must be processed only after 20101222.
Ok first glance it looked simple, it use a variable to check this date value as number and... (2 Replies)
Discussion started by: naraink
2 Replies
9. Shell Programming and Scripting
Hi All,
I am completely new to Unix Shell Scripting.
I m validating(Basic File Validation) an .HHT file in TIBCO.
After that i need to do Data Validation through UNIX Shell scripting.
Rules in DataValidation:
1.) Need to Check Every field wheather it is a Char or Number?(Fields are... (1 Reply)
Discussion started by: Chaitanya K
1 Replies
10. Shell Programming and Scripting
Please help me to validate PAN card using unix
---------- Post updated at 03:06 AM ---------- Previous update was at 02:21 AM ----------
1) The first three letters are sequence of alphabets from AAA to zzz
2) The fourth character informs about the type of holder of the Card. Each assesse... (1 Reply)
Discussion started by: suryanarayana
1 Replies
LEARN ABOUT LINUX
comp_err
COMP_ERR(1) MySQL Database System COMP_ERR(1)
NAME
comp_err - compile MySQL error message file
SYNOPSIS
comp_err [options]
DESCRIPTION
comp_err creates the errmsg.sys file that is used by mysqld to determine the error messages to display for different error codes. comp_err
normally is run automatically when MySQL is built. It compiles the errmsg.sys file from the plaintext file located at sql/share/errmsg.txt
in MySQL source distributions.
comp_err also generates mysqld_error.h, mysqld_ername.h, and sql_state.h header files.
For more information about how error messages are defined, see the MySQL Internals Manual[1].
Invoke comp_err like this:
shell> comp_err [options]
comp_err supports the following options.
o --help, -?
Display a help message and exit.
o --charset=path, -C path
The character set directory. The default is ../sql/share/charsets.
o --debug=debug_options, -# debug_options
Write a debugging log. A typical debug_options string is d:t:O,file_name. The default is d:t:O,/tmp/comp_err.trace.
o --debug-info, -T
Print some debugging information when the program exits.
o --header_file=file_name, -H file_name
The name of the error header file. The default is mysqld_error.h.
o --in_file=file_name, -F file_name
The name of the input file. The default is ../sql/share/errmsg.txt.
o --name_file=file_name, -N file_name
The name of the error name file. The default is mysqld_ername.h.
o --out_dir=path, -D path
The name of the output base directory. The default is ../sql/share/.
o --out_file=file_name, -O file_name
The name of the output file. The default is errmsg.sys.
o --statefile=file_name, -S file_name
The name for the SQLSTATE header file. The default is sql_state.h.
o --version, -V
Display version information and exit.
COPYRIGHT
Copyright (C) 1997, 2014, Oracle and/or its affiliates. All rights reserved.
This documentation is free software; you can redistribute it and/or modify it only under the terms of the GNU General Public License as
published by the Free Software Foundation; version 2 of the License.
This documentation is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with the program; if not, write to the Free Software Foundation,
Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA or see http://www.gnu.org/licenses/.
NOTES
1. MySQL Internals Manual
http://dev.mysql.com/doc/internals/en
SEE ALSO
For more information, please refer to the MySQL Reference Manual, which may already be installed locally and which is also available online
at http://dev.mysql.com/doc/.
AUTHOR
Oracle Corporation (http://dev.mysql.com/).
MySQL 5.5 01/30/2014 COMP_ERR(1)