Sponsored Content
Top Forums UNIX for Advanced & Expert Users Wanted best way to validate delimited file records Post 302718299 by DGPickett on Friday 19th of October 2012 12:06:41 PM
Old 10-19-2012
If you just want to validate the general form of records, sed gives you regex and demux:
Code:
sed '
  /^[A-Z][a-z]\{3\}|[a-z]\{8\}|[a-z]\{10\}|[a-z]\{7\}|[a-z]\{6\}|$/{
    w good_recs
    d
    }
 ' in_file >bad_recs

You just need to generate the regex from your file specs.

For bulk processing speed, write a simple C program that reads lines, checks the line length, locates all pipes (offsets in a big array of integers), which can be checked against field count and field length command line arguments, and any other field filters you desire can be added integer, decimal, float, text, no white space, upper case, etc.). Of course, you could have standard regex for each field type you care to filter.

Last edited by DGPickett; 10-19-2012 at 01:13 PM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Converting Tab delimited file to Comma delimited file in Unix

Hi, Can anyone let me know on how to convert a Tab delimited file to Comma delimited file in Unix Thanks!! (22 Replies)
Discussion started by: charan81
22 Replies

2. Shell Programming and Scripting

Delete Duplicate records from a tilde delimited file

Hi All, I want to delete duplicate records from a tilde delimited file. Criteria is considering the first 2 fields, the combination of which has to be unique, below is a sample of records in the input file 1620000010338~2446694087~0~20061130220000~A00BCC1CT... (5 Replies)
Discussion started by: irshadm
5 Replies

3. UNIX for Dummies Questions & Answers

Extract records by column value - file non-delimited

the data in my file is has no delimiters. it looks like this: H52082320024740010PH333200612290000930 0.0020080131 D5208232002474000120070306200703060580T1502 TT 1.00 H52082320029180003PH333200702150001 30 100.0020080205 D5208232002918000120070726200707260580T1502 ... (3 Replies)
Discussion started by: jclanc8
3 Replies

4. Shell Programming and Scripting

How to convert a space delimited file into a pipe delimited file using shellscript?

Hi All, I have space delimited file similar to the one as shown below.. I need to convert it as a pipe delimited, the values inside the pipe delimited file should be as highlighted... AA ATIU2345098809 009697 005374 BB ATIU2345097809 005445 006518 CC ATIU9685098809 003215 003571 DD... (7 Replies)
Discussion started by: nithins007
7 Replies

5. Shell Programming and Scripting

how to Insert values in multiple lines(records) within a pipe delimited text file in specific cols

this is Korn shell unix. The scenario is I have a pipe delimited text file which needs to be customized. say for example,I have a pipe delimited text file with 15 columns(| delimited) and 200 rows. currently the 11th and 12th column has null values for all the records(there are other null columns... (4 Replies)
Discussion started by: vasan2815
4 Replies

6. Shell Programming and Scripting

Print records which do not have expected number of fields in a comma delimited file

Hi, I have a comma (,) delimited file, in which few fields are enclosed with in double quotes " ". I have to print the records in the file which donot have expected number of field with the line number. File1 ==== name,desgnation,doj,project #header#... (7 Replies)
Discussion started by: machomaddy
7 Replies

7. Shell Programming and Scripting

what is the better way to validate records in a file.

hi all, We are checking for the delimited file records validation Delimited file will have data like this: Aaaa|sdfhxfgh|sdgjhxfgjh|sdgjsdg|sgdjsg| Aaaa|sdfhxfgh|sdgjhxfgjh|sdgjsdg|sgdjsg| Aaaa|sdfhxfgh|sdgjhxfgjh|sdgjsdg|sgdjsg| Aaaa|sdfhxfgh|sdgjhxfgjh|sdgjsdg|sgdjsg| So we are... (4 Replies)
Discussion started by: Seshendranath
4 Replies

8. UNIX for Dummies Questions & Answers

How to convert a comma delimited string to records or lines of text?

Hi, I am not sure if I've posted this question before. Anyway, I previously asked about converting lines of text into a comma delimited string. Now I am needing to do the other way around ... :( :o Can anyone advise how is this possible? Example as below: Converting records/lines to... (2 Replies)
Discussion started by: newbie_01
2 Replies

9. Shell Programming and Scripting

Delimited records splitted into different lines

Hi I am using delimited sequence file. Delimter we are using is pipe .But for some of the records for one of the column the values are getting split into different lines as shown below "113"|"0155"|"2016-04-27 07:59:04"|"1930"|"TEST@TEST"|"2016-04-27 11:04:04.357000000"|"BO"|"Hard... (13 Replies)
Discussion started by: ginrkf
13 Replies

10. UNIX for Beginners Questions & Answers

Need help on an old post - How to convert a comma delimited string to records or lines of text?

Hi, Apologies in advance to the moderator if I am posting this the wrong way. I've searched and found the solution to an old post but as it is a very old post, I don't see an option to update it with additional question. The question I have is in relation to the following post: How to... (6 Replies)
Discussion started by: newbie_01
6 Replies
ACHECK-RULES.5(5)					User Contributed Perl Documentation					 ACHECK-RULES.5(5)

NAME
set.rules - Rules set for acheck DESCRIPTION
Rules set files contain rules to be check by acheck. Lines beginning with a number sign (`#') and empty lines will be ignored. Spaces at the beginning and the end of a line will also be ignored as well as tabulators. If you need spaces at the end or the beginning of a value you can use apostrophes (`"'). A comment starts with the number sign, there can be any number of spaces and/or tab stops in front of the #. Long lines can be broken into multiple lines ending with a backslash (`'). Some possible examples: # this line is ignored field value field value # this is a comment field "value ending with space " field value continuing on the next line You have to escape number signs with a backslash to use it in a value and use apostrophes if a value ends with a backslash. Rule sets are made of lists of rules. Theses lists can be repeated a number of times, until or while a condition happens. A rule detects an error if the corresponding test succeeds and none of its validation tests does. Each rule can then produce some fixes, a warning or an error, and provide hints to help the operator to correct the error. Parts of the text can be set as comments and so no rule and no spell check will be performed on them. RULE FILE SYNTAX
SYNTAX RULES
list A list starts at a `list' statement, and stops at the first `end_list' or at the end of the file. Mandatory fields: type type until, perform the list until the current line matches `test' while, perform the lint while the current line matches `test' loop, perform the list `test' times test regex / number A regex for `until' and `while' lists. The number of times the list must be performed for `loop', or `0' for infinite loop. Optional fields: name name Use this to give the list a name. spell yes / no Set it to `yes' (default) or `no', if you want or don't want spelling to be checked in the lines matching this list. This value will be inherited by the nested lists. Sub-rules: list [name] rule [name] comment [name] Followed by the name of the sub-rule or its definition for an anonymous sub-rule. rule A rule starts at a `rule' statement, and stops at the first `end_rule' or at the beginning of a comment or a list. Mandatory fields: type type fix, rule provides fixes and hints thought a menu autofix, rule fixes the mistake with no interaction warning, rule issues a warning error, rule issues an error nop, special rule that do nothing, no other field is required regex regex The regex to be match to found this error. Patterns can be captures and then used in the `fix' expression. fix expr Provides a correction for the rule, this field can be repeated to provides more than one choice. Only the first one will be used for `autofix' rules. `warning' and `error' do not provides fixes. The captured patterns can be used here with variables `$1', `$2', and so on. hint text Provides some explanations, this will be used as reviewer comments in review mode. Optional fields: name name Use this to give the rule a name. valid [name] Provides a validation test, it can be named or anonymous. For anonymous validation, the test definition must follow. This field can be repeated more than once, if any of the validation test succeed, the rule does not apply. valid A validation test starts at a `valid' statement, and stops at the first `end_valid' or at the beginning of a rule, a comment, a list or another validation test. Mandatory field: pre regex in regex> post regex Try the regex before, in or after the match of the regex rule. At least one of these test must be provided. If all tests are success- ful, the rule won't apply. Optional field: name name Use this to give the validation test a name. comment A comment starts at a `comment' statement, and stops at the first `end_comment' or at the beginning of a rule, a comment or a list. Comments are just skipped, no other rule and no spelling are performed on them. Mandatory field: skip regex A regex matching the text of the comment. Do not use `start' and `stop' with this. start regex stop regex Regexs defining the beginning and the end of the comment, all the text between will be considered as comment. Do not use `skip' with this. Optional fields: name name Use this to give the comment a name. start_offset stop_offset Defines where the comment really starts or end. Values are `s' for the place the match starts, `s+<n>' for n characters after the match starts, `e' for the place the match ends, or `e-<n>' for n characters before the match ends. Defaults are `s' for `start' matches and `e' for `stop' matches. SEE ALSO
acheck(1), acheck(5) AUTHOR
Nicolas Bertolissio <nico.bertol@free.fr> perl v5.8.4 2003-07-20 ACHECK-RULES.5(5)
All times are GMT -4. The time now is 01:49 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy