06-07-2007
Help with a Flat File!!!
Hi All,
I need a help with a shelll script program.I get a '|' separated file which sometime has a encrypted column.And this encryption sometime breaks the last column data into a new line and this is not picked by the ETL.So when i run a script,it should append back the broken new line data back to its end of record and if there is any empty row,just delete them.
Input:
xxxxx|Jun 5 2007 11:57:27:486PM|1014aaaa|xxx|dddd|aa|qwq|2cvcxv| |TP|#$@1
%%^&&&
xxxxx|Jun 5 2007 11:57:27:486PM|101aa05|xxx|dddd|aa|qwq|2cvcxv| |TP|#$@1%%^&&&
xxxxx|Jun 5 2007 11:57:27:486PM|1014aa05|xxx|dddd|aa|qwq|2cvcxv| |TP|#$
@1%%^&&&
xxxxx|Jun 5 2007 11:57:27:486PM|10aa2905|xxx|dddd|aa|qwq|2cvcxv| |TP|#$@1%%^&&&
Output :
xxxxx|Jun 5 2007 11:57:27:486PM|1014aaaa|xxx|dddd|aa|qwq|2cvcxv| |TP|#$@1%%^&&&
xxxxx|Jun 5 2007 11:57:27:486PM|1014aa05|xxx|dddd|aa|qwq|2cvcxv| |TP|#$@1%%^&&&
xxxxx|Jun 5 2007 11:57:27:486PM|10aa2905|xxx|dddd|aa|qwq|2cvcxv| |TP|#$@1%%^&&&
xxxxx|Jun 5 2007 11:57:27:486PM|1014aa05|xxx|dddd|aa|qwq|2cvcxv| |TP|#$@1%%^&&&
Your help is appreciated,
Thanks,
Kumar
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Here is on more go ! Need a shortcut for my problem !
problem is i have a look_update with fixed sequence of column
that is :
MANDT:SERAIL:SERSCHA:SEREX:EQTYP:BSTVP
I will be getting data in a flat file having same number of column but the sequence could be different in each... (5 Replies)
Discussion started by: jambesh
5 Replies
2. Shell Programming and Scripting
Hi All,
I get a flat file everyday with some records being invalid.Some records come with less number '|'.I just want to add the missing '|'s ,So the it doesnt give an error insufficient fields while loading...There area total of 32 pipes in each record.
Input:
ASADASD |Y|B|SDFGDSDFD| ... (3 Replies)
Discussion started by: kumarsaravana_s
3 Replies
3. Shell Programming and Scripting
Hi
I have a flat file looking this
QWER 2:35 7044 00001 ROUT 1188 EA SS
ASD 2:36 7044 00010 ROUT
BSD 2:37 7044 00011 ROUT
END
QWER 3:35 7044 00011 ROUT 1088 EA SS
ASD 3:36 7044 00010 ROUT
BSD 3:37 7044 00011 ROUT... (9 Replies)
Discussion started by: Krishnaramjis
9 Replies
4. Shell Programming and Scripting
Hi
I have a file like below . IF the record starts with ASD then change the 20th offset to "K" follwed by that 20th offset value & if the record starts with ASDR then change the 38th offset to "K" followed by 38th offset value . But here the condition is the next value ... (0 Replies)
Discussion started by: Krishnaramjis
0 Replies
5. Shell Programming and Scripting
Hi ,
I have a file with contents as below:
Contract Cancellation Report UARCNCL
LOS CODE DATE REAS TYPE AMOUNT AMOUNT LETTER BY
========= ======= ==== ==== ==== ========= ==== ====
8174739 7641509 1S NONE CRCD 30-JUN-2008 NPAR N .00 .00 CCAN
8678696 8091709 1S NONE DDEB 30-JUN-2008... (14 Replies)
Discussion started by: rkumudha
14 Replies
6. Programming
Before i start doing something, I wanted to know whether the approach to compare XML file with UNIX file system structure. I have a pre-configured file(contains a list of paths to executables) and i need to check against the UNIX directory structure. what are the various approches should i use ? I... (6 Replies)
Discussion started by: shafi2all
6 Replies
7. Shell Programming and Scripting
i have a flat file with around 700 columns . i want to break it into a flat file with say five columns like
col1,col2,col3,col4
col5,col6,col7,col8
col9...... ,col700
how can i do this (5 Replies)
Discussion started by: r_t_1601
5 Replies
8. Shell Programming and Scripting
Need to develop a unix shell script for the below requirement and I need your assistance:
1) search for file.log and file.bad file in a directory and read them
2) pull out "Load_Start_Time", "Data_File_Name", "Error_Type" from log file
4) concatinate each row from bad file as... (3 Replies)
Discussion started by: mlpathir
3 Replies
9. Shell Programming and Scripting
hi
i have written a script for reading a csv file and creating a flat file, suggest if this script can be optimized
#----------------
FILENAME="$1"
SCRIPT=$(basename $0)
#-----------------------------------------//
function usage
{
echo "\nUSAGE: $THIS_SCRIPT file_to_process\n"... (3 Replies)
Discussion started by: mprakasheee
3 Replies
10. UNIX for Beginners Questions & Answers
Hi Team
there is a requirement to do validate on Flat files using Shell Scripting.
Suppose we have two flat files ( F1, F2).
Validation
1. Row count between F1 and F2
a) Write it in a file with results of count of each file and differences if any
2. Apply checksum or any... (1 Reply)
Discussion started by: shlearner
1 Replies
LEARN ABOUT DEBIAN
config::model::loader
Config::Model::Loader(3pm) User Contributed Perl Documentation Config::Model::Loader(3pm)
NAME
Config::Model::Loader - Load serialized data into config tree
VERSION
version 2.021
SYNOPSIS
use Config::Model;
use Log::Log4perl qw(:easy);
Log::Log4perl->easy_init($WARN);
# define configuration tree object
my $model = Config::Model->new;
$model->create_config_class(
name => "Foo",
element => [
[qw/foo bar/] => {
type => 'leaf',
value_type => 'string'
},
]
);
$model ->create_config_class (
name => "MyClass",
element => [
[qw/foo bar/] => {
type => 'leaf',
value_type => 'string'
},
hash_of_nodes => {
type => 'hash', # hash id
index_type => 'string',
cargo => {
type => 'node',
config_class_name => 'Foo'
},
},
[qw/lista listb/] => {
type => 'list',
cargo => {type => 'leaf',
value_type => 'string'
}
},
],
) ;
my $inst = $model->instance(root_class_name => 'MyClass' );
my $root = $inst->config_root ;
# put data
my $step = 'foo=FOO hash_of_nodes:fr foo=bonjour -
hash_of_nodes:en foo=hello
! lista=foo,bar lista:2=baz
listb:0=foo listb:1=baz';
$root->load( step => $step );
print $root->describe,"
" ;
# name value type comment
# foo FOO string
# bar [undef] string
# hash_of_nodes <Foo> node hash keys: "en" "fr"
# lista foo,bar,baz list
# listb foo,baz list
# delete some data
$root->load( step => 'lista~2' );
print $root->describe(element => 'lista'),"
" ;
# name value type comment
# lista foo,bar list
# append some data
$root->load( step => q!hash_of_nodes:en foo.=" world"! );
print $root->grab('hash_of_nodes:en')->describe(element => 'foo'),"
" ;
# name value type comment
# foo "hello world" string
DESCRIPTION
This module is used directly by Config::Model::Node to load serialized configuration data into the configuration tree.
Serialized data can be written by the user or produced by Config::Model::Dumper while dumping data from a configuration tree.
CONSTRUCTOR
new ( )
No parameter. The constructor should be used only by Config::Model::Node.
load string syntax
The string is made of the following items (also called "actions") separated by spaces:
- Go up one node
! Go to the root node of the configuration tree.
xxx Go down using "xxx" element. (For "node" type element)
xxx:yy Go down using "xxx" element and id "yy" (For "hash" or "list" element with "node" cargo_type)
xxx=~/yy/
Go down using "xxx" element and loop over the ids that match the regex. (For "hash")
For instance, with "OpenSsh" model, you could do
Host=~/.*.debian.org/ user='foo-guest'
to set "foo-user" users for all your debian accounts.
xxx~yy Delete item referenced by "xxx" element and id "yy". For a list, this is equivalent to "splice xxx,yy,1". This command does not go
down in the tree (since it has just deleted the element). I.e. a '"-"' is generally not needed afterwards.
xxx=zz Set element "xxx" to value "yy". load also accepts to set elements with a quoted string. (For "leaf" element)
For instance "foo="a quoted string"". Note that you cannot embed double quote in this string. I.e "foo="a "quoted" string"" will
fail.
xxx~ Undef element "xxx"
xxx=z1,z2,z3
Set list element "xxx" to list "z1,z2,z3". Use ",," for undef values, and "" for empty values.
I.e, for a list "('a',undef,'','c')", use "a,,"",c".
xxx:yy=zz
For "hash" element containing "leaf" cargo_type. Set the leaf identified by key "yy" to value "zz".
Using "xxx=~/yy/=zz" is also possible.
xxx.=zzz
Will append "zzz" value to current values (valid for "leaf" elements).
xxx#zzz or xxx:yyy#zzz
Element annotation. Can be quoted or not quoted. Note that annotations are always placed at the end of an action item.
I.e. "foo#comment", "foo:bar#comment" or "foo:bar=baz#comment" are valid. "foo#comment:bar" is not valid.
Quotes
You can surround indexes and values with double quotes. E.g.:
a_string=""titi" and "toto""
Methods
load ( ... )
Load data into the node tree (from the node passed with "node") and fill values as we go following the instructions passed with "step".
("step" can also be an array ref).
Parameters are:
node
node ref of the root of the tree (of sub-root) to start the load from.
step
A string or an array ref containing the steps to load. See above for a description of the string.
experience
Specify the experience level used during the load (default: "master"). The experience can be "intermediate advanced master". The load
will raise an exception if the step of the load string tries to access an element with experience higher than user's experience.
check
Whether to check values while loading. Either "yes" (default), "no" or "skip". Loading with "skip" will discard bad values.
AUTHOR
Dominique Dumont, (ddumont at cpan dot org)
SEE ALSO
Config::Model,Config::Model::Node,Config::Model::Dumper
perl v5.14.2 2012-11-09 Config::Model::Loader(3pm)