Help with a Flat File!!!


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Help with a Flat File!!!
# 1  
Old 06-07-2007
Help with a Flat File!!!

Hi All,

I need a help with a shelll script program.I get a '|' separated file which sometime has a encrypted column.And this encryption sometime breaks the last column data into a new line and this is not picked by the ETL.So when i run a script,it should append back the broken new line data back to its end of record and if there is any empty row,just delete them.

Input:

xxxxx|Jun 5 2007 11:57:27:486PM|1014aaaa|xxx|dddd|aa|qwq|2cvcxv| |TP|#$@1
%%^&&&
xxxxx|Jun 5 2007 11:57:27:486PM|101aa05|xxx|dddd|aa|qwq|2cvcxv| |TP|#$@1%%^&&&

xxxxx|Jun 5 2007 11:57:27:486PM|1014aa05|xxx|dddd|aa|qwq|2cvcxv| |TP|#$
@1%%^&&&
xxxxx|Jun 5 2007 11:57:27:486PM|10aa2905|xxx|dddd|aa|qwq|2cvcxv| |TP|#$@1%%^&&&

Output :

xxxxx|Jun 5 2007 11:57:27:486PM|1014aaaa|xxx|dddd|aa|qwq|2cvcxv| |TP|#$@1%%^&&&
xxxxx|Jun 5 2007 11:57:27:486PM|1014aa05|xxx|dddd|aa|qwq|2cvcxv| |TP|#$@1%%^&&&
xxxxx|Jun 5 2007 11:57:27:486PM|10aa2905|xxx|dddd|aa|qwq|2cvcxv| |TP|#$@1%%^&&&
xxxxx|Jun 5 2007 11:57:27:486PM|1014aa05|xxx|dddd|aa|qwq|2cvcxv| |TP|#$@1%%^&&&

Your help is appreciated,

Thanks,
Kumar
# 2  
Old 06-07-2007
Code:
sed -n "$ p;N;/\n[^|]*$/s/\n//p" filename

# 3  
Old 06-07-2007
Code:
awk '{ getline tmp; if ( tmp !~ /^x/ ) { printf "%s%s\n", $0, tmp } }' filename

# 4  
Old 06-07-2007
Quote:
Originally Posted by anbu23
Code:
sed -n "$ p;N;/\n[^|]*$/s/\n//p" filename

HI Anbu,

When i execute the below command,it gives me an error.I'm working on Sun OS 5.10

> sed -n "$ p;N;/\n[^|]*$/s/\n//p" aaa.Jun-06-2007
Illegal variable name.
>
# 5  
Old 06-07-2007
Quote:
Originally Posted by kumarsaravana_s
HI Anbu,

When i execute the below command,it gives me an error.I'm working on Sun OS 5.10

> sed -n "$ p;N;/\n[^|]*$/s/\n//p" aaa.Jun-06-2007
Illegal variable name.
>
Use single quotes
Code:
sed -n '$ p;N;/\n[^|]*$/s/\n//p' aaa.Jun-06-2007

# 6  
Old 06-07-2007
Quote:
Originally Posted by matrixmadhan
Code:
awk '{ getline tmp; if ( tmp !~ /^x/ ) { printf "%s%s\n", $0, tmp } }' filename

Hi Madhan,
When i run the awk command on the input file,it is not appending the new line data back to the previous record.It is just removing the record leaving the new line data as it is

INput:
aaaaaa|May 22 2007 3:22:10:150PM| |AAAAA |aaaaa|0|0|0|6058|qalog3|[?#è^*Ä?L)
äD?)
11818948|May 22 2007 3:22:10:686PM| |AAAA |3fsdf1|0|0|0|0438|qalog3||(ðÁ] Ë?¢
l8ßfsdfds|May 22 2007 3:22:10:960PM||ASDDE|3112300|0|0|0|7756|qdsd|\bd?¼6?SÉÇ?ß?R2
AASA|May 22 2007 3:22:11:373PM||AAQAA|sdfds0|0|0|7334|fsdfdssd|>¡Ð??iI?;Ç})þ

Output
äD?)
l8ß
fsdfds|May 22 2007 3:22:10:960PM||ASDDE|3112300|0|0|0|7756|qdsd|\bd?¼6?SÉÇ?ß?R2
AASA|May 22 2007 3:22:11:373PM||AAQAA|sdfds0|0|0|7334|fsdfdssd|>¡Ð??iI?;Ç})þ
# 7  
Old 06-07-2007
Quote:
Originally Posted by anbu23
Use single quotes
Code:
sed -n '$ p;N;/\n[^|]*$/s/\n//p' aaa.Jun-06-2007

Hi Anbu,

The Sed command fixes the broken column data but it is omitting the good records too.

Input:
asdasd|asdas|fsdfds|fsdfsd|dgsdfgfsdg|&&^**^(
^*(*(
asdhgasd|ashakljas|ffasdfdsf|Fsfsdfsdf|dfsdfds|&^&*
&^&*^07^%
fksdljflsdkj|lkjjklsd|dflkshjfklsd|sdfjkhsdlf|alfkfhsd|&*^&*%^*&
ajskfhsdkjfh|sdfjklhsdkfj|sdfjklhsd|sdfjsdfhsd|asjhsdf|&^%#@^@
sdfsdfFSDF|SDFJKHFSDJ|wfhywufihwf|wfohfsd|fsduofhsdf|^&$^%#$
#!!&@@
fasdjkgsdfh|fujghsdfjsd|skjfsd|dlskfhlsdjkf|sdjfklsdhf|$#%!&^%!

Output:

asdasd|asdas|fsdfds|fsdfsd|dgsdfgfsdg|&&^**^(^*(*(
asdhgasd|ashakljas|ffasdfdsf|Fsfsdfsdf|dfsdfds|&^&*&^&*^07^%sdfsdfFSDF|SDFJKHFSDJ|wfhywufihwf|wfohfsd|fsduofhsdf|^&$^%#$#!!&@@
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Flat File Validation

Hi Team there is a requirement to do validate on Flat files using Shell Scripting. Suppose we have two flat files ( F1, F2). Validation 1. Row count between F1 and F2 a) Write it in a file with results of count of each file and differences if any 2. Apply checksum or any... (1 Reply)
Discussion started by: shlearner
1 Replies

2. Shell Programming and Scripting

reading a csv file and creating a flat file

hi i have written a script for reading a csv file and creating a flat file, suggest if this script can be optimized #---------------- FILENAME="$1" SCRIPT=$(basename $0) #-----------------------------------------// function usage { echo "\nUSAGE: $THIS_SCRIPT file_to_process\n"... (3 Replies)
Discussion started by: mprakasheee
3 Replies

3. Shell Programming and Scripting

Searching for Log / Bad file and Reading and writing to a flat file

Need to develop a unix shell script for the below requirement and I need your assistance: 1) search for file.log and file.bad file in a directory and read them 2) pull out "Load_Start_Time", "Data_File_Name", "Error_Type" from log file 4) concatinate each row from bad file as... (3 Replies)
Discussion started by: mlpathir
3 Replies

4. Shell Programming and Scripting

format a flat file

i have a flat file with around 700 columns . i want to break it into a flat file with say five columns like col1,col2,col3,col4 col5,col6,col7,col8 col9...... ,col700 how can i do this (5 Replies)
Discussion started by: r_t_1601
5 Replies

5. Programming

compare XML/flat file with UNIX file system structure

Before i start doing something, I wanted to know whether the approach to compare XML file with UNIX file system structure. I have a pre-configured file(contains a list of paths to executables) and i need to check against the UNIX directory structure. what are the various approches should i use ? I... (6 Replies)
Discussion started by: shafi2all
6 Replies

6. Shell Programming and Scripting

Awk to convert a flat file to CSV file

Hi , I have a file with contents as below: Contract Cancellation Report UARCNCL LOS CODE DATE REAS TYPE AMOUNT AMOUNT LETTER BY ========= ======= ==== ==== ==== ========= ==== ==== 8174739 7641509 1S NONE CRCD 30-JUN-2008 NPAR N .00 .00 CCAN 8678696 8091709 1S NONE DDEB 30-JUN-2008... (14 Replies)
Discussion started by: rkumudha
14 Replies

7. Shell Programming and Scripting

Flat File in HP-UX

Hi I have a file like below . IF the record starts with ASD then change the 20th offset to "K" follwed by that 20th offset value & if the record starts with ASDR then change the 38th offset to "K" followed by 38th offset value . But here the condition is the next value ... (0 Replies)
Discussion started by: Krishnaramjis
0 Replies

8. Shell Programming and Scripting

Flat file

Hi I have a flat file looking this QWER 2:35 7044 00001 ROUT 1188 EA SS ASD 2:36 7044 00010 ROUT BSD 2:37 7044 00011 ROUT END QWER 3:35 7044 00011 ROUT 1088 EA SS ASD 3:36 7044 00010 ROUT BSD 3:37 7044 00011 ROUT... (9 Replies)
Discussion started by: Krishnaramjis
9 Replies

9. Shell Programming and Scripting

Help with a flat file!!!

Hi All, I get a flat file everyday with some records being invalid.Some records come with less number '|'.I just want to add the missing '|'s ,So the it doesnt give an error insufficient fields while loading...There area total of 32 pipes in each record. Input: ASADASD |Y|B|SDFGDSDFD| ... (3 Replies)
Discussion started by: kumarsaravana_s
3 Replies

10. Shell Programming and Scripting

Look up column in a flat file

Here is on more go ! Need a shortcut for my problem ! problem is i have a look_update with fixed sequence of column that is : MANDT:SERAIL:SERSCHA:SEREX:EQTYP:BSTVP I will be getting data in a flat file having same number of column but the sequence could be different in each... (5 Replies)
Discussion started by: jambesh
5 Replies
Login or Register to Ask a Question