Sponsored Content
Full Discussion: help with data formatting
Top Forums Shell Programming and Scripting help with data formatting Post 302587718 by varman on Thursday 5th of January 2012 05:48:04 PM
Old 01-05-2012
help with data formatting

Hi,

I have data coming in like below. Not all data is like that, these are the problem records that is causing the ETL load to fail. Can you pls help me with combining theese broken records!

Code:
001800018000000guyMMAAY~acct name~acct type~~"address part 1
address part2"~city~STATE~ZIP~COUNTRY~(123) 123-1234~~~~~~~~~~~~~~~~~^M
0018000000gwQ63AAE~acct name~acct type~~"address part 1
address part2"~city~state~zip~country~(123) 123-1234~(123) 123-1234~~~~~~~~~~~~~~~~^M

Appreciate your time and effort!

Thanks,

Last edited by radoulov; 01-05-2012 at 07:00 PM.. Reason: Code tags!
 

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Formatting Data

i am writing a script that reads in a file and i just want it to print each element on a new line here is my code and the data that i want to read in #!/usr/bin/perl use strict; use CGI qw(:standard); use CGI qw(:cgi); my $data_file = "/tmp/results.txt"; my $configuration; my... (3 Replies)
Discussion started by: nmeliasp
3 Replies

2. Shell Programming and Scripting

Re-formatting of data display

Hi All, I have been trying to re-arrange the below data using AWK or Perl. Can anybody help me ? Thanks in advance. Input: 111 222 333 444 AAA BBB CCC DDD 555 666 777 888 EEE FFF GGG HHH Output: (6 Replies)
Discussion started by: Raynon
6 Replies

3. Shell Programming and Scripting

Script for data formatting

Hi I have to convert the data in a file ******* 01-20-09 11:14AM 60928 ABC Valuation-2009.xls 01-20-09 11:16AM 55808 DEF GHI Equation-2009.xls 01-20-09 11:02AM 52736 ABC DF Valuation-2009.xls 01-20-09 11:06AM 89600 THE... (6 Replies)
Discussion started by: shekhar_v4
6 Replies

4. UNIX for Dummies Questions & Answers

Data manipulation/ formatting question

How would I get this output to look $ cat newfile 13114 84652 84148 LIKE THIS?: 13114,84652,84148 sed,cut awk? syntax? (2 Replies)
Discussion started by: ddurden7
2 Replies

5. Shell Programming and Scripting

Formatting Data - CSV

I want to check whether if any column data has any + , - , = prefixed to it then convert it in such a form that in excel its not read as formula. echo "$DATA" | awk 'BEGIN { OFS="," } -F" " {print $1,$2,$3,$4,$5,$6,$7,$8.$9,$10,$11,$12}' (4 Replies)
Discussion started by: dinjo_jo
4 Replies

6. Shell Programming and Scripting

Formatting input data

Hello everybody, I have a file containing some statistics regarding CPU usage. The file has this syntax : Fri Jul 16 14:27:16 EEST 2010 Cpu(s): 15.2%us, 1.4%sy, 0.0%ni, 82.3%id, 0.1%wa, 0.0%hi, 0.9%si, 0.0%st Fri Jul 16 15:02:17 EEST 2010 Cpu(s): 15.3%us, 1.4%sy, 0.0%ni, 82.3%id, ... (9 Replies)
Discussion started by: spiriad
9 Replies

7. UNIX for Advanced & Expert Users

formatting the data

HI I want to make it single row if start with braces i.e. { .Any idea {1:XXX2460275191}{2:SEC00687921131112201641N}{3:{58910}}{4: :R:GENL :C::xx//xx1 :20C::yy//yy1 :2S:xxT} {1:XXX2460275190}{2:SEC00687921131112201641y}{3:{58911}}{4: :z:GENL :v::xx//xx1 :10C::yy//yy1 :4S:xxT ... (2 Replies)
Discussion started by: mohan705
2 Replies

8. Shell Programming and Scripting

Help with data formatting

Hi, I am generating the following output from my script. Country,A,B,C,D,E,F INDIA ,3755019,774604,484749,329838,7333612,442031 CHINA ,3716520,889197,530899,379754,6198475,355768 JAPAN ,52038,30462,231224,147275,1272,498 USA,9494,1130,0,0,15303,451... (5 Replies)
Discussion started by: karumudi7
5 Replies

9. Shell Programming and Scripting

Data formatting using awk

Need assistance on the data extraction using awk Below is the format and would like to extract the data in another format ------------------------------------------------------------------------------------------------- Minimum Temperature (deg F ) DAY 1 2 3 4 5 6 7 8 9 10 11... (4 Replies)
Discussion started by: ajayram_arya
4 Replies
Fields(3pm)						User Contributed Perl Documentation					       Fields(3pm)

NAME
Sort::Fields - Sort lines containing delimited fields SYNOPSIS
use Sort::Fields; @sorted = fieldsort [3, '2n'], @lines; @sorted = fieldsort '+', [-1, -3, 0], @lines; $sort_3_2n = make_fieldsort [3, '2n'], @lines; @sorted = $sort_3_2n->(@lines); DESCRIPTION
Sort::Fields provides a general purpose technique for efficiently sorting lists of lines that contain data separated into fields. Sort::Fields automatically imports two subroutines, "fieldsort" and "make_fieldsort", and two variants, "stable_fieldsort" and "make_sta- ble_fieldsort". "make_fieldsort" generates a sorting subroutine and returns a reference to it. "fieldsort" is a wrapper for the "make_fieldsort" subroutine. The first argument to make_fieldsort is a delimiter string, which is used as a regular expression argument for a "split" operator. The delimiter string is optional. If it is not supplied, make_fieldsort splits each line using "/s+/". The second argument is an array reference containing one or more field specifiers. The specifiers indicate what fields in the strings will be used to sort the data. The specifier "1" indicates the first field, "2" indicates the second, and so on. A negative specifier like "-2" means to sort on the second field in reverse (descending) order. To indicate a numeric rather than alphabetic comparison, append "n" to the specifier. A specifier of "0" means the entire string ("-0" means the entire string, in reverse order). The order in which the specifiers appear is the order in which they will be used to sort the data. The primary key is first, the secondary key is second, and so on. "fieldsort [1, 2], @data" is roughly equivalent to "make_fieldsort([1, 2])->(@data)". Avoid calling fieldsort repeatedly with the same sort specifiers. If you need to use a particular sort more than once, it is more efficient to call "make_fieldsort" once and reuse the subroutine it returns. "stable_fieldsort" and "make_stable_fieldsort" are like their "unstable" counterparts, except that the items that compare the same are maintained in their original order. EXAMPLES
Some sample data (in array @data): 123 asd 1.22 asdd 32 ewq 2.32 asdd 43 rewq 2.12 ewet 51 erwt 34.2 ewet 23 erww 4.21 ewet 91 fdgs 3.43 ewet 123 refs 3.22 asdd 123 refs 4.32 asdd # alpha sort on column 1 print fieldsort [1], @data; 123 asd 1.22 asdd 123 refs 3.22 asdd 123 refs 4.32 asdd 23 erww 4.21 ewet 32 ewq 2.32 asdd 43 rewq 2.12 ewet 51 erwt 34.2 ewet 91 fdgs 3.43 ewet # numeric sort on column 1 print fieldsort ['1n'], @data; 23 erww 4.21 ewet 32 ewq 2.32 asdd 43 rewq 2.12 ewet 51 erwt 34.2 ewet 91 fdgs 3.43 ewet 123 asd 1.22 asdd 123 refs 3.22 asdd 123 refs 4.32 asdd # reverse numeric sort on column 1 print fieldsort ['-1n'], @data; 123 asd 1.22 asdd 123 refs 3.22 asdd 123 refs 4.32 asdd 91 fdgs 3.43 ewet 51 erwt 34.2 ewet 43 rewq 2.12 ewet 32 ewq 2.32 asdd 23 erww 4.21 ewet # alpha sort on column 2, then alpha on entire line print fieldsort [2, 0], @data; 123 asd 1.22 asdd 51 erwt 34.2 ewet 23 erww 4.21 ewet 32 ewq 2.32 asdd 91 fdgs 3.43 ewet 123 refs 3.22 asdd 123 refs 4.32 asdd 43 rewq 2.12 ewet # alpha sort on column 4, then numeric on column 1, then reverse # numeric on column 3 print fieldsort [4, '1n', '-3n'], @data; 32 ewq 2.32 asdd 123 refs 4.32 asdd 123 refs 3.22 asdd 123 asd 1.22 asdd 23 erww 4.21 ewet 43 rewq 2.12 ewet 51 erwt 34.2 ewet 91 fdgs 3.43 ewet # now, splitting on either literal period or whitespace # sort numeric on column 4 (fractional part of decimals) then # numeric on column 3 (whole part of decimals) print fieldsort '(?:.|s+)', ['4n', '3n'], @data; 51 erwt 34.2 ewet 43 rewq 2.12 ewet 23 erww 4.21 ewet 123 asd 1.22 asdd 123 refs 3.22 asdd 32 ewq 2.32 asdd 123 refs 4.32 asdd 91 fdgs 3.43 ewet # alpha sort on column 4, then numeric on the entire line # NOTE: produces warnings under -w print fieldsort [4, '0n'], @data; 32 ewq 2.32 asdd 123 asd 1.22 asdd 123 refs 3.22 asdd 123 refs 4.32 asdd 23 erww 4.21 ewet 43 rewq 2.12 ewet 51 erwt 34.2 ewet 91 fdgs 3.43 ewet # stable alpha sort on column 4 (maintains original relative order # among items that compare the same) print stable_fieldsort [4], @data; 123 asd 1.22 asdd 32 ewq 2.32 asdd 123 refs 3.22 asdd 123 refs 4.32 asdd 43 rewq 2.12 ewet 51 erwt 34.2 ewet 23 erww 4.21 ewet 91 fdgs 3.43 ewet BUGS
Some rudimentary tests now. Perhaps something should be done to catch things like: fieldsort '.', [1, 2], @lines; '.' translates to "split /./" -- probably not what you want. Passing blank lines and/or lines containing the wrong kind of data (alphas instead of numbers) can result in copious warning messages under "-w". If the regexp contains memory parentheses ("(...)" rather than "(?:...)"), split will function in "delimiter retention" mode, capturing the contents of the parentheses as well as the stuff between the delimiters. I could imagine how this could be useful, but on the other hand I could also imagine how it could be confusing if encountered unexpectedly. Caveat sortor. Not really a bug, but if you are planning to sort a large text file, consider using sort(1). Unless, of course, your operating system doesn't have sort(1). AUTHOR
Joseph N. Hall, joseph@5sigma.com SEE ALSO
perl(1). perl v5.8.8 2008-03-25 Fields(3pm)
All times are GMT -4. The time now is 03:06 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy