Sponsored Content
Top Forums UNIX for Dummies Questions & Answers Remove untagged and junk data from an XML Post 302913373 by Corona688 on Friday 15th of August 2014 05:10:10 PM
Old 08-15-2014
A modification of the smallest XML parser I have:

Code:
BEGIN {
        FS=">"
        OFS=">"
        RS="<"
}

NR==1 { next } # The first "line" is blank when RS=<

/^[!?]/ { printf("%s", RS $0 ); next    }   # print XML specification junk

# Handle open-tags
match($0, /^[^\/ \r\n\t>]+/) {
        TAG=substr(toupper($0), RSTART, RLENGTH);
        TAGS=TAG "%" TAGS;
}

# Handle close-tags
/^[\/]/ {
        sub(/^\//, "", $1);
        sub("^.*" toupper($1) "%", "", TAGS);
        $1="/"$1
        if(length(TAGS) == 0) # Strip out out-of-xml cdata
        {
                NF=2
                $2="\n"
        }
}

{       printf("<%s", $0);      } # Print everything

Code:
$ awk -f xmlclean.awk data

<?xml version="1.0" encoding="IBM037"?><spd><timestamp>07-04-2014 00:15:04</timestamp></spd>
<?xml version="1.0" encoding="IBM037"?><spd><timestamp>07-04-2014 00:15:04</timestamp></spd>
<?xml version="1.0" encoding="IBM037"?><spd><timestamp>07-04-2014 00:15:04</timestamp></spd>

$

This User Gave Thanks to Corona688 For This Post:
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

How to remove junk (^Ò) character while FTPing

Hi All, I have been trying to FTP some data files from Windows directory to a UNIX server. The txt file in the windows contails the following data: "111~XYZ~1~Contact person’s phone number~COMMENTS~~~~" but the same line is appearing as "111~XYZ~1~Contact person^Òs phone number~COMMENTS~~~~"... (8 Replies)
Discussion started by: vkumbhakarna
8 Replies

2. Shell Programming and Scripting

is is possible remove junk chars from the strings?

Hello friendz, dfl;g435hkd.fg ..this is what I am getting. I want to print strings without junk chars. I want to exactly like this... dflg435hkd.fg ...need some specific operators also. for example, dot or comma should allow. plz help me out ya! ~Balan:confused: (1 Reply)
Discussion started by: balan_mca
1 Replies

3. Shell Programming and Scripting

Remove all JUNK character from file.

Hi Team, I have a file having size greater than 1 GB. What i want to do is to check if it contains any JUNK character (ie any special charater thats not on the key board stroke). This file has 532 column & seperated with ^~^. I have found some solution from the file, but it is for a... (4 Replies)
Discussion started by: Amit.Sagpariya
4 Replies

4. Shell Programming and Scripting

Remove junk characters using Perl

Guys, can you help me in removing the junk character "^S" from the below line using perl Reference Data Not Recognised ^S Where a value is provided by the consuming system, which is not reco Thanks, M.Mohan (1 Reply)
Discussion started by: mohan_xunil
1 Replies

5. UNIX for Dummies Questions & Answers

XML file shows Junk Characters in UNIX

Hello sir, I have generated XML file from VS 2005. It works well in windows but it shows some junk characters in unix. Can any help me with this problem. Thank you in advance. Hema (6 Replies)
Discussion started by: hemavenkatesh
6 Replies

6. UNIX for Dummies Questions & Answers

How to remove JUNK characters (FROM�)

Hi I have to remove the junk characters from my file. Please help.. File content : CURITY_CODE_GSD) FROM� DL_CB_SOD_EOD_VALUATION WHERE� ASOF (1 Reply)
Discussion started by: arukuku
1 Replies

7. UNIX Desktop Questions & Answers

Help me to remove junk char

I wanted to remove junk char in my csv. :mad: Input file format: "17","9986782190","0","D","2" "17","9900918331","0","D","2" "13","9986782194","0","A","2" Output file format 9986782190 9900918331 9986782194 And one more thing all the time "13"," this will be different Ex: . (2 Replies)
Discussion started by: Siddartha
2 Replies

8. Shell Programming and Scripting

Remove all junk characters from a text file

I am using flatfile, in that flat file we are getting the junk chars 1)I21001f<82>^Me<85>!h49 Service Charge 2) I21001f‚ e...!h49 Service Charge please tell me how to remove all junk chars in unix scripts. (1 Reply)
Discussion started by: Talari
1 Replies

9. UNIX for Dummies Questions & Answers

How to replace and remove few junk characters from a specific field?

I would like to remove all characters starting with "%" and ending with ")" in the 4th field - please help!! 1412007819.864 /device/services/heartbeatxx 204 0.547%!i(int=0) 0.434 0.112 1412007819.866 /device/services/heartbeatxx 204 0.547%!i(int=1) 0.423 0.123... (10 Replies)
Discussion started by: snemuk14
10 Replies

10. UNIX for Beginners Questions & Answers

Need to remove Junk characters

Hi All, I have a issue that we are getting Junk characters from source and i am not able to load that records to Database. Line breakers Junk Characters (Â and different every time) Japanese Characters Every time I am using grep command and awk -F "\007" to find them and delete that... (1 Reply)
Discussion started by: spradeep86
1 Replies
XML::UM(3)						User Contributed Perl Documentation						XML::UM(3)

NAME
XML::UM - Convert UTF-8 strings to any encoding supported by XML::Encoding SYNOPSIS
use XML::UM; # Set directory with .xml files that comes with XML::Encoding distribution # Always include the trailing slash! $XML::UM::ENCDIR = '/home1/enno/perlModules/XML-Encoding-1.01/maps/'; # Create the encoding routine my $encode = XML::UM::get_encode ( Encoding => 'ISO-8859-2', EncodeUnmapped => &XML::UM::encode_unmapped_dec); # Convert a string from UTF-8 to the specified Encoding my $encoded_str = $encode->($utf8_str); # Remove circular references for garbage collection XML::UM::dispose_encoding ('ISO-8859-2'); DESCRIPTION
This module provides methods to convert UTF-8 strings to any XML encoding that XML::Encoding supports. It creates mapping routines from the .xml files that can be found in the maps/ directory in the XML::Encoding distribution. Note that the XML::Encoding distribution does install the .enc files in your perl directory, but not the.xml files they were created from. That's why you have to specify $ENCDIR as in the SYNOPSIS. This implementation uses the XML::Encoding class to parse the .xml file and creates a hash that maps UTF-8 characters (each consisting of up to 4 bytes) to their equivalent byte sequence in the specified encoding. Note that large mappings may consume a lot of memory! Future implementations may parse the .enc files directly, or do the conversions entirely in XS (i.e. C code.) get_encode (Encoding => STRING, EncodeUnmapped => SUB) The central entry point to this module is the XML::UM::get_encode() method. It forwards the call to the global $XML::UM::FACTORY, which is defined as an instance of XML::UM::SlowMapperFactory by default. Override this variable to plug in your own mapper factory. The XML::UM::SlowMapperFactory creates an instance of XML::UM::SlowMapper (and caches it for subsequent use) that reads in the .xml encod- ing file and creates a hash that maps UTF-8 characters to encoded characters. The get_encode() method of XML::UM::SlowMapper is called, finally, which generates an anonimous subroutine that uses the hash to convert multi-character UTF-8 blocks to the proper encoding. dispose_encoding ($encoding_name) Call this to free the memory used by the SlowMapper for a specific encoding. Note that in order to free the big conversion hash, the user should no longer have references to the subroutines generated by get_encode(). The parameters to the get_encode() method (defined as name/value pairs) are: o Encoding The name of the desired encoding, e.g. 'ISO-8859-2' o EncodeUnmapped (Default: &XML::UM::encode_unmapped_dec) Defines how Unicode characters not found in the mapping file (of the specified encoding) are printed. By default, they are converted to decimal entity references, like '&#123;' Use &XML::UM::encode_unmapped_hex for hexadecimal constants, like '&#xAB;' CAVEATS
I'm not exactly sure about which Unicode characters in the range (0 .. 127) should be mapped to themselves. See comments in XML/UM.pm near %DEFAULT_ASCII_MAPPINGS. The encodings that expat supports by default are currently not supported, (e.g. UTF-16, ISO-8859-1), because there are no .enc files avail- able for these encodings. This module needs some more work. If you have the time, please help! AUTHOR
Send bug reports, hints, tips, suggestions to Enno Derksen at <enno@att.com>. perl v5.8.0 2000-02-17 XML::UM(3)
All times are GMT -4. The time now is 02:06 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy