Login or Register to Ask a Question and Join Our Community


UNIX for Dummies Questions & Answers

Remove untagged and junk data from an XML

 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Remove untagged and junk data from an XML
# 1  
Old 08-15-2014
Remove untagged and junk data from an XML
Hi All , I have seen a lot of code samples which suggest how to remove the junk data from and XML , I need a code in unix which removes the junk characters as well as the valid characters those are not in XML tags , for example my XML is as follows :

Code:
<?xml version="1.0" encoding="IBM037"?><spd><timestamp>07-04-2014 00:15:04</timestamp></spd>
abcd
<?xml version="1.0" encoding="IBM037"?><spd><timestamp>07-04-2014 00:15:04</timestamp></spd>
pqrs $$ faf»
»
<?xml version="1.0" encoding="IBM037"?><spd><timestamp>07-04-2014 00:15:04</timestamp></spd>

I want to remove "abcd" and "pqrs $$ faf»" type of records which are not in tags , from the xml in Unix .
Is that possible ?
# 2  
Old 08-15-2014
For the sample data you've shown us, try:
Code:
grep -F '<' file

seems to do what you want. It will not, however, remove all of the occurrences of 07-04-2014 00:15:04 which is also data that is not in any tag.
# 3  
Old 08-15-2014
thanks Don...
Code:
07-04-2014 00:15:04

is still a valid data as it is between the valid tags of time stamp which start before it and ends after it.

But the data like "abcd" and "pqrs $$ faf»" is not enclosed in tags , which doesnt allow it to form a valid xml.
# 4  
Old 08-15-2014
Quote:
Originally Posted by IshuGupta
thanks Don...
Code:
07-04-2014 00:15:04

is still a valid data as it is between the valid tags of time stamp which start before it and ends after it.

But the data like "abcd" and "pqrs $$ faf»" is not enclosed in tags , which doesnt allow it to form a valid xml.
You know that because you know what tags are valid and what data is valid between certain tags. You can build that knowledge into a script, but, for the generic script, there is a lot of meta-data that needs to be provided to that script defining the valid tags, the valid nesting of tags, the formats allowed for data between certain tags, ... . But, you aren't going to get a parser like that from a forum like this.

If you can state some clear requirements to simplify the general problem to a more specific issue, we might be able to help.

Do you just want to throw away stuff (other than a single <newline> character after a </spd> flag) that does not start with a <?xml version="1.0" encoding="IBM037"?> tag and end with the next </spd> tag?

Most XML code I've seen would have a </xml> tag for each of the <?xml...> tags. Why aren't there any in your XML code?
# 5  
Old 08-15-2014
Your "XML" looks like a bunch of individual, separate XML files slapped together into one. Does the <?xml nonsense really need to be preserved?

Quote:
Originally Posted by IshuGupta
thanks Don...
Have you tried his suggestion? Does it work?
This User Gave Thanks to Corona688 For This Post:
IshuGupta
# 6  
Old 08-15-2014
A modification of the smallest XML parser I have:

Code:
BEGIN {
        FS=">"
        OFS=">"
        RS="<"
}

NR==1 { next } # The first "line" is blank when RS=<

/^[!?]/ { printf("%s", RS $0 ); next    }   # print XML specification junk

# Handle open-tags
match($0, /^[^\/ \r\n\t>]+/) {
        TAG=substr(toupper($0), RSTART, RLENGTH);
        TAGS=TAG "%" TAGS;
}

# Handle close-tags
/^[\/]/ {
        sub(/^\//, "", $1);
        sub("^.*" toupper($1) "%", "", TAGS);
        $1="/"$1
        if(length(TAGS) == 0) # Strip out out-of-xml cdata
        {
                NF=2
                $2="\n"
        }
}

{       printf("<%s", $0);      } # Print everything

Code:
$ awk -f xmlclean.awk data

<?xml version="1.0" encoding="IBM037"?><spd><timestamp>07-04-2014 00:15:04</timestamp></spd>
<?xml version="1.0" encoding="IBM037"?><spd><timestamp>07-04-2014 00:15:04</timestamp></spd>
<?xml version="1.0" encoding="IBM037"?><spd><timestamp>07-04-2014 00:15:04</timestamp></spd>

$

This User Gave Thanks to Corona688 For This Post:
IshuGupta
# 7  
Old 08-18-2014
Thanks Don and Corona, The code works now , based on your script and suggestion.
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Need to remove Junk characters

Hi All, I have a issue that we are getting Junk characters from source and i am not able to load that records to Database. Line breakers Junk Characters (Â and different every time) Japanese Characters Every time I am using grep command and awk -F "\007" to find them and delete that... (1 Reply)
Discussion started by: spradeep86
1 Replies

2. UNIX for Dummies Questions & Answers

How to replace and remove few junk characters from a specific field?

I would like to remove all characters starting with "%" and ending with ")" in the 4th field - please help!! 1412007819.864 /device/services/heartbeatxx 204 0.547%!i(int=0) 0.434 0.112 1412007819.866 /device/services/heartbeatxx 204 0.547%!i(int=1) 0.423 0.123... (10 Replies)
Discussion started by: snemuk14
10 Replies

3. Shell Programming and Scripting

Remove all junk characters from a text file

I am using flatfile, in that flat file we are getting the junk chars 1)I21001f<82>^Me<85>!h49 Service Charge 2) I21001f‚ e...!h49 Service Charge please tell me how to remove all junk chars in unix scripts. (1 Reply)
Discussion started by: Talari
1 Replies

4. UNIX Desktop Questions & Answers

Help me to remove junk char

I wanted to remove junk char in my csv. :mad: Input file format: "17","9986782190","0","D","2" "17","9900918331","0","D","2" "13","9986782194","0","A","2" Output file format 9986782190 9900918331 9986782194 And one more thing all the time "13"," this will be different Ex: . (2 Replies)
Discussion started by: Siddartha
2 Replies

5. UNIX for Dummies Questions & Answers

How to remove JUNK characters (FROM�)

Hi I have to remove the junk characters from my file. Please help.. File content : CURITY_CODE_GSD) FROM� DL_CB_SOD_EOD_VALUATION WHERE� ASOF (1 Reply)
Discussion started by: arukuku
1 Replies

6. UNIX for Dummies Questions & Answers

XML file shows Junk Characters in UNIX

Hello sir, I have generated XML file from VS 2005. It works well in windows but it shows some junk characters in unix. Can any help me with this problem. Thank you in advance. Hema (6 Replies)
Discussion started by: hemavenkatesh
6 Replies

7. Shell Programming and Scripting

Remove junk characters using Perl

Guys, can you help me in removing the junk character "^S" from the below line using perl Reference Data Not Recognised ^S Where a value is provided by the consuming system, which is not reco Thanks, M.Mohan (1 Reply)
Discussion started by: mohan_xunil
1 Replies

8. Shell Programming and Scripting

Remove all JUNK character from file.

Hi Team, I have a file having size greater than 1 GB. What i want to do is to check if it contains any JUNK character (ie any special charater thats not on the key board stroke). This file has 532 column & seperated with ^~^. I have found some solution from the file, but it is for a... (4 Replies)
Discussion started by: Amit.Sagpariya
4 Replies

9. Shell Programming and Scripting

is is possible remove junk chars from the strings?

Hello friendz, dfl;g435hkd.fg ..this is what I am getting. I want to print strings without junk chars. I want to exactly like this... dflg435hkd.fg ...need some specific operators also. for example, dot or comma should allow. plz help me out ya! ~Balan:confused: (1 Reply)
Discussion started by: balan_mca
1 Replies

10. UNIX for Dummies Questions & Answers

How to remove junk (^Ò) character while FTPing

Hi All, I have been trying to FTP some data files from Windows directory to a UNIX server. The txt file in the windows contails the following data: "111~XYZ~1~Contact person’s phone number~COMMENTS~~~~" but the same line is appearing as "111~XYZ~1~Contact person^Òs phone number~COMMENTS~~~~"... (8 Replies)
Discussion started by: vkumbhakarna
8 Replies
Login or Register to Ask a Question