Sponsored Content
Top Forums Shell Programming and Scripting remove html tags,consecutive duplicate lines Post 302526974 by clicstic on Thursday 2nd of June 2011 03:52:19 AM
Old 06-02-2011
input
Code:
<html><head><title>CIS013: Operating System - Unix</title></head>
<body>
<h1>Week 1</h1>
<h2>Chapter 1</h2>
<h3>Getting Started With Unix</h3>
<p>Getting Started With Unix</p>
<h1>Week 2</h1>
<h2>Chapter 2</h2>
<h3>Using Directories and Files</h3>
<p>Using Directories and Files</p>
<h2>Chapter 3</h2>
<h3>Working with Your Shell</h3>
<p>Working with Your Shell</p>
<h1>Week 3</h1>
<h2>Chapter 4</h2>
<h3>Creating and Editing Files</h3>
<p>Creating and Editing Files</p>
<h2>Chapter 5</h2>
<h3>Controlling Ownership and Permissions</h3>
<p>Controlling Ownership and Permissions</p>
<h1>Week 4</h1>
<h2>Chapter 6</h2>
<h3>Manipulating Files</h3>
<p>Manipulating Files</p>
<h2>Chapter 7</h2>
<h3>Getting Information About the System</h3>
<p>Getting Information About the System</p>
<h1>Week 5</h1>
<h2>Chapter 8</h2>
<h3>Configuring Your Unix Environment</h3>
<p>Configuring Your Unix Environment</p>
<h2>Chapter 9</h2>
<h3>Running Scripts and Programs</h3>
<p>Running Scripts and Programs</p>
<h1>Week 6</h1>
<h2>Chapter 10</h2>
<h3>Writing Basic Scripts</h3>
<p>Writing Basic Scripts</p>
</body>
<html>
 Week 1

  Chapter 1

  Getting Started With Unix

  Getting Started With Unix
  Week 2

  Chapter 2

  Using Directories and Files

  Using Directories and Files
  Chapter 3

output

Week 1

Chapter 1

Getting Started With Unix



Week 2

Chapter 2

Using Directories and Files



Chapter 3

---------- Post updated at 03:52 AM ---------- Previous update was at 03:51 AM ----------

pipeline... I remember that now
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

how to remove duplicate lines

I have following file content (3 fields each line): 23 888 10.0.0.1 dfh 787 10.0.0.2 dssf dgfas 10.0.0.3 dsgas dg 10.0.0.4 df dasa 10.0.0.5 df dag 10.0.0.5 dfd dfdas 10.0.0.5 dfd dfd 10.0.0.6 daf nfd 10.0.0.6 ... as can be seen, that the third field is ip address and sorted. but... (3 Replies)
Discussion started by: fredao
3 Replies

2. Shell Programming and Scripting

Remove duplicate lines

Hi, I have a huge file which is about 50GB. There are many lines. The file format likes 21 rs885550 0 9887804 C C T C C C C C C C 21 rs210498 0 9928860 0 0 C C 0 0 0 0 0 0 21 rs303304 0 9941889 A A A A A A A A A A 22 rs303304 0 9941890 0 A A A A A A A A A The question is that there are a few... (4 Replies)
Discussion started by: zhshqzyc
4 Replies

3. Shell Programming and Scripting

remove consecutive duplicate rows

I have some data that looks like, 1 3300665.mol 3300665 5177008 102.093 2 3300665.mol 3300665 5177008 102.093 3 3294015.mol 3294015 5131552 102.114 4 3294015.mol 3294015 5131552 102.114 5 3293734.mol 3293734 5129625 104.152 6 3293734.mol ... (13 Replies)
Discussion started by: LMHmedchem
13 Replies

4. Shell Programming and Scripting

Remove lines with duplicate first field

Trying to cut down the size of some log files. Now that I write this out it looks more dificult than i thought it would be. Need a bash script or command that goes sequentially through all lines of a file, and does this: if field1 (space separated) is the number 2012 print the entire line. Do... (7 Replies)
Discussion started by: ajp7701
7 Replies

5. Homework & Coursework Questions

Script: Removing HTML tags and duplicate lines

Use and complete the template provided. The entire template must be completed. If you don't, your post may be deleted! 1. The problem statement, all variables and given/known data: You will write a script that will remove all HTML tags from an HTML document and remove any consecutive... (3 Replies)
Discussion started by: tburns517
3 Replies

6. UNIX for Dummies Questions & Answers

Remove Duplicate Lines

Hi I need this output. Thanks. Input: TAZ YET FOO FOO VAK TAZ BAR Output: YET VAK BAR (10 Replies)
Discussion started by: tara123
10 Replies

7. Shell Programming and Scripting

Remove duplicate lines from a file

Hi, I have a csv file which contains some millions of lines in it. The first line(Header) repeats at every 50000th line. I want to remove all the duplicate headers from the second occurance(should not remove the first line). I don't want to use any pattern from the Header as I have some... (7 Replies)
Discussion started by: sudhakar T
7 Replies

8. Shell Programming and Scripting

Check/print missing number in a consecutive range and remove duplicate numbers

Hi, In an ideal scenario, I will have a listing of db transaction log that gets copied to a DR site and if I have them all, they will be numbered consecutively like below. 1_79811_01234567.arc 1_79812_01234567.arc 1_79813_01234567.arc 1_79814_01234567.arc 1_79815_01234567.arc... (3 Replies)
Discussion started by: newbie_01
3 Replies

9. Shell Programming and Scripting

How to remove duplicate lines?

Hi All, I am storing the result in the variable result_text using the below code. result_text=$(printf "$result_text\t\n$name") The result_text is having the below text. Which is having duplicate lines. file and time for the interval 03:30 - 03:45 file and time for the interval 03:30 - 03:45 ... (4 Replies)
Discussion started by: nalu
4 Replies

10. Shell Programming and Scripting

Remove duplicate consecutive lines with specific string

Hello, I'm trying to remove the duplicate consecutive lines with specific string "WARNING". File.txt abc; WARNING 2345 WARNING 2345 WARNING 2345 WARNING 2345 WARNING 2345 bcd; abc; 123 123 123 WARNING 1234 WARNING 2345 WARNING 2345 efgh; (6 Replies)
Discussion started by: Mannu2525
6 Replies
Strip(3pm)						User Contributed Perl Documentation						Strip(3pm)

NAME
HTML::Strip - Perl extension for stripping HTML markup from text. SYNOPSIS
use HTML::Strip; my $hs = HTML::Strip->new(); my $clean_text = $hs->parse( $raw_html ); $hs->eof; DESCRIPTION
This module simply strips HTML-like markup from text in a very quick and brutal manner. It could quite easily be used to strip XML or SGML from text as well; but removing HTML markup is a much more common problem, hence this module lives in the HTML:: namespace. It is written in XS, and thus about five times quicker than using regular expressions for the same task. It does not do any syntax checking (if you want that, use HTML::Parser), instead it merely applies the following rules: 1. Anything that looks like a tag, or group of tags will be replaced with a single space character. Tags are considered to be anything that starts with a "<" and ends with a ">"; with the caveat that a ">" character may appear in either of the following without ending the tag: Quote Quotes are considered to start with either a "'" or a """ character, and end with a matching character not preceded by an even number or escaping slashes (i.e. """ does not end the quote but "\\"" does). Comment If the tag starts with an exclamation mark, it is assumed to be a declaration or a comment. Within such tags, ">" characters do not end the tag if they appear within pairs of double dashes (e.g. "<!-- <a href="old.htm">old page</a> -->" would be stripped completely). 2. Anything the appears within so-called strip tags is stripped as well. By default, these tags are "title", "script", "style" and "applet". HTML::Strip maintains state between calls, so you can parse a document in chunks should you wish. If one chunk ends half-way through a tag, quote, comment, or whatever; it will remember this, and expect the next call to parse to start with the remains of said tag. If this is not going to be the case, be sure to call $hs->eof() between calls to $hs->parse(). METHODS new() Constructor. Can optionally take a hash of settings (with keys corresponsing to the "set_" methods below). For example, the following is a valid constructor: my $hs = HTML::Strip->new( striptags => [ 'script', 'iframe' ], emit_spaces => 0 ); parse() Takes a string as an argument, returns it stripped of HTML. eof() Resets the current state information, ready to parse a new block of HTML. clear_striptags() Clears the current set of strip tags. add_striptag() Adds the string passed as an argument to the current set of strip tags. set_striptags() Takes a reference to an array of strings, which replace the current set of strip tags. set_emit_spaces() Takes a boolean value. If set to false, HTML::Strip will not attempt any conversion of tags into spaces. Set to true by default. set_decode_entities() Takes a boolean value. If set to false, HTML::Strip will decode HTML entities. Set to true by default. LIMITATIONS Whitespace Despite only outputting one space character per group of tags, and avoiding doing so when tags are bordered by spaces or the start or end of strings, HTML::Strip can often output more than desired; such as with the following HTML: <h1> HTML::Strip </h1> <p> <em> <strong> fast, and brutal </strong> </em> </p> Which gives the following output: " HTML::Strip fast, and brutal " Thus, you may want to post-filter the output of HTML::Strip to remove excess whitespace (for example, using "tr/ / /s;"). (This has been improved since previous releases, but is still an issue) HTML Entities HTML::Strip will only attempt decoding of HTML entities if HTML::Entities is installed. EXPORT None by default. AUTHOR
Alex Bowley <kilinrax@cpan.org> SEE ALSO
perl, HTML::Parser, HTML::Entities perl v5.14.2 2011-11-15 Strip(3pm)
All times are GMT -4. The time now is 10:44 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy