Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

I need help to removing repetitive lines

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting I need help to removing repetitive lines
# 1  
Old 03-17-2016
I need help to removing repetitive lines
Hello SuperUsers! First i wanna say my english sucks.. Don't hate me for that. Smilie

I need to make a Bash for removing smilar lines from an output file.
My output file always same.

Line 1 & 2 Stays. And others similar to this lines needs to be delete.
HTML Code:
</UsageData><?xml version="1.0" encoding="UTF-8"?>
<UsageData broadcastday="2016-03-16">
(only date changes)

And Last line Stays. And others similar to this line needs to delete.
HTML Code:
</UsageData>
It's easy but i'm newbie and i couldn't figure how to do that. Smilie


There is my sample XML.

PHP Code:
<?xml version="1.0" encoding="UTF-8"?>
<UsageData broadcastday="2016-03-16">
    <Hh hhID="48800301">
        <Inst instID="000002B9"/>
        <Live>
            <Station>516</Station>
            <From>Wed Mar 16 2016 09:52:47 GMT+0000 (UTC)</From>
            <DurSec>58077</DurSec>
            <Viewer>
                <HhMem>569de65c9c3ab0cf7bfa2df2</HhMem>
            </Viewer>
        </Live>
    </Hh>
    <Hh hhID="46920403">
        <Inst instID="000002A8"/>
        <Live>
            <Station>5000</Station>
            <From>Wed Mar 16 2016 12:42:17 GMT+0000 (UTC)</From>
            <DurSec>47908</DurSec>
            <Viewer>
                <HhMem>56caee95f915e09335fd976f</HhMem>
            </Viewer>
        </Live>
    </Hh>
</UsageData><?xml version="1.0" encoding="UTF-8"?>
<UsageData broadcastday="2016-03-16">
    <Hh hhID="15260304">
        <Inst instID="000000A5"/>
        <Live>
            <Station>5000</Station>
            <From>Wed Mar 16 2016 12:57:48 GMT+0000 (UTC)</From>
            <DurSec>28814</DurSec>
            <Viewer>
                <HhMem>565f181dd830d3cc7057c0b9</HhMem>
            </Viewer>
        </Live>
    </Hh>
</UsageData><?xml version="1.0" encoding="UTF-8"?>
<UsageData broadcastday="2016-03-16">
    <Hh hhID="50100501">
        <Inst instID="0000022D"/>
        <Live>
            <Station>560</Station>
            <From>Wed Mar 16 2016 14:21:19 GMT+0000 (UTC)</From>
            <DurSec>41967</DurSec>
            <Viewer>
                <HhMem>56c4412de6a8ff4da18fd4ae</HhMem>
                <HhMem>56c4412de6a8ff4da18fd4cb</HhMem>
            </Viewer>
        </Live>
    </Hh>
</UsageData><?xml version="1.0" encoding="UTF-8"?>
<UsageData broadcastday="2016-03-16">
    <Hh hhID="36110404">
        <Inst instID="00000104"/>
        <Live>
            <Station>545</Station>
            <From>Wed Mar 16 2016 15:01:04 GMT+0000 (UTC)</From>
            <DurSec>671</DurSec>
            <Viewer>
                <HhMem>568ce8acbd0e486a951d41ce</HhMem>
                <HhMem>568ce8acbd0e486a951d41dc</HhMem>
                <HhMem>568ce8acbd0e486a951d41c5</HhMem>
            </Viewer>
        </Live>
    </Hh>
</UsageData>
Thank you so much for your answers. Smilie
# 2  
Old 03-17-2016
Would this help?
Code:
awk  '/<\/UsageData><\?xml version="1.0" encoding="UTF-8"\?>/ {getline; next}1' file

Last edited by RudiC; 03-17-2016 at 06:43 PM.. Reason: typo
This User Gave Thanks to RudiC For This Post:
morphin
# 3  
Old 03-17-2016
Code:
l=$(wc -l file | cut -d' ' -f1); sed "3,$(( l-1 ))s/.*UsageData.*//" file

This User Gave Thanks to balajesuri For This Post:
morphin
# 4  
Old 03-17-2016
Quote:
Originally Posted by RudiC
Woult this help?
Code:
awk  '/<\/UsageData><\?xml version="1.0" encoding="UTF-8"\?>/ {getline; next}1' file

Thank youu very much!
It's working like magic Smilie

But if you have a little time, can you explain me your commands, how you do that?
# 5  
Old 03-17-2016
Less magic, more logic.

Above checks for your pattern; if found, it gets the following line, and does nothing with either of them. The next advances to the next line, which then is printed (like all the others) as the default action for the 1 (= TRUE) pattern.

One caveat: your second pattern is not checked for, so any line pair fitting the first will be removed!
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to remove repetitive lines in a file with sed?

Hello, My goal is the make all x times repeated lines into a single line. I need to attain the expected output with sed -i , I need to overwrite the MyFile MyFile: Hello World Welcome Hello World Welcome Back This is my test Expected output: Hello World Welcome Welcome Back This is... (6 Replies)
Discussion started by: baris35
6 Replies

2. Shell Programming and Scripting

Removing multiple lines from input file, if multiple lines match a pattern.

GM, I have an issue at work, which requires a simple solution. But, after multiple attempts, I have not been able to hit on the code needed. I am assuming that sed, awk or even perl could do what I need. I have an application that adds extra blank page feeds, for multiple reports, when... (7 Replies)
Discussion started by: jxfish2
7 Replies

3. Shell Programming and Scripting

Joining broken lines and removing empty lines

Hi - I have req to join broken lines and remove empty lines but should NOT be in one line. It has to be as is line by line. The challenge here is there is no end of line/start of line char. thanks in advance Source:- 2003-04-34024|04-10-2003|Claims|Claim|01-13-2003|Air Bag:Driver;... (7 Replies)
Discussion started by: Jackceasar123
7 Replies

4. UNIX for Dummies Questions & Answers

Removing PATTERN from txt without removing lines and general text formatting

Hi Everybody! First post! Totally noobie. I'm using the terminal to read a poorly formatted book. The text file contains, in the middle of paragraphs, hyphenation to split words that are supposed to be on multiple pages. It looks ve -- ry much like this. I was hoping to use grep -v " -- "... (5 Replies)
Discussion started by: AxeHandle
5 Replies

5. Shell Programming and Scripting

Removing Lines that Contain a Certain String

I'm writing a UNIX script to interpret text files and search for certain strings to output and need to delete certain lines that contain characters that I would like removed from the file. The rest of the script is already completed and I have it output how I would like, but I cannot seem to get... (2 Replies)
Discussion started by: DetroitLolcat
2 Replies

6. Shell Programming and Scripting

Print lines between two repetitive patterns

Hi users I have one file which has number of occurrence of one pattern examples Adjustmenttype,11 xyz 10 dwe 9 abd 13 def 14 Adjustmenttype,11 xyz 24 dwe 34 abd 35 def 11 nmb 12 Adjustmenttype, not eleven .... ... ... (2 Replies)
Discussion started by: eranmoh
2 Replies

7. Shell Programming and Scripting

Removing lines with condition

Hello guys, I need help with a script for removing lines that does not satisfy a condition. For example if a file has these lines: aaaa bbbb cccc aaaa bbbb cccc dddd eeee ffff gggg hhhh iiii jjjj kkkk llll aaaa bbbb cccc jjjj kkkk lllll dddd eeee ffff dddd eeee ffff Then I want... (4 Replies)
Discussion started by: jaysean
4 Replies

8. Shell Programming and Scripting

Removing last two lines

I have an awk script that replaces ">" with % %> %< SOURCE", ++i %( PHASE 1", i I use the following script />/ { if ( FNR > 1 ) { print "%)" print "%>" } print "" print "%< SOURCE", ++i (11 Replies)
Discussion started by: kristinu
11 Replies

9. Shell Programming and Scripting

Removing empty lines(space) between two lines containing strings

Hi, Please provide shell script to Remove empty lines(space) between two lines containing strings in a file. Input File : A1/EXT "BAP_BSC6/07B/00" 844 090602 1605 RXOCF-465 PDTR11 1 SITE ON BATTERY A2/EXT... (3 Replies)
Discussion started by: sudhakaryadav
3 Replies

10. Shell Programming and Scripting

Removing lines having #

I have a script which goes to different directories and gives the values of all the input parameters, Something as follows cd /opt grep script-filter = yes *.conf grep user-and-group-in-same-suffix = yes *.conf grep worker-threads = 300 *.conf grep failover-auth = *.conf grep... (9 Replies)
Discussion started by: openspark
9 Replies
Login or Register to Ask a Question