Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Deleting duplicated chunks in a file using awk/sed
Top Forums Shell Programming and Scripting Deleting duplicated chunks in a file using awk/sed Post 302974139 by Aia on Thursday 26th of May 2016 01:06:43 AM
Old 05-26-2016
Save as chunks.pl
Run as perl chunks.pl chunks.data

Code:
#!/usr/bin/perl
#
use strict;
use warnings;

my @chunks;
my $lines = 0;
while(<>){
    my @parts = split;
    push @{$chunks[$lines++]}, @parts;
    if ($lines == 4) {
        my %seen;
        my $count = 0;
        my @keep;
        for my $i (@{$chunks[2]}) {
            !$seen{$i}++ and push @keep, $count;
            ++$count;
        }

        for my $i (@chunks){
            my @returns;
            for my $j (@keep) {
                push @returns, @{$i}[$j];
            }
            print "@returns\n";
        }

        clean();
    }
}

sub clean {
    @chunks = ();
    $lines = 0;
}


Output:

Code:
path_sparc_ffu_dp_out_1885 path_sparc_ffu_dp_out_3115
R_1545/Q R_1545/Q
dp_ctl_synd_out_low[6] dp_ctl_synd_out_low[2]
0.926208 0.905082
path_sparc_ffu_dp_out_699 path_sparc_ffu_dp_out_712
R_1053/Q R_1053/Q
dp_ctl_synd_out_low[2] dp_ctl_synd_out_low[6]
0.945436 0.945436
path_sparc_ffu_dp_in_686
frf_dp_data[42]
dp_ctl_synd_out_high[6]
0.812538

This User Gave Thanks to Aia For This Post:
jypark22
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

using sed to get rid of duplicated columns...

I can not figure out this one, so I turn to unix.com for help, I have a file, in which there are some lines containing continuously duplicate columns, like the following adb abc abc asd adfj 123 123 123 345 234 444 444 444 444 444 23 and the output I want is adb abc asd adfj 123 345... (5 Replies)
Discussion started by: fedora
5 Replies

2. Shell Programming and Scripting

Deleting Doubled Block with sed or awk

hi there, i have a text file like that one: I like to delete the second block with the Start and End Line! Does anyone have a idea? Thanks for your help, Roland (4 Replies)
Discussion started by: rolandh
4 Replies

3. Shell Programming and Scripting

Deleting a line from a file with sed and awk?

cat file.txt fvnuiehuewf ruevhxncvkjrh zxjvurhfuwe jkhvBEGINvfnvf ijrgioe Trying to delete a line that has the pattern "BEGIN" cat sedtest filename=file.txt pattern=BEGIN sed "/^$pattern/d" "$filename" (9 Replies)
Discussion started by: cola
9 Replies

4. Shell Programming and Scripting

Deleting the first column with sed,awk or perl

336 brtr 256 hello Output: brtr hello How can i do this with sed,awk or perl? (5 Replies)
Discussion started by: cola
5 Replies

5. Shell Programming and Scripting

Deleting characters with sed,perl,awk

Input: :: gstreamer :: xine-lib :: xine-lib-extras Output should be: gstreamer xine-lib xine-lib-extras How can it be done with sed or perl? (12 Replies)
Discussion started by: cola
12 Replies

6. Shell Programming and Scripting

Can I use a shell script for deleting chunks from a watch folder?

Hello I have a unique problem of needing to delete large files slowly off of an XSan. I was wondering if there is a script I could use to delete 100gb chunks of files and folders that get placed in to a watch folder, slowly so as not to disrupt the other users. I would like to use Automator in... (0 Replies)
Discussion started by: ajsoto
0 Replies

7. Shell Programming and Scripting

deleting lines between patterns using sed or awk

hi, Here is excerpt from my xml file <!-- The custom module to do the authentication for LDAP --> </login-module> <login-module code="com.nlayers.seneca.security.LdapLogin" flag="sufficient"> <module-option... (1 Reply)
Discussion started by: sunrexstar
1 Replies

8. UNIX for Dummies Questions & Answers

Awk: Print out overlapping chunks of file - rows 0-20,10-30,20-40 etc.

First time poster, but the forum has saved my bacon more times than... Lots. Anyway, I have a text file, and wanted to use Awk (or any other sensible program) to print out overlapping sections, or arbitrary length. To describe by example, for file 1 2 3 4 5 etc... I want the out put... (3 Replies)
Discussion started by: matfald
3 Replies

9. Shell Programming and Scripting

awk for splitting file in constant chunks

Hi gurus, I wanted to split main file in 20 files with 2500 lines in each file. My main file conatins total 2500*20 lines. Following awk I made, but it is breaking with error. awk '{ for (i = 1; i <= 20; i++) { starts=2500*$i-1; ends=2500*$i; NR>=starts && NR<=ends {f=My$i".txt"; print >> f;... (10 Replies)
Discussion started by: mukesh.lalwani
10 Replies

10. Shell Programming and Scripting

Deleting lines containing duplicated strings

Dear all, I always appreciate your help. I would like to delete lines containing duplicated strings in the second column. test.txt 658 invert_d2e_q_reg_0_/Qalu_ecl_zlow_e 0.825692 659 invert_d2e_q_reg_0_/Qalu_byp_rd_data_e 0.825692 660 invert_d2e_q_reg_0_/Qalu_byp_rd_data_e 0.825692... (1 Reply)
Discussion started by: jypark22
1 Replies

LEARN ABOUT CENTOS

curs_memleaks

curs_memleaks(3X)														 curs_memleaks(3X)

NAME

       _nc_freeall _nc_free_and_exit - curses memory-leak checking

SYNOPSIS

       #include <curses.h>

       void _nc_freeall(void);
       void _nc_free_and_exit(int);

DESCRIPTION

       These  functions  are used to simplify analysis of memory leaks in the ncurses library.	They are normally not available; they must be con-
       figured into the library at build time using the --disable-leaks option.  That compiles-in code that frees memory that normally	would  not
       be freed.

       Any implementation of curses must not free the memory associated with a screen, since (even after calling endwin), it must be available for
       use in the next call to refresh.  There are also chunks of memory held for performance reasons.	That makes it hard to analyze  curses  ap-
       plications  for memory leaks.  To work around this, one can build a debugging version of the ncurses library which frees those chunks which
       it can, and provides these functions to free all of the memory allocated by the ncurses library.

       The _nc_free_and_exit function is the preferred one since some of the memory which is freed may be required for the application to continue
       running.  Its parameter is the code to pass to the exit routine.

RETURN VALUE

       These functions do not return a value.

PORTABILITY

       These functions are not part of the XSI interface.

SEE ALSO

       curses(3X).

																 curs_memleaks(3X)
All times are GMT -4. The time now is 04:50 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy