Sponsored Content
Top Forums Shell Programming and Scripting Delete duplicates via script? Post 302463512 by rdcwayx on Sunday 17th of October 2010 10:19:40 PM
Old 10-17-2010
which env? Is the diff command in your system support to diff folder?
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

An interactive way to delete duplicates

1)I am trying to write a script that works interactively lists duplicated records on certain field/column and asks user to delete one or more. And finally it deletes all the records the used has asked for. I have an idea to store those line numbers in an array, not sure how to do this in... (3 Replies)
Discussion started by: chvs2000
3 Replies

2. Shell Programming and Scripting

How can i delete the duplicates based on one column of a line

I have my data something like this (08/03/2009 22:57:42.414)(:) king aaaaaaaaaaaaaaaa bbbbbbbbbbbbbbbbbbbbbb (08/03/2009 22:57:42.416)(:) John cccccccccccc cccccvssssssssss baaaaa (08/03/2009 22:57:42.417)(:) Michael ddddddd tststststtststts (08/03/2009 22:57:42.425)(:) Ravi... (11 Replies)
Discussion started by: rdhanek
11 Replies

3. Shell Programming and Scripting

how can I delete duplicates in the log?

I have a log file and I am trying to run a script against it to search for key issues such as invalid users, errors etc. In one part, I grep for session closed and get a lot of the same thing,, ie. root username etc. I want to remove the multiple root and just have it do a count, like wc -l ... (5 Replies)
Discussion started by: taekwondo
5 Replies

4. Shell Programming and Scripting

Delete Duplicates on the basis of two column values.

Hi All, i need ti delete two duplicate processss which are running on the same device type (column 1) and port ID (column 2). here is the sample data p1sc1m1 15517 11325 0 01:00:24 ? 0:00 scagntclsx25octtcp 2967 in3v mvmp01 0 8000 N S 969 750@751@752@ p1sc1m1 15519 11325 0 01:00:24 ? ... (5 Replies)
Discussion started by: neeraj617
5 Replies

5. Shell Programming and Scripting

Fastest way to delete duplicates from a large filelist.....

OK I have two filelists...... The first is formatted like this.... /path/to/the/actual/file/location/filename.jpg and has up to a million records The second list shows filename.jpg where there is more then on instance. and has maybe up to 65,000 records I want to copy files... (4 Replies)
Discussion started by: Bashingaway
4 Replies

6. Shell Programming and Scripting

delete from line and remove duplicates

My Input.....file1 ABCDE4435 Connected to 107.71.136.122 (SubNetwork=ONRM_RootMo_R SubNetwork=XYVLTN29CRBR99 MeContext=ABCDE4435 ManagedElement=1) ABCDE4478 Connected to 166.208.30.57 (SubNetwork=ONRM_RootMo_R SubNetwork=KLFMTN29CR0R04 MeContext=ABCDE4478 ManagedElement=1) ABCDE4478... (5 Replies)
Discussion started by: pareshkp
5 Replies

7. Shell Programming and Scripting

Delete duplicates in CA bundle

I do have a big CA bundle certificate file and each time if i get request to add new certificate to the existing bundle i need to make sure it is not present already. How i can validate the duplicates. The alignment of the certificate within the bundle seems to be different. Example: Cert 1... (7 Replies)
Discussion started by: diva_thilak
7 Replies

8. Shell Programming and Scripting

Delete only if duplicates found in each record

Hi, i have another problem. I have been trying to solve it by myself but failed. inputfile ;; ID T08578 NAME T08578 SBASE 30696 EBASE 32083 TYPE P func just test func chronology func cholesterol func null INT 30765-37333 INT 37154-37318 Link 5546 Link 8142 (4 Replies)
Discussion started by: redse171
4 Replies

9. Shell Programming and Scripting

Script to compare partial filenames in two folders and delete duplicates

Background: I use a TV tuner card to capture OTA video files (.mpeg) and then my Plex Media Server automatically optimizes the files (transcodes for better playback) and places them in a new directory. I have another Plex Library pointing to the new location for the optimized .mp4 files. This... (2 Replies)
Discussion started by: shaky
2 Replies

10. Shell Programming and Scripting

To Delete the duplicates using Part of File Name

I am using the below script to delete duplicate files but it is not working for directories with more than 10k files "Argument is too long" is getting for ls -t. Tried to replace ls -t with find . -type f \( -iname "*.xml" \) -printf '%T@ %p\n' | sort -rg | sed -r 's/* //' | awk... (8 Replies)
Discussion started by: gold2k8
8 Replies
Net::Ifconfig::Wrapper(3pm)				User Contributed Perl Documentation			       Net::Ifconfig::Wrapper(3pm)

NAME
Net::Ifconfig::Wrapper - provides a unified way to configure network interfaces on FreeBSD, OpenBSD, Solaris, Linux, OS X, and WinNT (from Win2K). Version 0.11 SYNOPSIS
#!/usr/local/bin/perl -w # uni-ifconfig.pl # The unified ifconfig command. # Works the same way on FreeBSD, OpenBSD, Solaris, Linux, OS X, WinNT (from Win2K). # Note: due of Net::Ifconfig::Wrapper limitations 'inet' and 'down' commands # are not working on WinNT. +/-alias are working, of course. use strict; use Net::Ifconfig::Wrapper; my $Usage = << 'EndOfText'; uni-ifconfig.pl # Print this notice uni-ifconfig.pl -a # Print info about all interfaces uni-ifconfig.pl <iface> # Print info obout specified interface uni-ifconfig.pl <iface> down # Bring specified interface down uni-ifconfig.pl <iface> inet <AAA.AAA.AAA.AAA> mask <MMM.MMM.MMM.MMM> # Set the specified address on the specified interface # and bring this interface up uni-ifconfig.pl <iface> inet <AAA.AAA.AAA.AAA> mask <MMM.MMM.MMM.MMM> [+]alias # Set the specified alias address # on the specified interface uni-ifconfig.pl <iface> inet <AAA.AAA.AAA.AAA> [mask <MMM.MMM.MMM.MMM>] -alias # Remove specified alias address # from the specified interface EndOfText my $Info = Net::Ifconfig::Wrapper::Ifconfig('list', '', '', '') or die $@; scalar(keys(%{$Info})) or die "No one interface found. Something wrong? "; if (!scalar(@ARGV)) { print $Usage; exit 0; } if ($ARGV[0] eq '-a') { defined($ARGV[1]) and die $Usage; foreach (sort(keys(%{$Info}))) { print IfaceInfo($Info, $_); }; exit 0; }; $Info->{$ARGV[0]} or die "Interface '$ARGV[0]' is unknown "; if (!defined($ARGV[1])) { print IfaceInfo($Info, $ARGV[0]); exit 0; } my $CmdLine = join(' ', @ARGV); my $Result = undef; if ($CmdLine =~ m/As*([w{}-]+)s+downs*/i) { $Result = Net::Ifconfig::Wrapper::Ifconfig('down', $1, '', ''); } elsif ($CmdLine =~ m/As*([w{}-]+)s+inets+(d{1,3}(?:.d{1,3}){3})s+masks+(d{1,3}(?:.d{1,3}){3})s*/i) { $Result = Net::Ifconfig::Wrapper::Ifconfig('inet', $1, $2, $3); } elsif ($CmdLine =~ m/As*([w{}-]+)s+inets+(d{1,3}(?:.d{1,3}){3})s+masks+(d{1,3}(?:.d{1,3}){3})s++?aliass*/i) { $Result = Net::Ifconfig::Wrapper::Ifconfig('+alias', $1, $2, $3); } elsif ($CmdLine =~ m/As*([w{}-]+)s+inets+(d{1,3}(?:.d{1,3}){3})s+(:?masks+(d{1,3}(?:.d{1,3}){3})s+)?-aliass*/i) { $Result = Net::Ifconfig::Wrapper::Ifconfig('-alias', $1, $2, ''); } else { die $Usage; }; $Result or die $@; exit 0; sub IfaceInfo { my ($Info, $Iface) = @_; my $Res = "$Iface: ".($Info->{$Iface}{'status'} ? 'UP' : 'DOWN')." "; while (my ($Addr, $Mask) = each(%{$Info->{$Iface}{'inet'}})) { $Res .= sprintf(" inet %-15s mask $Mask ", $Addr); }; $Info->{$Iface}{'ether'} and $Res .= " ether ".$Info->{$Iface}{'ether'}." "; $Info->{$Iface}{'descr'} and $Res .= " descr '".$Info->{$Iface}{'descr'}."' "; return $Res; }; DESCRIPTION
This module provides a unified way to configure the network interfaces on FreeBSD, OpenBSD, Solaris, Linux, OS X, and WinNT (from Win2K) systems. Only "inet" (IPv4) and "ether" (MAC) addresses are supported at the moment On Unixes this module calls the system "ifconfig" command to perform the actions. On Windows the functions from IpHlpAPI.DLL are called. For all supported Unixes "Net::Ifconfig::Wrapper" expect "ifconfig" command to be "/sbin/ifconfig". Module was tested on FreeBSD 4.7,4.8,5.3 (Intel), RedHat 6.2,7.3,8.0 (Intel), Win2000 Pro (Intel), OpenBSD 3.1 (SPARC), Solaris 7 (SPARC), OS X 10.3 (aka Panther), OS X 10.4 (aka Tiger). In MSWin32 family only WinNT is supported. In WinNT family only Win2K or later is supported. The Net::Ifconfig::Wrapper methods "Ifconfig(Command, Interface, Address, Netmask);" The first and the last method of the "Net::Ifconfig::Wrapper" module. Do all the job. The particular action is described by the $Command parameter. $Command could be: 'list' "Ifconfig('list', '', '', '')" will return the reference to the hash contains the information about interfaces. The structure of this hash is the following: {IfaceName => {'status' => 0|1 # The status of the interface. 0 means down, 1 means up 'ether' => MACaddr, # The ethernet address of the interface if available 'descr' => Description, # The description of the interface if available 'inet' => {IPaddr1 => NetMask, # The IP address and his netmask, both are in AAA.BBB.CCC.DDD notation IPaddr2 => NetMask, ... }, ... }; Interface, Address, Netmask parameters are ignored. The following programs are called: FreeBSD "/sbin/ifconfig -a" Solaris "/sbin/ifconfig -a" OpenBSD "/sbin/ifconfig -A" Linux "/sbin/ifconfig -a" OS X "/sbin/ifconfig -a" MSWin32 "GetAdaptersInfo" function from "IpHlpAPI.DLL" Limitations: OpenBSD: "/sbin/ifconfig -A" command is not returning information about MAC addresses so we are trying to get it from '/usr/sbin/arp -a' command (first 'static' entry). If no one present the 'ff:ff:ff:ff:ff' address is returned. MSWin32: "GetAdaptersInfo" function is not returning information about interface which have address 127.0.0.1 binded so "Net::Ifconfig::Wrapper" have no ability to display it. Not limitation but little problem: MSWin32 interface names are not human-readable, they looks like "{843C2077-30EC-4C56-A401-658BB1E42BC7}" (on Win2K at least). 'inet' This function is used to set IPv4 address on interface. It have to be called as Ifconfig('inet', $IfaceName, $Addr, $Mask); $IfaceName is an interface name as displayed by 'list' command $Addr is an IPv4 address in the "AAA.AAA.AAA.AAA" notation $Mask is an IPv4 subnet mask in the "MMM.MMM.MMM.MMM" notation The following actual "ifconfig" programs are called FreeBSD "/sbin/ifconfig %Iface% inet %Addr% netmask %Mask% up" Solaris "/sbin/ifconfig %Iface% inet %Addr% netmask %Mask% up" OpenBSD "/sbin/ifconfig %Iface% inet %Addr% netmask %Mask% up" Linux "/sbin/ifconfig %Iface% inet %Addr% netmask %Mask% up" OS X "/sbin/ifconfig %Iface% inet %Addr% netmask %Mask% up" MSWin32: nothing :( Limitations: MSWin32: I did not find the relaible way to recognize the "main" address on the Win32 network interface, so I have disabled this functionality. If you know the way please let me know. 'up' Just a synonym for 'inet' 'down' This function is used to bring specified interface down. It have to be called as Ifconfig('inet', $IfaceName, '', ''); $IfaceName is an interface name as displayed by 'list' command Address and Netmask are ignored. The following actual "ifconfig" programs are called FreeBSD "/sbin/ifconfig %Iface% down" Solaris "/sbin/ifconfig %Iface% down" OpenBSD "/sbin/ifconfig %Iface% down" Linux "/sbin/ifconfig %Iface% down" OS X "/sbin/ifconfig %Iface% down" MSWin32 nothing :( Limitations: MSWin32: I did not find the way to implement the 'up' command so I did not implement 'down'. '+alias' This function is used to set IPv4 alias address on interface. It have to be called as Ifconfig('+alias', $IfaceName, $Addr, $Mask); $IfaceName is an interface name as displayed by 'list' command $Addr is an IPv4 address in the "AAA.AAA.AAA.AAA" notation $Mask is an IPv4 subnet mask in the "MMM.MMM.MMM.MMM" notation The following actual "ifconfig" programs are called FreeBSD "/sbin/ifconfig %Iface% inet %Addr% netmask %Mask% alias" Solaris "/sbin/ifconfig %Iface%:%Logic% inet %Addr% netmask %Mask% up" OpenBSD "/sbin/ifconfig %Iface% inet %Addr% netmask %Mask% alias" Linux "/sbin/ifconfig %Iface%:%Logic% inet %Addr% netmask %Mask% up" OS X "/sbin/ifconfig %Iface% inet %Addr% netmask %Mask% alias" MSWin32 "AddIPAddress" function from "IpHlpAPI.DLL" First available logic interface is taken automatically for Solaris and Linux 'alias' Just a synonim for '+alias' '-alias' This function is used to remove IPv4 alias address from interface. It have to be called as Ifconfig('-alias', $IfaceName, $Addr, ''); $IfaceName is an interface name as displayed by 'list' command $Addr is an IPv4 address in the "AAA.AAA.AAA.AAA" notation Netmask> parameter is ignored The following actual "ifconfig" programs are called FreeBSD "/sbin/ifconfig %Iface% inet %Addr% -alias" Solaris "/sbin/ifconfig %Iface%:%Logic% down" OpenBSD "/sbin/ifconfig %Iface% inet %Addr% -alias" Linux "/sbin/ifconfig %Iface%:%Logic% down" OS X "/sbin/ifconfig %Iface% inet %Addr% -alias" MSWin32 "DeleteIPAddress" function from "IpHlpAPI.DLL" Appropriate logic interface is obtained automatically for Solaris and Linux On success "Ifconfig(...)" returns the defined value. Actually, it is a reference to the array contains the output of the actual "ifconfig" program called. In case of troubles "Ifconfig(...)" returns 'undef' value, $@ variable contains the error message. EXPORT None by default. AUTHOR
Daniel Podolsky, <tpaba@cpan.org> SEE ALSO
ifconfig(8), Internet Protocol Helper in Platform SDK. perl v5.14.2 2012-01-19 Net::Ifconfig::Wrapper(3pm)
All times are GMT -4. The time now is 07:19 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy