I have a file 1181CUSTOMER-L061411_003500.dat.Z having duplicate records in it.
my query is how to remove the duplicate record from a file with the help of a Unix command. Here I have to remove the second data from the file.
How can i remove the duplicate lines from a file, for example
sample123456Sample
testing123456testing
XXXXX131323XXXXX
YYYYY423432YYYYY
fsdfdsf123456gsdfdsd
all the duplicates from column 6-12 , must be deleted. I want to consider the first row, if same comes in the given range i want to... (1 Reply)
hi.. i have a file in the following format :-
name-a
age -12
address-123
age-12
phone-22222
============
name-ab
age -11
address-123
age-11
phone-222223
=============
name-abc
age -12
address-1234
age-12
phone-2222223
============= (2 Replies)
Hi,
I need to remove duplicates from a file. The file will be like this
0003 10101 20100120 abcdefghi
0003 10101 20100121 abcdefghi
0003 10101 20100122 abcdefghi
0003 10102 20100120 abcdefghi
0003 10103 20100120 abcdefghi
0003 10103 20100121 abcdefghi
Here if the first colum and... (6 Replies)
I am a beginner in Unix. Though have been asked to write a script to filter(remove duplicates) data from a .dat file. File is very huge containig billions of records.
contents of file looks like
30002157,40342424,OTC,mart_rec,100, ,0
30002157,40343369,OTC,mart_rec,95, ,0... (6 Replies)
Heya there,
A small selection of my data is shown below.
DATE TIME FRAC_DAYS_SINCE_JAN1
2011-06-25 08:03:20.000 175.33564815
2011-06-25 08:03:25.000 175.33570602
2011-06-25 ... (4 Replies)
Hi,
The source system has created the file in the dat format and put into the linux directory as mentioned below. I want to do foloowing things.
a) Delete the Line started with <CR><LF> in the record
b)Also line
...........................................................<CR><LF>
... (1 Reply)
Hi,
I have a tablular separated file and I want to remove all the rows that have duplicates. The diuplicates I need to check are in column 13.
I have tried to use awk but I have no Idea how to keep the duplicate file.
awk 'FNR==NR{a++;next}(a> 1)' tomodify.txt tomodify.txt > new.txt
... (4 Replies)
Hi all,
I have a issues while loading a flat file to the DB. It is taking much time.
When analyzed i found out that there are duplicates entry in the flat file.
There are 2 type of Duplicate entry.
1) is entire row is duplicate. ( i can use sort | uniq) to remove the duplicated entry.
2) the... (4 Replies)
Discussion started by: samjoshuab
4 Replies
LEARN ABOUT LINUX
datadm
datadm(1M)datadm(1M)NAME
datadm - maintain DAT static registry file
SYNOPSIS
/usr/bin/datadm [-v] [-u] [-a service_provider.conf] [-r service_provider.conf]
The datadm utility maintains the DAT static registry file, dat.conf(4).
This administrative configuration program allows uDAPL service providers to add and remove themselves to the dat.conf file.
You can add or remove interface adapters that a service provider supports from a system after its installation. You can use datadm to
update the dat.conf file to reflect the current state of the system. A new set of interface adapters for all the service providers cur-
rently installed is regenerated.
The following options are supported:
-a service_provider.conf
Enumerate each device entry in the service_provider.conf(4) file into a list of interface adapters, that is, interfaces to external
network that are available to uDAPL consumers.
-r service_provider.conf
Remove the list of interface adapters that corresponds to the device entry in the service_provider.conf(4) file.
-u
Update the dat.conf to reflect the current state of the system with an up to date set of interface adapters for the service providers
that are currently listed in the DAT static registry.
-v
Display the DAT static registry file, dat.conf.
Example 1: Enumerating a Device Entry
The following example enumerates a device entry in the service_provider.conf(4) file into interface adapters in the dat.conf(4) file.
Assume that SUNW has a service provider library that supports the device tavor. It has a service_provider.conf(4) file installed in the
directory /usr/share/dat/SUNWudaplt.conf with a single entry as follows:
driver_name=tavor u1.2 nonthreadsafe default udapl_tavor.so.1 SUNW.1.0 ""
tavor is an Infiniband Host Channel Adapter with two ports. Both IB ports exist in a single IB partition, 0x8001. If an IP interface is
plumbed to each port, there are two IPoIB device instances, ibd0 and ibd1:
# ls -l /dev/ibd*
/dev/ibd0 -> /devices/pci@1/pci15b3,5a44@0/ibport@1,8001,ipib:ibd0
/dev/ibd1 -> /devices/pci@1/pci15b3,5a44@0/ibport@2,8001,ipib:ibd1
Running the command, datadm -a /usr/share/dat/SUNWudaplt.conf appends two new entries (if they do not already exists]) in the
/etc/dat/dat.conffile:
ibd0 u1.2 nonthreadsafe default udapl_tavor.so.1 SUNW.1.0 ""
"driver_name=tavor"
ibd1 u1.2 nonthreadsafe default udapl_tavor.so.1 SUNW.1.0 ""
"driver_name=tavor"
Example 2: Updating the dat.conf to Reflect the Current State of the System
A new IB partition, 0x8002 is added to the above example covering port 1 of the Host Channel Adapter. If a new IP interface is plumbed to
port 1/partition 0x8002, there is a third IPoIB device instance: ibd2.
# ls -l /dev/ibd*
/dev/ibd0 -> /devices/pci@1/pci15b3,5a44@0/ibport@1,8001,ipib:ibd0
/dev/ibd1 -> /devices/pci@1/pci15b3,5a44@0/ibport@2,8001,ipib:ibd1
/dev/ibd2 -> /devices/pci@1/pci15b3,5a44@0/ibport@1,8002,ipib:ibd2
Running datadm -u command, updates the /etc/dat/dat.conf file with a new entry added reflecting the current state of the system.
datadm -v shows that there are now three entries in the /etc/dat/dat.conf file:
ibd0 u1.2 nonthreadsafe default udapl_tavor.so.1 SUNW.1.0 ""
"driver_name=tavor"
ibd1 u1.2 nonthreadsafe default udapl_tavor.so.1 SUNW.1.0 ""
"driver_name=tavor"
ibd2 u1.2 nonthreadsafe default udapl_tavor.so.1 SUNW.1.0 ""
"driver_name=tavor"
/etc/dat/dat.conf DAT static registry file
See attributes(5) for descriptions of the following attributes:
+-----------------------------+-----------------------------+
| ATTRIBUTE TYPE | ATTRIBUTE VALUE |
+-----------------------------+-----------------------------+
|Availability |SUNWudaplu |
+-----------------------------+-----------------------------+
|Interface Stability |Evolving |
+-----------------------------+-----------------------------+
pkgadd(1M), pkgrm(1M), libdat(3LIB), dat.conf(4), service_provider.conf(4). attributes(5)
18 Jun 2004 datadm(1M)