Sponsored Content
Top Forums Shell Programming and Scripting List dublicated files into a file Post 302960769 by bulpak on Thursday 19th of November 2015 12:00:12 PM
Old 11-19-2015
List dublicated files into a file

Dear All,

I have many files in a directory similar in below format (in order to understand each group files are separated from others by blank lines ). I want to find duplicate filenames and write them into a new file line by line. I tried several scripts but I couldn't be successful.

Do you have any suggestion?

Code:
2222.00.AAA.AHE.DAT
2222.00.AAA.AHN.DAT
2222.00.AAA.AHZ.DAT

2222.01.BBB.AHE.DAT
2222.02.BBB.AHE.DAT
2222.03.BBB.AHE.DAT
2222.01.BBB.AHN.DAT
2222.02.BBB.AHN.DAT
2222.03.BBB.AHN.DAT
2222.04.BBB.AHN.DAT
2222.01.BBB.AHZ.DAT
2222.02.BBB.AHZ.DAT

2222.00.CCC.AHE.DAT
2222.00.CCC.AHN.DAT
2222.00.CCC.AHZ.DAT

2222.01.DDD.AHE.DAT
2222.02.DDD.AHE.DAT
2222.03.DDD.AHE.DAT
2222.04.DDD.AHE.DAT
2222.01.DDD.AHN.DAT
2222.02.DDD.AHN.DAT
2222.01.DDD.AHZ.DAT
2222.02.DDD.AHZ.DAT
2222.03.DDD.AHZ.DAT

It should be below format after scripting.

Code:
2222.01.BBB.AHE.DAT
2222.02.BBB.AHE.DAT
2222.03.BBB.AHE.DAT
2222.01.BBB.AHN.DAT
2222.02.BBB.AHN.DAT
2222.03.BBB.AHN.DAT
2222.04.BBB.AHN.DAT
2222.01.BBB.AHZ.DAT
2222.02.BBB.AHZ.DAT
2222.01.DDD.AHE.DAT
2222.02.DDD.AHE.DAT
2222.03.DDD.AHE.DAT
2222.04.DDD.AHE.DAT
2222.01.DDD.AHN.DAT
2222.02.DDD.AHN.DAT
2222.01.DDD.AHZ.DAT
2222.02.DDD.AHZ.DAT
2222.03.DDD.AHZ.DAT

 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

filesize from a file which has the list of files.

i have a file myfile. it has the below entries /temp/firstfile /temp/secondfile and many more.. okay, now, i want to addup all the space occupied by this file hmmm, but i met with a problem in getting each file out. i did a silly command like more myfile | grep temp | ls -ltr and it... (3 Replies)
Discussion started by: yls177
3 Replies

2. UNIX for Dummies Questions & Answers

how to list the files using File Descriptors

hello, I have written a script named listall.sh with the following codes init. #!/bin/bash PATH="/proj/cmon/$1" echo $PATH if ; then echo "Usage: $0 ***" exit 1 else ls -l $PATH/*.sc fi Here there are 3 subdirectories (namely - src, data and jobs)under /proj/cmon, so... (2 Replies)
Discussion started by: shyjuezy
2 Replies

3. Shell Programming and Scripting

ls > file - Creating file containing the list of all files present in a directory

Hi All, I need to create a file which contains the list of all the files present in that directory. e.g., ls /export/home/user/*.dat > list_file.dat but what i am getting is: $ ls /export/home/user/*.dat > list_file.dat /export/home/user/*.dat: No such file or directory But I have... (1 Reply)
Discussion started by: pranavagarwal
1 Replies

4. Shell Programming and Scripting

List the file or files with last modification date

hi. I need help my programing friends :p I need to list all the files with a certain name (for example FileName) by last modification date but only the one with the last date. If there are two files with the same name and same modification date it should print the both. For example in this set... (6 Replies)
Discussion started by: KitFisto
6 Replies

5. Shell Programming and Scripting

Delete old files but with exclusion with file list

Hello Can you please help and check what im missing on script below the goal is to delete the old files more than 7 days old but not the excluded file list inside excluded.dat file #!/bin/sh EXCLUDE=/path/to/exclude/exclude.dat FIND=/bin/find for xfile in '(read $EXCLUDE)' do $FIND... (9 Replies)
Discussion started by: angst_nu
9 Replies

6. Shell Programming and Scripting

Take a list if strings from a file and search them in a list of files and report them

I have a file 1.txt with the below contents. -----cat 1.txt----- 1234 5678 1256 1234 1247 ------------------- I have 3 more files in a folder -----ls -lrt------- A1.txt A2.txt A3.txt ------------------- The contents of those three files are similar format with different data values... (8 Replies)
Discussion started by: realspirituals
8 Replies

7. Shell Programming and Scripting

File name from a List of files

Hi, Greetings!! I'm grepping a string from a series of files, using the below code (how ever the awk is not grepping between 'from' & 'to' time!) awk '$0>=$from&&$0<=$to' from=$START_TIME to=$STOP_TIME $logpath/UL_`date +%Y%m%d`_Scheduler*.log.csv > temp-grep.txt Out of 50 files,... (9 Replies)
Discussion started by: bhargav_k
9 Replies

8. Shell Programming and Scripting

File that Contains a List of Files

I have a first text file (LoopFiles.txt) that contains another list of text files. I need to run NAWK commands on each of the files that are listed in the first text file. I have proven the existence of the first file with ls -l But I get a message that my first file doesnt exist. cd... (5 Replies)
Discussion started by: he204035
5 Replies

9. Shell Programming and Scripting

How to list files that are not first two files date & last file date for every month-year?

Hi All, I need to find all files other than first two files dates & last file date for month and month/year wise list. lets say there are following files in directory Mar 19 2012 c.txt Mar 19 2012 cc.txt Mar 21 2012 d.txt Mar 22 2012 f.txt Mar 24 2012 h.txt Mar 25 2012 w.txt Feb 12... (2 Replies)
Discussion started by: Makarand Dodmis
2 Replies

10. Shell Programming and Scripting

While loop a file containing list of file names until the files are found?

Hi, I have a control file which will contain all filenames(300) files. Loop through all the file names in the control files and check the existence of this file in another directory(same server). I need to infinitely(2 hrs) run this while loop until all the files are found. Once a file is found,... (5 Replies)
Discussion started by: laknar
5 Replies
dat_ia_open(3DAT)				     Direct Access Transport Library Functions					 dat_ia_open(3DAT)

NAME
dat_ia_open - open an Interface Adapter (IA) SYNOPSIS
cc [ flag... ] file... -ldat [ library... ] #include <dat/udat.h> DAT_RETURN dat_ia_open ( IN const DAT_NAME_PTR ia_name_ptr, IN DAT_COUNT async_evd_min_qlen, INOUT DAT_EVD_HANDLE *async_evd_handle, OUT DAT_IA_HANDLE *ia_handle ) PARAMETERS
ia_name_ptr Symbolic name for the IA to be opened. The name should be defined by the Provider registration. async_evd_min_qlen Minimum length of the Asynchronous Event Dispatcher queue. async_evd_handle Pointer to a handle for an Event Dispatcher for asynchronous events generated by the IA. This parameter can be DAT_EVD_ASYNC_EXISTS to indicate that there is already EVD for asynchronous events for this Interface Adapter or DAT_HANDLE_NULL for a Provider to generate EVD for it. ia_handle Handle for an open instance of a DAT IA. This handle is used with other functions to specify a particular instance of the IA. DESCRIPTION
The dat_ia_open() function opens an IA by creating an IA instance. Multiple instances (opens) of an IA can exist. The value of DAT_HANDLE_NULL for async_evd_handle (*async_evd_handle == DAT_HANDLE_NULL) indicates that the default Event Dispatcher is created with the requested async_evd_min_qlen. The async_evd_handle returns the handle of the created Asynchronous Event Dispatcher. The first Consumer that opens an IA must use DAT_HANDLE_NULL because no EVD can yet exist for the requested ia_name_ptr. The Asynchronous Event Dispatcher (async_evd_handle) is created with no CNO (DAT_HANDLE_NULL). Consumers can change these values using dat_evd_modify_cno(3DAT). The Consumer can modify parameters of the Event Dispatcher using dat_evd_resize(3DAT) and dat_evd_modify_cno(). The Provider is required to provide a queue size at least equal to async_evd_min_qlen, but is free to provide a larger queue size or dynam- ically enlarge the queue when needed. The Consumer can determine the actual queue size by querying the created Event Dispatcher instance. If async_evd_handle is not DAT_HANDLE_NULL, the Provider does not create an Event Dispatcher for an asynchronous event and the Provider ignores the async_evd_min_qlen value. The async_evd_handle value passed in by the Consumer must be an asynchronous Event Dispatcher created for the same Provider (ia_name_ptr). The Provider does not have to check for the validity of the Consumer passed in async_evd_handle. It is the Consumer responsibility to guarantee that async_evd_handle is valid and for this Provider. How the async_evd_handle is passed between DAT Consumers is out of scope of the DAT specification. If the Provider determines that the Consumer-provided async_evd_handle is invalid, the operation fails and returns DAT_INVALID_HANDLE. The async_evd_handle remains unchanged, so the returned async_evd_handle is the same the Consumer passed in. All asynchronous notifications for the open instance of the IA are directed by the Provider to the Consumer passed in Asynchronous Event Dispatcher specified by async_evd_handle. Consumer can specify the value of DAT_EVD_ASYNC_EXISTS to indicate that there exists an event dispatcher somewhere else on the host, in user or kernel space, for asynchronous event notifications. It is up to the Consumer to ensure that this event dispatcher is unique and unambiguous. A special handle may be returned for the Asynchronous Event Dispatcher for this scenario, DAT_EVD_OUT_OF_SCOPE, to indicate that there is a default Event Dispatcher assigned for this Interface Adapter, but that it is not in a scope where this Consumer may directly invoke it. The Asynchronous Event Dispatcher is an Object of both the Provider and IA. Each Asynchronous Event Dispatcher bound to an IA instance is notified of all asynchronous events, such that binding multiple Asynchronous Event Dispatchers degrades performance by duplicating asyn- chronous event notifications for all Asynchronous Event Dispatchers. Also, transport and memory resources can be consumed per Event Dis- patcher bound to an IA As with all Event Dispatchers, the Consumer is responsible for synchronizing access to the event queue. Valid IA names are obtained from dat_registry_list_providers(3DAT). RETURN VALUES
DAT_SUCCESS The operation was successful. DAT_INSUFFICIENT_RESOURCES The operation failed due to resource limitations. DAT_INVALID_PARAMETER Invalid parameter. DAT_PROVIDER_NOT_FOUND The specified provider was not registered in the registry. DAT_INVALID_HANDLE Invalid DAT handle; async_evd_handle is invalid. USAGE
The dat_ia_open() function is the root method for the Provider, and, thus, all Objects. It is the root handle through which the Consumer obtains all other DAT handles. When the Consumer closes its handle, all its DAT Objects are released. The dat_ia_open() function is the workhorse method that provides an IA instance. It can also initialize the Provider library or do any other registry-specific functions. The dat_ia_open() function creates a unique handle for the IA to the Consumer. All further DAT Objects created for this Consumer reference this handle as their owner. The dat_ia_open() function can use a reference count for the Provider Library to ensure that the Provider Library cannot be removed when it is in use by a DAT Consumer. ATTRIBUTES
See attributes(5) for descriptions of the following attributes: +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | +-----------------------------+-----------------------------+ |Interface Stability |Standard: uDAPL, 1.1, 1.2 | +-----------------------------+-----------------------------+ |MT-Level |Safe | +-----------------------------+-----------------------------+ SEE ALSO
dat_evd_modify_cno(3DAT), dat_evd_resize(3DAT), dat_ia_close(3DAT), dat_registry_list_providers(3DAT), libdat(3LIB), attributes(5) SunOS 5.10 16 Jul 2004 dat_ia_open(3DAT)
All times are GMT -4. The time now is 10:26 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy