Sponsored Content
Top Forums Shell Programming and Scripting ISSUE in handling multiple same name files :-( Post 302819479 by krishmaths on Tuesday 11th of June 2013 02:28:15 AM
Old 06-11-2013
Shell script solution

I tried this shell scripting solution.

Code:
DIR=/your/directory
PATTERNS=/tmp/available_patterns.txt
SORTED_PATTERNS=/tmp/unique_patterns.txt
NON_REPEATING=/tmp/non_repeating_files.dat

#Get all available file patterns or prefixes before the second "_"
for FILENAME in `ls $DIR`
do
 PATTERN=$(echo $FILENAME | awk -F"_" '{print $1"_"$2}' )
 echo $PATTERN >> $PATTERNS
done

#Get the unique patterns. Either sort -u or command uniq would work
sort -u $PATTERNS > $UNIQUE_PATTERNS

#From each unique pattern get the nr of occurrences and split the files
for FILENAME in `cat $UNIQUE_PATTERNS`
do
 OCCURS=$(ls ${FILENAME}* | wc -l)
 if [ $OCCURS -eq 1 ]	#Move the file to non_repeating
 then
	ls ${FILENAME}* >> $NON_REPEATING
 else	#Else sort by file size and move the max sized file to repeating
	ls -l ${FILENAME}* | sort -nrk5,5 | head -1 >>$REPEATING
 fi
done

Note: This is untested and loops through the directory twice
This User Gave Thanks to krishmaths For This Post:
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Multiple file handling

Dear All, I have two files, which looks like: File 1 124 235 152 178 156 142 178 163 159 File 2 124|5623 452|6698 178|9995 (8 Replies)
Discussion started by: rochitsharma
8 Replies

2. Emergency UNIX and Linux Support

rsync transferring multiple files issue

Hi, I want to specify multiple remote directories but want to transfer them in a single command with one connection with remote server. This avoids entering passwords repeatedly and is also efficient. e.g. rsync -vrt --size-only --delete user@host:/home/user1/dir1... (9 Replies)
Discussion started by: sardare
9 Replies

3. Shell Programming and Scripting

handling multiple files using awk command and wants to get separate out file for each

hai all I am new to the world of shell scripting I wanted to extract two columns from multiple files say around 25 files and i wanted to get the separate outfile for each input file tired using the following command to extract two columns from 25 files awk... (2 Replies)
Discussion started by: hema dhevi
2 Replies

4. Shell Programming and Scripting

Issue with Error handling,not able to continue the script further

Hi, I am trying to write a script to cleanup files in a log directory .. cd log find Datk** -mtime +7 -exec rm -f {} \; 2> /dev/null Have used the above to clean up files in log directory more then 7 days older. The file can be something like ( auto-generate by some processes and... (2 Replies)
Discussion started by: nss280
2 Replies

5. Shell Programming and Scripting

[SOLVED] Handling multiple files using awk

Hi, I am trying to process 2 files simultaneously using awk satisfying following condition, Both files contain 3 columns. It should take entry from column 1 from first file, look for that entry in file 2 and if found, add column 2 and column 3 from both files and output to third file. For e.g.... (4 Replies)
Discussion started by: muazfarooqaslam
4 Replies

6. Shell Programming and Scripting

UNIX file handling issue

I have a huge file semicolon( ; ) separated records are Pipe(|) delimited. e.g abc;def;ghi|jkl;mno;pqr|123;456;789 I need to replace the 50th field(semicolon separated) of each record with 9006. The 50th field can have no value e.g. ;; Can someone help me with the appropriate command. (3 Replies)
Discussion started by: Gurkamal83
3 Replies

7. UNIX for Dummies Questions & Answers

Large file data handling issue

I have a single record large file, semicolon ';' and pipe '|' separated. I am doing a vi on the file. It is throwing an error "File to long" I need to actually remove the last | symbol from this file. sed -e 's/\|*$//' filename is working fine for small files. But not working on this big... (13 Replies)
Discussion started by: Gurkamal83
13 Replies

8. UNIX for Dummies Questions & Answers

File handling issue

Hi All, I am running into an issue. I have a very big file. Wants to split it in smaller chunks. This file has multiple header/ trailers. Also, between each header/trailer there are records. Number of records in each header trailer combination can vary. Also, headers can start with... (3 Replies)
Discussion started by: Gurkamal83
3 Replies

9. Shell Programming and Scripting

awk script issue redirecting to multiple files after matching pattern

Hi All I am having one awk and sed requirement for the below problem. I tried multiple options in my sed or awk and right output is not coming out. Problem Description ############################################################### I am having a big file say file having repeated... (4 Replies)
Discussion started by: kshitij
4 Replies

10. UNIX for Beginners Questions & Answers

Issue with search and replacing multiple items in multiple files

Im having an issue when trying to replace the first column with a new set of values in multiple files. The results from the following code only replaces the files with the last set of values in val.txt. I want to replace all the files with all the values. for date in {1..31} do for val in... (1 Reply)
Discussion started by: ncwxpanther
1 Replies
Task::Weaken(3) 					User Contributed Perl Documentation					   Task::Weaken(3)

NAME
Task::Weaken - Ensure that a platform has weaken support DESCRIPTION
One recurring problem in modules that use Scalar::Util's "weaken" function is that it is not present in the pure-perl variant. While this isn't necesarily always a problem in a straight CPAN-based Perl environment, some operating system distributions only include the pure-Perl versions, don't include the XS version, and so weaken is then "missing" from the platform, despite passing a dependency on Scalar::Util successfully. Most notably this is RedHat Linux at time of writing, but other come and go and do the same thing, hence "recurring problem". The normal solution is to manually write tests in each distribution to ensure that "weaken" is available. This restores the functionality testing to a dependency you do once in your Makefile.PL, rather than something you have to write extra tests for each time you write a module. It should also help make the package auto-generators for the various operating systems play more nicely, because it introduces a dependency that they have to have a proper weaken in order to work. How this Task works Part of the problem seems to stem from the fact that some distributions continue to include modules even if they fail some of their tests. To get around that for this module, it will do a few dirty tricks. If Scalar::Util is not available at all, it will issue a normal dependency on the module. However, if Scalar::Util is relatively new ( it is >= 1.19 ) and the module does not have weaken, the install will bail out altogether with a long error encouraging the user to seek support from their vendor (this problem happens most often in vendor-packaged Perl versions). This distribution also contains tests to ensure that weaken is available using more normal methods. So if your module uses "weaken", you can just add the following to your Module::Install-based Makefile.PL (or equivalent). requires 'Task::Weaken' => 0; SUPPORT
Bugs should be always be reported via the CPAN bug tracker at <http://rt.cpan.org/NoAuth/ReportBug.html?Queue=Task-Weaken> For other issues,contact the author. AUTHOR
Adam Kennedy <adamk@cpan.org> SEE ALSO
Task, Scalar::Util, <http://ali.as/> COPYRIGHT
Copyright 2006 - 2011 Adam Kennedy. This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself. The full text of the license can be found in the LICENSE file included with this module. perl v5.16.3 2011-03-08 Task::Weaken(3)
All times are GMT -4. The time now is 03:18 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy