Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: ISSUE in handling multiple same name files :-(
Top Forums Shell Programming and Scripting ISSUE in handling multiple same name files :-( Post 302819479 by krishmaths on Tuesday 11th of June 2013 02:28:15 AM
Old 06-11-2013
Shell script solution
I tried this shell scripting solution.

Code:
DIR=/your/directory
PATTERNS=/tmp/available_patterns.txt
SORTED_PATTERNS=/tmp/unique_patterns.txt
NON_REPEATING=/tmp/non_repeating_files.dat

#Get all available file patterns or prefixes before the second "_"
for FILENAME in `ls $DIR`
do
 PATTERN=$(echo $FILENAME | awk -F"_" '{print $1"_"$2}' )
 echo $PATTERN >> $PATTERNS
done

#Get the unique patterns. Either sort -u or command uniq would work
sort -u $PATTERNS > $UNIQUE_PATTERNS

#From each unique pattern get the nr of occurrences and split the files
for FILENAME in `cat $UNIQUE_PATTERNS`
do
 OCCURS=$(ls ${FILENAME}* | wc -l)
 if [ $OCCURS -eq 1 ]	#Move the file to non_repeating
 then
	ls ${FILENAME}* >> $NON_REPEATING
 else	#Else sort by file size and move the max sized file to repeating
	ls -l ${FILENAME}* | sort -nrk5,5 | head -1 >>$REPEATING
 fi
done

Note: This is untested and loops through the directory twice
This User Gave Thanks to krishmaths For This Post:
emily
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Multiple file handling

Dear All, I have two files, which looks like: File 1 124 235 152 178 156 142 178 163 159 File 2 124|5623 452|6698 178|9995 (8 Replies)
Discussion started by: rochitsharma
8 Replies

2. Emergency UNIX and Linux Support

rsync transferring multiple files issue

Hi, I want to specify multiple remote directories but want to transfer them in a single command with one connection with remote server. This avoids entering passwords repeatedly and is also efficient. e.g. rsync -vrt --size-only --delete user@host:/home/user1/dir1... (9 Replies)
Discussion started by: sardare
9 Replies

3. Shell Programming and Scripting

handling multiple files using awk command and wants to get separate out file for each

hai all I am new to the world of shell scripting I wanted to extract two columns from multiple files say around 25 files and i wanted to get the separate outfile for each input file tired using the following command to extract two columns from 25 files awk... (2 Replies)
Discussion started by: hema dhevi
2 Replies

4. Shell Programming and Scripting

Issue with Error handling,not able to continue the script further

Hi, I am trying to write a script to cleanup files in a log directory .. cd log find Datk** -mtime +7 -exec rm -f {} \; 2> /dev/null Have used the above to clean up files in log directory more then 7 days older. The file can be something like ( auto-generate by some processes and... (2 Replies)
Discussion started by: nss280
2 Replies

5. Shell Programming and Scripting

[SOLVED] Handling multiple files using awk

Hi, I am trying to process 2 files simultaneously using awk satisfying following condition, Both files contain 3 columns. It should take entry from column 1 from first file, look for that entry in file 2 and if found, add column 2 and column 3 from both files and output to third file. For e.g.... (4 Replies)
Discussion started by: muazfarooqaslam
4 Replies

6. Shell Programming and Scripting

UNIX file handling issue

I have a huge file semicolon( ; ) separated records are Pipe(|) delimited. e.g abc;def;ghi|jkl;mno;pqr|123;456;789 I need to replace the 50th field(semicolon separated) of each record with 9006. The 50th field can have no value e.g. ;; Can someone help me with the appropriate command. (3 Replies)
Discussion started by: Gurkamal83
3 Replies

7. UNIX for Dummies Questions & Answers

Large file data handling issue

I have a single record large file, semicolon ';' and pipe '|' separated. I am doing a vi on the file. It is throwing an error "File to long" I need to actually remove the last | symbol from this file. sed -e 's/\|*$//' filename is working fine for small files. But not working on this big... (13 Replies)
Discussion started by: Gurkamal83
13 Replies

8. UNIX for Dummies Questions & Answers

File handling issue

Hi All, I am running into an issue. I have a very big file. Wants to split it in smaller chunks. This file has multiple header/ trailers. Also, between each header/trailer there are records. Number of records in each header trailer combination can vary. Also, headers can start with... (3 Replies)
Discussion started by: Gurkamal83
3 Replies

9. Shell Programming and Scripting

awk script issue redirecting to multiple files after matching pattern

Hi All I am having one awk and sed requirement for the below problem. I tried multiple options in my sed or awk and right output is not coming out. Problem Description ############################################################### I am having a big file say file having repeated... (4 Replies)
Discussion started by: kshitij
4 Replies

10. UNIX for Beginners Questions & Answers

Issue with search and replacing multiple items in multiple files

Im having an issue when trying to replace the first column with a new set of values in multiple files. The results from the following code only replaces the files with the last set of values in val.txt. I want to replace all the files with all the values. for date in {1..31} do for val in... (1 Reply)
Discussion started by: ncwxpanther
1 Replies

LEARN ABOUT FREEBSD

info

INFO(1) 							   User Commands							   INFO(1)

NAME

       info - read Info documents

SYNOPSIS

       info [OPTION]... [MENU-ITEM...]

DESCRIPTION

       Read documentation in Info format.

OPTIONS

       --apropos=STRING
	      look up STRING in all indices of all manuals.

       -d, --directory=DIR
	      add DIR to INFOPATH.

       --dribble=FILENAME
	      remember user keystrokes in FILENAME.

       -f, --file=FILENAME
	      specify Info file to visit.

       -h, --help
	      display this help and exit.

       --index-search=STRING
	      go to node pointed by index entry STRING.

       -n, --node=NODENAME
	      specify nodes in first visited Info file.

       -o, --output=FILENAME
	      output selected nodes to FILENAME.

       -R, --raw-escapes
	      output "raw" ANSI escapes (default).

       --no-raw-escapes
	      output escapes as literal text.

       --restore=FILENAME
	      read initial keystrokes from FILENAME.

       -O, --show-options, --usage
	      go to command-line options node.

       --subnodes
	      recursively output menu items.

       -w, --where, --location
	      print physical location of Info file.

       --vi-keys
	      use vi-like and less-like key bindings.

       --version
	      display version information and exit.

       The first non-option argument, if present, is the menu entry to start from; it is searched for in all `dir' files along INFOPATH.  If it is
       not present, info merges all `dir' files and shows the result.  Any remaining arguments are treated as the names of menu items relative	to
       the initial node visited.

EXAMPLES

       info   show top-level dir menu

       info emacs
	      start at emacs node from top-level dir

       info emacs buffers
	      start at buffers node within emacs manual

       info --show-options emacs
	      start at node with emacs' command line options

       info -f ./foo.info
	      show file ./foo.info, not searching dir

REPORTING BUGS

       Email   bug   reports   to   bug-texinfo@gnu.org,   general   questions	 and  discussion  to  help-texinfo@gnu.org.   Texinfo  home  page:
       http://www.gnu.org/software/texinfo/

COPYRIGHT

       Copyright (C) 2004 Free Software Foundation, Inc.  There is NO warranty.  You may redistribute this software under the  terms  of  the  GNU
       General Public License.	For more information about these matters, see the files named COPYING.

info 4.8							   December 2004							   INFO(1)
All times are GMT -4. The time now is 11:39 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy