Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Multiple occurance of file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Multiple occurance of file
# 1  
Old 10-27-2012
Multiple occurance of file
Hi all,
I have file structure as file.log
Code:
84t-rw-r--r-- 1 emily04 us_cms 24492717 Oct  5 13:29 vgtree_84_1_K3L.root
85t-rw-r--r-- 1 emily04 us_cms 50410380 Oct  5 16:06 vgtree_85_1_uZv.root
85t-rw-r--r-- 1 emily04 us_cms 50567380 Oct  5 16:06 vgtree_85_1_hjv.root
86t-rw-r--r-- 1 emily04 us_cms 29777963 Oct  5 13:28 vgtree_86_1_YzK.root
87t-rw-r--r-- 1 emily04 us_cms 25127236 Oct  5 12:37 vgtree_87_1_8Fq.root
87t-rw-r--r-- 1 emily04 us_cms 98712736 Oct  5 12:37 vgtree_87_1_ukj.root
87t-rw-r--r-- 1 emily04 us_cms 25987236 Oct  5 12:37 vgtree_87_1_okh.root
88t-rw-r--r-- 1 emily04 us_cms 47853471 Oct  5 13:46 vgtree_88_1_EGK.root

For me I would like to have the output of the duplicates, in the given file. The duplicates are
Code:
85t-rw-r--r-- 1 emily04 us_cms 50410380 Oct  5 16:06 vgtree_85_1_uZv.root
85t-rw-r--r-- 1 emily04 us_cms 50567380 Oct  5 16:06 vgtree_85_1_hjv.root
87t-rw-r--r-- 1 emily04 us_cms 25127236 Oct  5 12:37 vgtree_87_1_8Fq.root
87t-rw-r--r-- 1 emily04 us_cms 98712736 Oct  5 12:37 vgtree_87_1_ukj.root
87t-rw-r--r-- 1 emily04 us_cms 25987236 Oct  5 12:37 vgtree_87_1_okh.root


Thanks
emily
# 2  
Old 10-27-2012
Try:
Code:
grep -o "^.." file | sort | uniq -c | awk '$1>1{print $2}' | xargs -i grep "^{}" file

# 3  
Old 10-27-2012
Code:
sort -n file |awk -F"t" '{id=$1;if(id == pid){print prev;print $0;}prev=$0;pid=$1;}'  |uniq

# 4  
Old 10-27-2012
I do not get what 'duplicates' means? Are there files whose content is identical to another file with a different name? It does not look like it.

How are you manually figuring out duplicate? The first three digits of the log file seems to be what you are using.

This produces a list (in a file ) without duplicates
Code:
awk ' ! arr[substr($0,1,3)]++' file.log > newfile.log

# 5  
Old 10-27-2012
Thanks all..;-)
It worked..

emily,

---------- Post updated at 11:37 AM ---------- Previous update was at 10:21 AM ----------

Hi,
Can somebody help me?
I am testing a demo with the given function
Code:
PATH525[1]="/uscms/home/emily/READme/extra/data/"
TEMP=temp
FileName=DataFileName

CopyFiles() {
#    PATHNAME="$paths"                                                                                                                                

#    if [ "$2" = "525" ]; then                                                                                                                        
#       PATHNAME="$PATH525[*]"                                                                                                                        
#   elif [ "$2" = "533" ]; then                                                                                                                       
#         PATHNAME="$PATH533[*]"                                                                                                                      
#    elif [ "$2" = "ZY" ]; then                                                                                                                       
#       PATHNAME="$PATHZY[*]"                                                                                                                         
#   fi                                                                                                                                                
#  echo "pathname is $PATHNAME"                                                                                                                       

    echo 'Being Called'
    for FileNameIndx in "${PATH525[@]}"
      do
      if [[ ! -e "dest_path/$FileNameIndx" ]]; then
          ls -ltr "$FileNameIndx" | grep root | awk '{print string path $9}' string="$CONSTANT" path="$FileNameIndx"  >> "$File0"
          echo 'file0 :'$File0
          sort -nrk5 < $File0 | awk -F_ '!x[$3]++' > $FileName
          echo 'FileName :' $FileName
          echo "$FileNameIndx is copied"
      else
          echo "Check the FileName in ${PATHNAME[@]}"
      fi
      echo "---------------------------------------------------------"
      echo ">>> DataFiles are from :" ${PATH533[@]}
      echo "---------------------------------------------------------"
    done
}

Now, the "PATH525" has following content
Code:
 
-rw-r--r-- 1 emily us_cms  9 Oct 27 10:28 vgtee_1_ujh.root
-rw-r--r-- 1 emily us_cms 100 Oct 27 10:28 vgtee_1_ujf.root
-rw-r--r-- 1 emily us_cms 12 Oct 27 10:28 vgtee_2_ujf.root
-rw-r--r-- 1 emily us_cms 10 Oct 27 10:28 vgtee_3_ujf.root
-rw-r--r-- 1 emily us_cms  6 Oct 27 10:28 vgtee_3_ujh.root
-rw-r--r-- 1 emily us_cms  7 Oct 27 10:28 vgtee_4_ujh.root
-rw-r--r-- 1 emily us_cms  9 Oct 27 10:28 vgtee_5_ujh.root

And I was expecting the final result should be all files except the duplicates.
But is it not working SmilieSmilie

I get the content of the "DataFileName" as :
Code:
 /uscms/home/emily/READme/extra/data/vgtee_5_ujh.root
/uscms/home/emily/READme/extra/data/vgtee_3_ujf.root

Whereas I WANT IS FOLLOWING
Code:
 
/uscms/home/emily/READme/extra/data/vgtee_1_ujf.root
/uscms/home/emily/READme/extra/data/vgtee_2_ujf.root
/uscms/home/emily/READme/extra/data/vgtee_3_ujf.root
/uscms/home/emily/READme/extra/data/vgtee_4_ujh.root
/uscms/home/emily/READme/extra/data/vgtee_5_ujh.root

Also please take notice of the fact that in case of DUPLICATION of file, I would LIKE TO HAVE BIGGER SIZE FILE TO BE IN DATAFILENAME file
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Help search and replace the last occurance of match in a file

Hi I want to replace only the last occurance of "union all" in input file with ";" I tried with sed 's/union all/;/g' in my input file, it replaced in all lines of input file Eg: select column1,column2 from test1 group by 2 union all select column1,column2 from test2 group by 2 union all ... (9 Replies)
Discussion started by: antosr7
9 Replies

2. Shell Programming and Scripting

Extract multiple occurance of strings between 2 patterns

I need to extract multiple occurance strings between 2 different patterns in given line. For e.g. in below as input ------------------------------------------------------------------------------------- mike(hussey) AND mike(donald) AND mike(ryan) AND mike(johnson)... (8 Replies)
Discussion started by: sameermohite
8 Replies

3. Shell Programming and Scripting

Number of occurance with multiple conditions??

Hi, I the following sample of out put: + 6.07875 10 0 cbr 210 ------- 2 10.0 2.3 1461 19715 - 6.07875 10 0 cbr 210 ------- 2 10.0 2.3 1461 19715 + 6.07875 22 0 cbr 210 ------- 2 22.0 2.9 1301 19716 - 6.07875 22 0 cbr 210 ------- 2 22.0 2.9 1301 19716 r 6.07922 0 1 cbr 210 ------- 1 30.0... (6 Replies)
Discussion started by: ENG_MOHD
6 Replies

4. Shell Programming and Scripting

Remove the last occurance in a file

Hi, I have hunted for an answer to this but still can't come up with a working solution. How do I delete the last of a non unique line The lines I am looking for is case "$0" in esac This can appear n times I only want the last deleted. I have tried with sed and... (1 Reply)
Discussion started by: miyoung999
1 Replies

5. Shell Programming and Scripting

Count occurance of multiple strings using grep command

How to grep multiple string occurance in input file using single grep command? I have below input file with many IDP, RRBE messages. Out put should have count of each messages. I have used below command but it is not working grep -cH "(sent IDP Request)(Recv RRBCSM)" *.txt ... (5 Replies)
Discussion started by: sushmab82
5 Replies

6. UNIX for Dummies Questions & Answers

i need 100th occurance of a letter in file

Hi to all, I am looking a file in vi editor to get 100th occurance of a latter in that file. Can any one help me in this? Thanks Sathish (1 Reply)
Discussion started by: bsathishmca
1 Replies

7. Shell Programming and Scripting

How to insert values in 1st occurance out of two occurance in a file

Hi I have a file which contains the following two lines which are same But I would like to insert the value=8.8.8.8 in the 1st occurance line and value=9.9.9.9 in the 2nd occurance line. <parameter name="TestIp1" value=""> <parameter name="TestIp1" value=""> Please suggest (1 Reply)
Discussion started by: madhusmita
1 Replies

8. Shell Programming and Scripting

How to replace specific text line out of multiple occurance

Hi I would like to replace specific line eg ExitAction = NONE to ExitAction = FALSE under only TASK sipsiproc and other ExitAction = NONE will remain as usual in the file(shell script) The file contains: TASK rgcdproc { CommandLine = $SSHOME/bin/rgcd.exe NewConsole... (5 Replies)
Discussion started by: madhusmita
5 Replies

9. UNIX for Dummies Questions & Answers

replace the n'th occurance in a file

Hi All, How can i replace the n'th occurance in a file. ? I have a property file like EAR;_TrackingEAR;META-INF/application.xml;xml;context-root;1;valeur EAR;_TrackingEAR;META-INF/application.xml;xml;context-root;2;valeur2... (2 Replies)
Discussion started by: subin_bala
2 Replies

10. UNIX for Dummies Questions & Answers

count string occurance in a file hourly

Hi, I file that has all the status for one day (24hours). Now what I want to do is to count the occurence of a string in its output hourly like for example count occurance of successful or asynchronous clear destinon for every hour and redirect it to file. Please see sample file below. Please... (2 Replies)
Discussion started by: ayhanne
2 Replies
Login or Register to Ask a Question