Running rename command on large files and make it faster


Login or Register for Dates, Times and to Reply

 
Thread Tools Search this Thread
# 1  
Running rename command on large files and make it faster

Hi All,

I have some 80,000 files in a directory which I need to rename. Below is the command which I am currently running and it seems, it is taking fore ever to run this command. This command seems too slow. Is there any way to speed up the command. I have have GNU Parallel installed on my system but I am not sure how to run this command using GNU Parallel.

What this script is doing is to rename all .xml.dat files to .xml

Code:
for f in *.xml.dat; do f2=`basename $f .dat`; echo "$f -> $f2"; mv $f $f2; done

I am using Linux with BASH.
# 2  
Look for "rename" command on your system and its man page.
This User Gave Thanks to yazu For This Post:
# 3  
yes, first I tried with this:
Code:
rename .xml.dat .xml *.xml.dat

but I got the message Argument list too long.

Then I came up with this command of renaming. Though it is working but the only problem is that its too slow.
# 4  
Try:
Code:
find DIR -name '*.xml.dat' | xargs rename '.xml.dat$' .xml

This User Gave Thanks to yazu For This Post:
# 5  
Code:
find DIR -name '*.xml.dat' | xargs rename '.xml.dat$' .xml

The command is not working; neither it is giving any messages (file name extensions are still the same), but I am trying my best to modify it and make it work. I replaced the DIR with the working directory (just to clear any doubts!) Smilie

If I could make it work, I'll post my command here.

---------- Post updated at 04:18 PM ---------- Previous update was at 04:06 PM ----------

Great. it worked just tweaked your command:

Code:
find . -name '*.xml.dat' | xargs rename '.xml.dat' .xml

Thanks for your help. Smilie
# 6  
This is probably what you are looking for:
Code:
parallel -j0 mv {} {.} ::: *.xml.dat
parallel -j0 echo {} -\\\> {.}\;mv {} {.} ::: *.xml.dat

This User Gave Thanks to tange For This Post:
# 7  
Quote:
Originally Posted by shoaibjameel123
Hi All,

I have some 80,000 files in a directory which I need to rename. Below is the command which I am currently running and it seems, it is taking fore ever to run this command.
If you can install mmv, you can do mmv '*.xml' '#1.dat' which will be faster than the version with parallel, because it doesn't need to run 80,000 separate instances of mv.

I don't think renaming 80,000 files is ever going to be fast, though. That's 80,000 file tree operations.
This User Gave Thanks to Corona688 For This Post:
Login or Register for Dates, Times and to Reply

Previous Thread | Next Thread
Thread Tools Search this Thread
Search this Thread:
Advanced Search

Test Your Knowledge in Computers #361
Difficulty: Medium
The rmdir command remove directories, even if they are not empty.
True or False?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to make awk command faster for large amount of data?

I have nginx web server logs with all requests that were made and I'm filtering them by date and time. Each line has the following structure: 127.0.0.1 - xyz.com GET 123.ts HTTP/1.1 (200) 0.000 s 3182 CoreMedia/1.0.0.15F79 (iPhone; U; CPU OS 11_4 like Mac OS X; pt_br) These text files are... (21 Replies)
Discussion started by: brenoasrm
21 Replies

2. Shell Programming and Scripting

How to make awk command faster?

I have the below command which is referring a large file and it is taking 3 hours to run. Can something be done to make this command faster. awk -F ',' '{OFS=","}{ if ($13 == "9999") print $1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12 }' ${NLAP_TEMP}/hist1.out|sort -T ${NLAP_TEMP} |uniq>... (13 Replies)
Discussion started by: Peu Mukherjee
13 Replies

3. Shell Programming and Scripting

awk changes to make it faster

I have script like below, who is picking number from one file and and searching in another file, and printing output. Bu is is very slow to be run on huge file.can we modify it with awk #! /bin/ksh while read line1 do echo "$line1" a=`echo $line1` if then echo "$num" cat file1|nawk... (6 Replies)
Discussion started by: mirwasim
6 Replies

4. Shell Programming and Scripting

Faster command to remove headers for files in a directory

Good evening Im new at unix shell scripting and im planning to script a shell that removes headers for about 120 files in a directory and each file contains about 200000 lines in average. i know i will loop files to process each one and ive found in this great forum different solutions... (5 Replies)
Discussion started by: alexcol
5 Replies

5. UNIX for Dummies Questions & Answers

Rename a large number of files in subdirectories

Hi, I have a large number of subdirectories (>200), and in each of these directories there is a file with a name like "opp1234.dat". I'd like to know how I could change the names of these files to say "out.dat" in all these subdirectories in one go. Thanks! (5 Replies)
Discussion started by: lost.identity
5 Replies

6. Shell Programming and Scripting

Make script faster

Hi all, In bash scripting, I use to read files: cat $file | while read line; do ... doneHowever, it's a very slow way to read file line by line. E.g. In a file that has 3 columns, and less than 400 rows, like this: I run next script: cat $line | while read line; do ## Reads each... (10 Replies)
Discussion started by: AlbertGM
10 Replies

7. Emergency UNIX and Linux Support

Help to make awk script more efficient for large files

Hello, Error awk: Internal software error in the tostring function on TS1101?05044400?.0085498227?0?.0011041461?.0034752266?.00397045?0?0?0?0?0?0?11/02/10?09/23/10???10?no??0??no?sct_det3_10_20110516_143936.txt What it is It is a unix shell script that contains an awk program as well as... (4 Replies)
Discussion started by: script_op2a
4 Replies

8. Red Hat

Re:How to make the linux pc faster

Hi, Can any one help me out in solving the problem i have a linux database server it is tooo slow that i am unable to open even the terminial is there any solution to get rid of this problem.How to make this server faster. Thanks & Regards Venky (0 Replies)
Discussion started by: venky_vemuri
0 Replies

9. Shell Programming and Scripting

Can anyone make this script run faster?

One of our servers runs Solaris 8 and does not have "ls -lh" as a valid command. I wrote the following script to make the ls output easier to read and emulate "ls -lh" functionality. The script works, but it is slow when executed on a directory that contains a large number of files. Can anyone make... (10 Replies)
Discussion started by: shew01
10 Replies

10. UNIX for Advanced & Expert Users

Error when running the make command

Hi, Not really sure whether this question should go to this forum but am giving it a shot. I have compiled a simple C program test.c. #include <stdio.h> #include <stdlib.h> #include <string.h> static int a; int test() { a=a+1; return a; } When I run a make command, I get this: ... (2 Replies)
Discussion started by: nattynatty
2 Replies

Featured Tech Videos