I am a new ubuntu user (have to use it for work) and I am trying to learn and familiarize myself with commands that I will be using frequently.
I would like some help in how I can get a list of all files with certain keywords in the filename.
For example, I have a directory with numerous subdirectories that have a bunch of files in them. Two of those files contain the following format in the name: numericals_eddy_corrected.nii.gz and numericals_eddy_corrected_brain_mask.nii.gz.
I want to get a list of all the eddy_corrected.nii.gz and eddy_corrected_brain_mask.nii.gz files.
I am wondering, what command do I type in terminal so that I get a list, including the path, of all the files that have eddy_corrected in the name without the other eddy_corrected_brain_mask' showing up as well and vice versa? Is there a way to have terminal make a .txt file of the list?
Second thing I need help with is renaming some files in those subdirectories. These files end in the extension .bvec and .bval and have absurdly long names with numbers in the format of subjectnumberDTISiementsTClessnumericals.bvec/.bval or 1000047785DTISiemensTCless005.bvec etc.
How can I rename all the files in the subdirectories to something simpler or shorter like subjectnumber_DTI.bvec?
I have heard of find and grep but I'm not sure how to go about combining commands to achieve what I need to do.
I will be required to do similar tasks in the future so I figured I should learn it once and for all. I would appreciate any help.
If I have posted this in the wrong forum, please feel free to move it to the correct location.
Thank you in advance.
Moderator's Comments:
Please use ICODE tags, not B tags for partial line sample input, output, and code segments.
Last edited by Don Cragun; 04-02-2016 at 02:33 AM..
I am a new ubuntu user (have to use it for work) and I am trying to learn and familiarize myself with commands that I will be using frequently.
I would like some help in how I can get a list of all files with certain keywords in the filename.
Welcome to the UNIX & Linux Forums. We are here to help you.
Quote:
For example, I have a directory with numerous subdirectories that have a bunch of files in them. Two of those files contain the following format in the name: numericals_eddy_corrected.nii.gz and numericals_eddy_corrected_brain_mask.nii.gz.
I want to get a list of all the eddy_corrected.nii.gz and eddy_corrected_brain_mask.nii.gz files.
Please search the forum for similar questions that have already been answered before posting a new question. Without knowing what the format of other filenames in your directories, you could use a find command searching exactly for filenames ending with those two strings in the file hierarchy rooted in the current working directory:
If there aren't any other names containing eddy_corrected that end with nii.gz you could use the simpler:
Quote:
I am wondering, what command do I type in terminal so that I get a list, including the path, of all the files that have eddy_corrected in the name without the other eddy_corrected_brain_mask' showing up as well and vice versa? Is there a way to have terminal make a .txt file of the list?
The terminal application running in your window manager emulates a terminal from days of yore. The terminal application just emulates hardware, it doesn't run commands, it doesn't create file, it doesn't do anything except emulate a terminal device. When you start (or your system starts) the terminal application, you specify a program to run in that terminal window. Usually the program that you run in a terminal window will be a command interpreter such as bash, csh, ksh, or zsh. These command interpreters are known as shells. The commands recognized by different shells varies. All of the examples I'm suggesting in this post will work with bash, ksh, and zsh; some, but not all, will work in csh.
Adding > filename to the end of either of the above commands will redirect the output from those commands to the file named filename (creating the file if it does not exist, and overwriting its previous contents if it did exist). Using >> filename will append the output of those commands will create the file if it did not exist and append the output to the end of the file if it did exist.
Quote:
Second thing I need help with is renaming some files in those subdirectories. These files end in the extension .bvec and .bval and have absurdly long names with numbers in the format of subjectnumberDTISiementsTClessnumericals.bvec/.bval or 1000047785DTISiemensTCless005.bvec etc.
How can I rename all the files in the subdirectories to something simpler or shorter like subjectnumber_DTI.bvec?
You could try something like:
(this assumes that you want to preserve the .bvec or .bval at the ends of the pathnames) and if the output looks like it correctly supplies the proper arguments to rename files the way you want them to be, remove the echo shown in red and rerun the command to actually rename the files. If you want to the new names of all of the files to have the extension .bvec, change the two lines:
to be just:
Quote:
I have heard of find and grep but I'm not sure how to go about combining commands to achieve what I need to do.
I will be required to do similar tasks in the future so I figured I should learn it once and for all. I would appreciate any help.
If I have posted this in the wrong forum, please feel free to move it to the correct location.
This is a perfectly reasonable forum for this topic.
Quote:
Thank you in advance.
Moderator's Comments:
Please use ICODE tags, not B tags for partial line sample input, output, and code segments.
Without knowing what the format of other filenames in your directories, you could use a find command searching exactly for filenames ending with those two strings in the file hierarchy rooted in the current working directory:
If there aren't any other names containing eddy_corrected that end with nii.gz you could use the simpler:
Could you explain the syntax of the find code you have provided? I had searched google and used the following which is a bit different from what you have and I am wondering if you could explain the difference so I can understand it better? find . -name '*eddy_corrected_brain.nii.gz*'
Quote:
The terminal application running in your window manager emulates a terminal from days of yore. The terminal application just emulates hardware, it doesn't run commands, it doesn't create file, it doesn't do anything except emulate a terminal device. When you start (or your system starts) the terminal application, you specify a program to run in that terminal window. Usually the program that you run in a terminal window will be a command interpreter such as bash, csh, ksh, or zsh. These command interpreters are known as shells. The commands recognized by different shells varies. All of the examples I'm suggesting in this post will work with bash, ksh, and zsh; some, but not all, will work in csh.
Ahh I see now. Sorry I'm still figuring out Linux and how things are referred to in this environment. I think the work computer uses bash.
Quote:
Adding > filename to the end of either of the above commands will redirect the output from those commands to the file named filename (creating the file if it does not exist, and overwriting its previous contents if it did exist). Using >> filename will append the output of those commands will create the file if it did not exist and append the output to the end of the file if it did exist.
Okay so to make sure I understand correctly, if I use find . -name '*eddy_corrected*nii.gz' > filename01 it will create a file with all of the results that show up? How do I make it so that the file is a .txt file? Do I just add > filename.txt?
Quote:
You could try something like:
(this assumes that you want to preserve the .bvec or .bval at the ends of the pathnames) and if the output looks like it correctly supplies the proper arguments to rename files the way you want them to be, remove the echo shown in red and rerun the command to actually rename the files. If you want to the new names of all of the files to have the extension .bvec, change the two lines:
to be just:
This is a perfectly reasonable forum for this topic.
Again, if it's not too much trouble, could you explain the lines of code so I can understand what those words/commands are doing? It's just that at first glance, it all seems a bit daunting so perhaps if I know the purpose of each line/word/character - I can get a better idea of how the command works.
If I just want to find the .bvec and .bval files separately (run the command twice) do I change the code as such?
Thank you again!
Could you explain the syntax of the find code you have provided? I had searched google and used the following which is a bit different from what you have and I am wondering if you could explain the difference so I can understand it better? find . -name '*eddy_corrected_brain.nii.gz*'
Ahh I see now. Sorry I'm still figuring out Linux and how things are referred to in this environment. I think the work computer uses bash.
Okay so to make sure I understand correctly, if I use find . -name '*eddy_corrected*nii.gz' > filename01 it will create a file with all of the results that show up? How do I make it so that the file is a .txt file? Do I just add > filename.txt?
Again, if it's not too much trouble, could you explain the lines of code so I can understand what those words/commands are doing? It's just that at first glance, it all seems a bit daunting so perhaps if I know the purpose of each line/word/character - I can get a better idea of how the command works.
If I just want to find the .bvec and .bval files separately (run the command twice) do I change the code as such?
Thank you again!
Hi azurite,
First, a suggestion: Don't be afraid to try things and watch what happens!
And, another suggestion: Instead of searching the web to try to figure out what a utility on your system will do, read the manual page for that utility on your system. For instance, the command:
will show you the manual page for the find utility on your system.
You still haven't told us what shell you're using. The manual page for your shell will describe pipelines, the while loop construct, the read built-in utility, variable assignments, parameter expansions, and file redirections. And, the mv manual page will describe how it can be used to rename files.
Unlike Windows, UNIX systems determine the type of a file by its contents; not by its name.
The command: find . -name '*eddy_corrected_brain.nii.gz*' searches the file hierarchy rooted in the current directory (.) for files with names that contain the string eddy_corrected_brain.nii.gz proceeded and followed by strings of 0 or more characters (*) and writes the pathname of each matching filename on a line by itself. Note that I did NOT include an asterisk at the end of the pattern I used in the find commands I suggested because I thought you only wanted filenames ending in .gz; not filenames containing .gz followed by other arbitrary text.
Since pathnames cannot contain NUL bytes, this output will be a text file unless the pathnames produced are longer than the number of bytes your system allows in a line in a text file. The number of types allowed on all UNIX and Linux systems is at least 2048. The actual number on your system can be found using the command:
With no file redirection (as above) the output is displayed on your terminal. If you redirect the output as in any of the following:
the output file will be a text file. By convention, storing text in a file with the filename extension .pdf is a VERY bad idea, but the name you choose for your output does not in any way affect the output placed in that file by the find utility.
If you want to see what a shell script is doing, turn on tracing and watch what it does while it is running. For example to add tracing to the script:
The set -xv at the start of the script turns tracing on and the set +xv at the end of the script turns tracing off.
In the above pipeline, find prints the pathnames of all filenames containing the string DTISiemensTCless that end with the string .bvec or (-o) with the string .bval, the read command sets the variable path to the contents of one of those lines of output from find and the while loop contains parameter expansions and variable assignments that set dir to the name of the directory specified in path, file to the last component of path , subjectnumber to the part of that file's name before the string DTI, ext to the file's extension (not including the period), and the mv command renames the file named by that line of output from find to be a file in the same directory, with the same subject number that was in the original file followed by the string DTI followed by the filename extension that was on the original file.
This User Gave Thanks to Don Cragun For This Post:
Hi azurite,
First, a suggestion: Don't be afraid to try things and watch what happens!
And, another suggestion: Instead of searching the web to try to figure out what a utility on your system will do, read the manual page for that utility on your system. For instance, the command:
will show you the manual page for the find utility on your system.
You still haven't told us what shell you're using.
....
....
Hello,
Thank you for the explanations. I think I'm beginning to understand it a bit better now. I work on ubuntu on the work computer so I can't really experiment right now lest I mess something up. I'm in the process of trying to install ubuntu on my laptop via virtualbox so hopefully I can start experimenting soon.
I think the work computer uses bash but I will check soon. Thank you for the instructions on how to do that.
I have one more question, I found this bit of commands in a tutorial that was somewhat related to my work and I was wondering if you could tell me what the command does?
Again, thank you for your assistance!
Moderator's Comments:
Please use CODE tags (not ICODE tags) for full-line an multi-line sample input, output, and code segments.
Last edited by Don Cragun; 04-05-2016 at 12:33 AM..
Reason: Change ICODE tags to CODE tags.
Thank you for the explanations. I think I'm beginning to understand it a bit better now. I work on ubuntu on the work computer so I can't really experiment right now lest I mess something up. I'm in the process of trying to install ubuntu on my laptop via virtualbox so hopefully I can start experimenting soon.
I think the work computer uses bash but I will check soon. Thank you for the instructions on how to do that.
... ... ...
I have one more question, I found this bit of commands in a tutorial that was somewhat related to my work and I was wondering if you could tell me what the command does?
Again, thank you for your assistance!
Moderator's Comments:
Please use CODE tags (not ICODE tags) for full-line an multi-line sample input, output, and code segments.
Hi,
You're welcome. First, note that the commands:
will make ABSOLUTELY NO CHANGES to any files on your system unless you remove the echo. Let me expand a little on what I said earlier: There is NO WAY to learn how to run shell commands and write shell scripts other than to run shell commands and write and run shell scripts. Run the above commands, go through the trace output line by line (referring to the bash man page) if you don't understand how the various parameter expansions are working when it is extracting the directory (dir), filename (file), the subject number (subjectnumber), and the filename extension (ext) from the pathnames (path) found by the find command, and look at the mv command that is printed by the echo command.
And, for the commands:
... I repeat: You have to run shell commands to learn what shell commands do!
And, if you try running those commands you are very likely to get a diagnostic message saying that there is no command named [bash]mv unless there is a utility named amv, bmv, hmv, smv, or [bash]mv on your system. Once you practice running commands and learn how pathname expansions work, you'll understand why I gave that list of utility names AND you'll recognize that [bash] and [/bash] are tags used in the book you were looking at indicating that stuff between those tags is a command to be given to the bash shell command language interpreter; not part of the text of the commands themselves.
Note however, that running a mv command will actually move files. So do this in a directory with test files you have set up with matching names; do not try this in a directory where you have files with names ending in .bvec or .bval and directories named bvecs and bvals unless you actually want to move all of the files with names ending in .bval to the directory named bvals and want to move all of the files with names ending in .bvec to the directory named bvecs.
If those directories don't exist, what happens will vary depending on how many files match those filename patterns. Why don't you look at the mv man page and tell me what will happen in each of the following cases when you run the command:
and:
no file matches the pattern *.bvec and there is a directory named bvecs,
no file matches the pattern *.bvec and there is a regular file named bvecs,
no file matches the pattern *.bvec and there is no file named bvecs,
one file matches the pattern *.bvec and there is a directory named bvecs,
one file matches the pattern *.bvec and there is a regular file named bvecs,
one file matches the pattern *.bvec and there is no file named bvecs,
two or more files match the pattern *.bvec and there is a directory named bvecs,
two or more files match the pattern *.bvec and there is a regular file named bvecs, and if
two or more files match the pattern *.bvec and there is no file named bvecs.
Wow! Thank you for taking the time to write great explanations! I have printed it out and will re-read everything to really understand it.
Instead of creating new threads, is it okay if I keep posting new questions in this thread?
If so, I have a question on creating variables. I have tried to research and come up with a solution (to the text quoted below) on my own but I do not know if I am on the right track. If possible could you tell me if I am moving in the right direction?
Quote:
So as far as trying to distinguish between "100_eddy_corrected_brain.nii.gz" and "100_eddy_corrected_brain_mask.nii.gz" if you could define the first as a variable, say $eddy_file or something of the sort, you could pipe it through to a cut command defining a period as your delimiter and take everything in front of it, so something like:
{echo $eddy_file | cut -d. -f1}_mask.nii.gz
But that would depend on using the "set" command to create the variable.
I have come up with the following: though I am not sure what is meant by using the 'set' command.
The eddy_corrected_brain.txt is a text file with pathnames of all the files that match eddy_corrected_brain.nii.gz from the initial find command/results.
Hi,
In sftp script to get files, I have to rename all the files which I am picking. Rename command does not work here. Is there any way to do this?
I am using #!/bin/ksh
For eg: sftp user@host <<EOF
cd /path
get *.txt
rename *.txt *.txt.done
... (7 Replies)
Hello,
I am looking for a command line that will rename name files :
f700_abc_o_t_MASTERID_AS_AE_20130323.csv
like this
f700_abc_o_t_MASTERID_AS_AE_20130324.csv
The great idea could be to get the date stamp 20130323
and change any part of it, instead of just change the... (4 Replies)
Hi
i have to achieve the following
i have files as xyz001.csv, xyz002.csv.......xyz0025.csv in a folder, i need to keep xyz001.csv as it is
but want to remove the extra zero on filename from 10 say
xyz0010 should be renamed to xyz010
xyz0025 should be renamed as xyz025
Note xyz... (8 Replies)
Hi All
I have a folder that contains hundreds of file with a names
3.msa
4.msa
21.msa
6.msa
345.msa
456.msa
98.msa
...
...
...
I need rename each of this file by adding "core_" in the begiining of each file such as
core_3.msa
core_4.msa
core_21.msa (4 Replies)
Hello,
I wanted to rename one file where filename contains space.. How can i rename in unix?
The file name is ABC XYZ.TXT
I wanted to rename this file as ABCXYZ.TXT.
Any help is greatly appreciated...
Regards. (4 Replies)
Hi,
I have a very urgent requirement here. I have to find all files in the specified directory but not in the sub directories(The directory name is stored in a variable) which are older than the current date as well as current time and rename it as filename_yyyymmddhhmmss.ext and move it into a... (7 Replies)
I want to rename the files by taking part of the file and appending date to it. please help
e.g.
abc-390.csv
xyz-908.csv
desired format is abc_YYYYMMDD.csv
This is what I have but it is not working
for each in *.csv;
do
mv $each /abc/data/"`date '+test_%Y%M%M'`".csv
done (2 Replies)
Hello,
i wanna rename my files which names are written in movies.txt
films.txt = amovie
bmovie
cmovie
dmovie
emovie
and i wanna find this files and rename the files to 1_amovie
... (12 Replies)
Hi,
I have a challenging task,in which i have to find the duplicate files by its name and size,then i need to take anyone of the file.Then i need to open the file and find for more than one pattern and count of that pattern.
Note:These are the samples of two files,but i can have more... (2 Replies)