Shell Programming and Scripting

I am trying to setup my plex server to use symlinks rather than host the movie files.
in my storage directory, i have both movies(some in subdirectory of the name and some just in the parent directory) and tvshows, which have subdirectories for each season, which contains the episodes

i would like to create a one-time script to parse the storage directory, and then create a symlink on the plex share directory, a directory for movies and one for tvshows.

I have figured out that using ls -d /dl/**/* to display the full working directory, but I cant figure out how to parse that data out to pass it to a script to create the symlink. Also I am getting errors when I attempt to manipulate, so I assume i will have to do a while read loop of some sort.

Code:

[freenas] /dl# ls -d **/* | egrep ".avi|.mkv"
/bin/ls: Argument list too long.
[freenas] /dl# ls -d **/* | egrep ".avi|.mkv" > /movies.txt
/bin/ls: Argument list too long.
[freenas] /dl# ls -d **/*  > /movies.txt
/bin/ls: Argument list too long.

I would expect that I have 1-2 thousand files to create symlinks with.

Could anyone point me in a direction of how I should do this?

Scripting is not one of my strengths, i can usually pick apart a simple script to see what it is doing, but i cant seem to just create one from scratch.

Thanks
Band

Moderator's Comments:

edit by bakunin: please use CODE-tags for code, terminal output and data, as required by the rules and like i have edited them into your text. Thank you.

First the explanation why your command failed: wildcards ('*', '?', etc.) are expanded not by the commands you write but by the shell before it calls the commands. Suppose you have 3 files in a directory, fileA, fileB and fileC. If you issue a command like:

Code:

/some/command *

The shell will first expand "*" to the names of all files, replace the asterisk with their names and then execute the resulting command line:

Code:

/some/command *                       # your commandline
/some/command fileA fileB fileC       # after expanding the asterisk

Now the commandline only allows for so many parameters to be passed to a command and if there are thousands or even millions of files the list might become quite huge - to big for the commandline. This has happened here and is the reason for ls to complain.

When you want to process such a lot of files you can use one of the following methods, both of which avoid the problem with the expanding wildcards:

First, you can use the fact that ls per default lists all filenames if no parameter is passed. That means ls * and ls do the same thing. Pass the output to a loop in which you work on your files:

Code:

ls | while read FILE ; do
     if [ -f "$FILE ] ; then     # only process files (i.e. not subdirs)
          command1 "$FILE"
          command2 "$FILE"
     fi
     [....]
done

This works fine, but only if you want to process every file in a certain directory and you do not want to recurse through many subdirectories. If you want to process files in all subdirectories from some path onwards this method falls short.

For such cases there is the find command. Use it to produce a list of directory entries based on many different attributes (file/dir/link/..., certain ranges of modification dates, ownerships, etc. - see the man page of find for details) and use its -exec clause to process each directory entry found that way.

For instance, the following moves every file with a name starting with "A" to a certain directory. The starting directory is searched recursively so that all its subdirectories are included in the search:

Code:

find /some/start/dir -type f -name "A*" -exec mv {} /other/dir \;

Instead of a simple command like mv you could also provide a script written by yourself to do more complex manipulations. The "{}" will be expanded to a single filename found by find and the script be called with that parameter once for every entry found that way.

I hope this helps.

bakunin

Thank you for correcting the code tags...

in your example, in line 2

Code:

ls | while read FILE ; do
     if [ -f "$FILE" ] ; then     # only process files (i.e. not subdirs)
          command1 "$FILE"
          command2 "$FILE"
     fi
     [....]
done

Shouldnt there be a command preceding it? maybe

Code:

ls

?

---------- Post updated at 07:27 PM ---------- Previous update was at 07:12 PM ----------

so playing around with this, i found that freebsd doesnt like -f

Code:

ls | while read FILE ;do
        if [ test -f "$FILE"]; then
#               ln -s /dl/"$FILE" /Movies/"$FILE"
                ln -s /test1/"$FILE" /dest1/"$FILE"
        fi
done

with the above command, i get the following output

Code:

[root@freenas] /test1# bash ~/basic_ln.sh
/root/basic_ln.sh: line 2: [: missing `]'

Not sure why it thinks it is missing a backtick

It looks like bakunin was missing a double quote (shown in red on the 2nd line in the script below), but a space between the test argument and the closing square bracket is crucial. Try:

Code:

ls | while read FILE ; do
     if [ -f "$FILE" ] ; then     # only process files (i.e. not subduers)
          command1 "$FILE"
          command2 "$FILE"
     fi
     [....]
done