Sponsored Content
Top Forums Shell Programming and Scripting sort retrieved data files in different dirs Post 302511947 by leafei on Friday 8th of April 2011 03:15:52 AM
Old 04-08-2011
sort retrieved data files in different dirs

Hi,

I have a script to retrieve data files from server and store them in a directory on local disk. The script runs everyday as cron job. But now those files are too many so my boss wants me to put the files into different directories base on dates.

Those files look like this:

Code:
FCNCP104500104_SPA_2011-04-08_04-53-56-GMT_P08-00.nar

My original script:


Code:
#! /bin/bash

EMC_HOST='3.3.3.3'
EMC_USER='admin'
EMC_PASSWORD='password'
NAV_LOG_FILE='/var/log/naviseccli'
NAVISECCLI='/opt/Navisphere/bin/naviseccli'
NAV_BASE_CMD="${NAVISECCLI} -h ${EMC_HOST} -user ${EMC_USER} -password ${EMC_PASSWORD} -scope 0"
RETRIEVE_DIR='/mnt/emc_data'

#
# Simple logger.
#
log () {
    echo ${1} >> ${NAV_LOG_FILE}
}

#
# Retrieve file.
#
retrieve () {
    log "Retrieve file ${1}..."
    cmd="$NAV_BASE_CMD analyzer -archive -file ${1} -o"
    eval $cmd
    if [ $? -ne 0 ] ; then
        log "$(date): Retrieve $1 failed"
        log "command: $cmd"
        log "error code: $?"
        exit -1
    fi
}

#
# List all archives on EMC.
#
list () {
    file_list=$($NAV_BASE_CMD analyzer -archive -list | awk 'BEGIN {IGNORECASE=1;} /nar$/ { print $NF }')
    if [ $? -ne 0 ] ; then
        log "$(date): list archives failed"
        log "error code: $?"
        exit -2
    fi
}

cd $RETRIEVE_DIR || exit -1

list
for f in ${file_list} ; do
    [ -f ${f} ] || retrieve ${f}
done

Any suggestions to improve (rewrite) the script? Smilie
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

I need to ls all files in 4-6 deep dirs

I need to print to file , a listing of all files below a certain directory. Example: I need to print to file a listing of all files below the etc dir (including the subdirectories) with their full path. Any ideas on how to do this with one command. Or is this something I need to do on all... (4 Replies)
Discussion started by: gforty
4 Replies

2. Shell Programming and Scripting

script find files in two dirs HELP

I have a directory which is /home/mark/files/ , inside this particular I have a bunch of filles (see examples below) TST_SHU_00014460_20090302.txt TST_SHU_00016047_20090302.txt TST_SHU_00007838_20090303.txt TST_SHU_00056485_20090303.txt TST_SHU_00014460_20090303.txt... (2 Replies)
Discussion started by: fierusbentus
2 Replies

3. Shell Programming and Scripting

AWK help print dirs with files in it

Hi, I'm writing some start of day checks for my work. I want to check some dirs for files that have been created longer than 10 mins ago and not been transfered. I've already used a find command to write a list of files that meet this criteria to a log called sod.log i.e. ... (1 Reply)
Discussion started by: elcounto
1 Replies

4. Homework & Coursework Questions

Shell script calling Perl function, sort and find data, write to new files

Use and complete the template provided. The entire template must be completed. If you don't, your post may be deleted! 1. The problem statement, all variables and given/known data: I must write a shell script that calls two external Perl functions--one of which sorts the data in a file, and... (6 Replies)
Discussion started by: kowit010
6 Replies

5. UNIX for Dummies Questions & Answers

FTP SERVER logging of files retrieved

Is there any addon or finctionality in ftp (SFTP) Server that definitively indicates if a file was retireved and successfully? I am looking for this type of command with logging so -files retrieved -files deleted by remote users can be logged Must be compatible with AIX and UBUNTU. ... (0 Replies)
Discussion started by: cdc01
0 Replies

6. Shell Programming and Scripting

Creating a list of files retrieved from MGET

Is there a way to create a txt file with the names of the files I retreive using mget *.file. I want to use this file as input to a delete command. Using HP-UX (3 Replies)
Discussion started by: jagf
3 Replies

7. Debian

Problem with files/dirs deletion

Hi, The other day i installed a PHP based CMS (modx) on my shell account and noticed that i couldn't delete any of files/dirs it created after. Also, i noticed that all that stuff is owned by username-www instead of username. I tried chown, chmod and using a PHP script to do the same wti... (4 Replies)
Discussion started by: pentago
4 Replies

8. Shell Programming and Scripting

Moving files into dirs corresponding to dates

I am trying to find a way to move files into corresponding date files. i=0 while read line do array="$line" (( i++ )) done < <(ls) cd $(echo ${array}) echo ${array}} pwd #cd "$(array}" ] || mkdir 2015 cd "2015" ] || mkdir 02-February ] || mkdir 03-March ] || mkdir... (10 Replies)
Discussion started by: newbie2010
10 Replies

9. Shell Programming and Scripting

Merge and Sort tabular data from different text files

I have 42 text files; each containing up to 34 lines with following structure; file1 H-01 23 H-03 5 H-05 9 H-02 14 . . file2 H-01 17 H-02 43 H-04 7 H-05 8 H-03 7 . . file3 (6 Replies)
Discussion started by: Syeda Sumayya
6 Replies

10. Shell Programming and Scripting

Read info from api website and show retrieved data

good evening, i'm still new in scripting but i'm learning every day and i'm enjoying it. so i have api website (htt p://api.nobelprize.org/v1/prize.json), i want to make a script that allows me to give it two arguments like ./test.sh 2005 physics, 2000 is for the year and physics is category... (1 Reply)
Discussion started by: kalbsghir
1 Replies
SYSPROFILE(8)						      System Manager's Manual						     SYSPROFILE(8)

NAME
sysprofile - modular centralized shell configuration DESCRIPTION
sysprofile is a generic approach to configure shell settings in a modular and centralized way mostly aimed at avoiding work for lazy sysad- mins. It has only been tested to work with the bash shell. It basically consists of the small /etc/sysprofile shell script which invokes other small shell scripts having a .bash suffix which are contained in the /etc/sysprofile.d/ directory. The system administrator can drop in any script he wants without any naming convention other than that the scripts need to have a .bash suffix to enable automagic sourcing by /etc/sysprofile. This mechanism is set up by inserting a small shell routine into /etc/profile for login shells and optionally into /etc/bashrc and/or /etc/bash.bashrc for non-login shells from where the actual /etc/sysprofile script is invoked: if [ -f /etc/sysprofile ]; then . /etc/sysprofile fi For using "sysprofile" under X11, one can source it in a similar way from /etc/X11/Xsession or your X display manager's Xsession file to provide the same shell environment as under the console in X11. See the example files in /usr/share/doc/sysprofile/ for illustration. For usage of terminal emulators with a non-login bash shell under X11, take care to enable sysprofile via /etc/bash.bashrc. If not set this way, your terminal emulators won't come up with the environment defined by the scripts in /etc/sysprofile.d/. Users not wanting /etc/sysprofile to be sourced for their environment can easily disable it's automatic mechanism. It can be disabled by simply creating an empty file called $HOME/.nosysprofile in the user's home directory using e.g. the touch(1) command. Any single configuration file in /etc/sysprofile.d/ can be overridden by any user by creating a private $HOME/.sysprofile.d/ directory which may contain a user's own version of any configuration file to be sourced instead of the system default. It's names have just to match exactly the system's default /etc/sysprofile.d/ configuration files. Empty versions of these files contained in the $HOME/.syspro- file.d/ directory automatically disable sourcing of the system wide version. Naturally, users can add and include their own private script inventions to be automagically executed by /etc/sysprofile at login time. OPTIONS
There are no options other than those dictated by shell conventions. Anything is defined within the configuration scripts themselves. SEE ALSO
The README files and configuration examples contained in /etc/sysprofile.d/ and the manual pages bash(1), xdm(1x), xdm.options(5), and wdm(1x). Recommended further reading is everything related with shell programming. If you need a similar mechanism for executing code at logout time check out the related package syslogout(8) which is a very close compan- ion to sysprofile. BUGS
sysprofile in its current form is mainly restricted to bash(1) syntax. In fact it is actually a rather embarrassing quick and dirty hack than anything else - but it works. It serves the practical need to enable a centralized bash configuration until something better becomes available. Your constructive criticism in making this into something better" is very welcome. Before i forget to mention it: we take patches... ;-) AUTHOR
sysprofile was developed by Paul Seelig <pseelig@debian.org> specifically for the Debian GNU/Linux system. Feel free to port it to and use it anywhere else under the conditions of either the GNU public license or the BSD license or both. Better yet, please help to make it into something more worthwhile than it currently is. SYSPROFILE(8)
All times are GMT -4. The time now is 11:26 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy