sort retrieved data files in different dirs


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting sort retrieved data files in different dirs
# 1  
Old 04-08-2011
sort retrieved data files in different dirs

Hi,

I have a script to retrieve data files from server and store them in a directory on local disk. The script runs everyday as cron job. But now those files are too many so my boss wants me to put the files into different directories base on dates.

Those files look like this:

Code:
FCNCP104500104_SPA_2011-04-08_04-53-56-GMT_P08-00.nar

My original script:


Code:
#! /bin/bash

EMC_HOST='3.3.3.3'
EMC_USER='admin'
EMC_PASSWORD='password'
NAV_LOG_FILE='/var/log/naviseccli'
NAVISECCLI='/opt/Navisphere/bin/naviseccli'
NAV_BASE_CMD="${NAVISECCLI} -h ${EMC_HOST} -user ${EMC_USER} -password ${EMC_PASSWORD} -scope 0"
RETRIEVE_DIR='/mnt/emc_data'

#
# Simple logger.
#
log () {
    echo ${1} >> ${NAV_LOG_FILE}
}

#
# Retrieve file.
#
retrieve () {
    log "Retrieve file ${1}..."
    cmd="$NAV_BASE_CMD analyzer -archive -file ${1} -o"
    eval $cmd
    if [ $? -ne 0 ] ; then
        log "$(date): Retrieve $1 failed"
        log "command: $cmd"
        log "error code: $?"
        exit -1
    fi
}

#
# List all archives on EMC.
#
list () {
    file_list=$($NAV_BASE_CMD analyzer -archive -list | awk 'BEGIN {IGNORECASE=1;} /nar$/ { print $NF }')
    if [ $? -ne 0 ] ; then
        log "$(date): list archives failed"
        log "error code: $?"
        exit -2
    fi
}

cd $RETRIEVE_DIR || exit -1

list
for f in ${file_list} ; do
    [ -f ${f} ] || retrieve ${f}
done

Any suggestions to improve (rewrite) the script? Smilie
# 2  
Old 04-08-2011
adding a
Code:
mkdir `date'+%Y-%m-%'`

and chmod it to 777 at beginning of script
then change the retrieve directory to this new directory
# 3  
Old 04-08-2011
Thanks for your reply, pkabali.

But the list function will list all files available on the server, that way would retrieve all files into each date '+%Y-%m-%' directory. It's hard to determine which file is already retrieved.
# 4  
Old 04-08-2011
Based on file names with the format:
Code:
FCNCP104500104_SPA_2011-04-08_04-53-56-GMT_P08-00.nar

The following code will extract "YYYYMMDD" from it and create a directory with the same name, if it does not already exists:
Code:
  mYYYYMMDD=$(echo ${mFname} | sed 's/.*\([0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}\).*/\1/;s/-//g')
  if [[ ! -d ${mYYYYMMDD} ]]; then
    mkdir ${mYYYYMMDD}
  fi

This User Gave Thanks to Shell_Life For This Post:
# 5  
Old 04-10-2011
Thanks, Shell_Life.

I used awk, but your way worked, too.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Read info from api website and show retrieved data

good evening, i'm still new in scripting but i'm learning every day and i'm enjoying it. so i have api website (htt p://api.nobelprize.org/v1/prize.json), i want to make a script that allows me to give it two arguments like ./test.sh 2005 physics, 2000 is for the year and physics is category... (1 Reply)
Discussion started by: kalbsghir
1 Replies

2. Shell Programming and Scripting

Merge and Sort tabular data from different text files

I have 42 text files; each containing up to 34 lines with following structure; file1 H-01 23 H-03 5 H-05 9 H-02 14 . . file2 H-01 17 H-02 43 H-04 7 H-05 8 H-03 7 . . file3 (6 Replies)
Discussion started by: Syeda Sumayya
6 Replies

3. Shell Programming and Scripting

Moving files into dirs corresponding to dates

I am trying to find a way to move files into corresponding date files. i=0 while read line do array="$line" (( i++ )) done < <(ls) cd $(echo ${array}) echo ${array}} pwd #cd "$(array}" ] || mkdir 2015 cd "2015" ] || mkdir 02-February ] || mkdir 03-March ] || mkdir... (10 Replies)
Discussion started by: newbie2010
10 Replies

4. Debian

Problem with files/dirs deletion

Hi, The other day i installed a PHP based CMS (modx) on my shell account and noticed that i couldn't delete any of files/dirs it created after. Also, i noticed that all that stuff is owned by username-www instead of username. I tried chown, chmod and using a PHP script to do the same wti... (4 Replies)
Discussion started by: pentago
4 Replies

5. Shell Programming and Scripting

Creating a list of files retrieved from MGET

Is there a way to create a txt file with the names of the files I retreive using mget *.file. I want to use this file as input to a delete command. Using HP-UX (3 Replies)
Discussion started by: jagf
3 Replies

6. UNIX for Dummies Questions & Answers

FTP SERVER logging of files retrieved

Is there any addon or finctionality in ftp (SFTP) Server that definitively indicates if a file was retireved and successfully? I am looking for this type of command with logging so -files retrieved -files deleted by remote users can be logged Must be compatible with AIX and UBUNTU. ... (0 Replies)
Discussion started by: cdc01
0 Replies

7. Homework & Coursework Questions

Shell script calling Perl function, sort and find data, write to new files

Use and complete the template provided. The entire template must be completed. If you don't, your post may be deleted! 1. The problem statement, all variables and given/known data: I must write a shell script that calls two external Perl functions--one of which sorts the data in a file, and... (6 Replies)
Discussion started by: kowit010
6 Replies

8. Shell Programming and Scripting

AWK help print dirs with files in it

Hi, I'm writing some start of day checks for my work. I want to check some dirs for files that have been created longer than 10 mins ago and not been transfered. I've already used a find command to write a list of files that meet this criteria to a log called sod.log i.e. ... (1 Reply)
Discussion started by: elcounto
1 Replies

9. Shell Programming and Scripting

script find files in two dirs HELP

I have a directory which is /home/mark/files/ , inside this particular I have a bunch of filles (see examples below) TST_SHU_00014460_20090302.txt TST_SHU_00016047_20090302.txt TST_SHU_00007838_20090303.txt TST_SHU_00056485_20090303.txt TST_SHU_00014460_20090303.txt... (2 Replies)
Discussion started by: fierusbentus
2 Replies

10. UNIX for Dummies Questions & Answers

I need to ls all files in 4-6 deep dirs

I need to print to file , a listing of all files below a certain directory. Example: I need to print to file a listing of all files below the etc dir (including the subdirectories) with their full path. Any ideas on how to do this with one command. Or is this something I need to do on all... (4 Replies)
Discussion started by: gforty
4 Replies
Login or Register to Ask a Question