Skipped duplicated files from log


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Skipped duplicated files from log
# 1  
Old 04-30-2013
Skipped duplicated files from log

I have a script to get files from remote server to local path the issue I wanna log the output of every collected files but look like something went wrong , I feel that my error is hide in what follows:
  1. In ftp function I did not manage to control if file exist in my local directory then skipped (no need to collect it again , in order not to overwrite the collected one )
  2. " Move file1xxxx 20130428 " message from output log "20130428.log" is accumulated thus if "file1xxxx" was already organized in directory 20130428 & still in remote server when ftp works it will be collected & organized again (although I'm need for any clue to overcome this but no matter as the size of file has no changed ) but it will be added to 20130428.log as duplicated one & unfortunately I can't (sort, uniq) that log to skip any duplication , thanks for follow up Smilie


Code:
for file in * ; do
dir=$( echo $file | cut -c1-8 ) 
    [ -d $dir ] || mkdir -p $dir     
echo "$dir was created "
[ -f $file  ] && mv  $file  $dir      
echo "Move $file  $dir"    >>  /user/$dir.log 
     sort -u  /user/$dir.log
done


Last edited by arm; 04-30-2013 at 10:36 AM..
# 2  
Old 04-30-2013
Quote:
unfortunately I can't (sort, uniq) that log to skip any duplication
Here is that answer again. Smilie
Code:
for file in * ; do
  dir=$( echo $file | cut -c1-8 )
  [ -d $dir ] || mkdir -p $dir
  echo "$dir was created"
  [ -f $file ] && mv $file $dir

  log=/user/$dir.log
  echo "Move $file $dir" >> $log
  sort -u $log > /tmp/temp.x
  mv /tmp/temp.x $log
done

# 3  
Old 04-30-2013
Ensure that files are really files and more than 8 characters long!
Quote variables in command arguments!
Code:
for file in ?????????* ; do
  [ -f "$file" ] || continue
  dir=$( echo "$file" | cut -c1-8 )
  [ -d "$dir" ] || mkdir -p "$dir"
...

This User Gave Thanks to MadeInGermany For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Join files, omit duplicated records from one file

Hello I have 2 files, eg more file1 file2 :::::::::::::: file1 :::::::::::::: 1 fromfile1 2 fromfile1 3 fromfile1 4 fromfile1 5 fromfile1 6 fromfile1 7 fromfile1 :::::::::::::: file2 :::::::::::::: 3 fromfile2 5 fromfile2 (4 Replies)
Discussion started by: CHoggarth
4 Replies

2. Shell Programming and Scripting

Merge files and remove duplicated rows

In a folder I'll several times daily receive new files that I want to combine into one big file, without any duplicate rows. The file name in the folder will look like e.q: MissingData_2014-08-25_09-30-18.txt MissingData_2014-08-25_09-30-14.txt MissingData_2014-08-26_09-30-12.txt The content... (9 Replies)
Discussion started by: Bergans
9 Replies

3. Shell Programming and Scripting

How to remove duplicated lines?

Hi, if i have a file like this: Query=1 a a b c c c d Query=2 b b b c c e . . . (7 Replies)
Discussion started by: the_simpsons
7 Replies

4. Shell Programming and Scripting

Command getting skipped

Hi All, Its been a long time!!!!!! Having one problem in our live server.... we have perl code which runs every day for almost 15 to 18hrs process and it creates almost 150 to 200 sub process (arc get) sequentially. However in which randomly one or few of the process getting skipped. But... (2 Replies)
Discussion started by: Shahul
2 Replies

5. UNIX for Dummies Questions & Answers

Duplicated UID

Hi folks! I need you help to discover what's the impact of a duplicated UID in an operating system. What's the meaning when someone put in different users the same UID? (3 Replies)
Discussion started by: phcostabh
3 Replies

6. Shell Programming and Scripting

Makefile rule being skipped

I can't seem to get a rule in my Makefile to ever run... even if I change the rule to force make to re-enter the rule, or if I change the dependent files the rule depends on. Any ideas why the second rule is being ignored here? #MAKEFILES = $(DIRS:%=$(ROOT)/%/Makefile) #$(MAKEFILES):... (0 Replies)
Discussion started by: foureightyeast
0 Replies

7. Shell Programming and Scripting

Shell Script - find, recursively, all files that are duplicated

Hi. I have a problem that i can't seem to resolve. I need to create a script that list all the files, that are found recursively, with the same name. For example if a file exists in more than one directory with the same name it list all the files that he founds with all the info. Could someone... (5 Replies)
Discussion started by: KitFisto
5 Replies

8. Shell Programming and Scripting

remove duplicated columns

hi all, i have a file contain multicolumns, this file is sorted by col2 and col3. i want to remove the duplicated columns if the col2 and col3 are the same in another line. example fileA AA BB CC DD CC XX CC DD BB CC ZZ FF DD FF HH HH the output is AA BB CC DD BB CC ZZ FF... (6 Replies)
Discussion started by: kamel.seg
6 Replies

9. UNIX for Advanced & Expert Users

Can root ID be duplicated

Hi, I have the root id and the uid is always 0 for the root. Can i create another user say admin with the uid as 0? If so will it have the same privilege as that of user. Will there be any effect while doing this? lorcan (6 Replies)
Discussion started by: lorcan
6 Replies

10. UNIX for Dummies Questions & Answers

Log files duplicated themselves?

Something strange happend today, I checked in /var/log has usual and I noticed that I have a "messages.1", "cron.1" "maillog.1", and so on. This has never happened before so I just want to know if this is normal? Before I always had "messages" "maillog" and "cron" but now I have 2 of each file.... (4 Replies)
Discussion started by: DISTURBED
4 Replies
Login or Register to Ask a Question