Login or Register to Ask a Question and Join Our Community


UNIX for Dummies Questions & Answers

Copy Files From a big list

 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Copy Files From a big list
# 1  
Old 11-21-2014
Copy Files From a big list
Requirement:

When I do
Code:
 ls -ltr /home/data/orders

I get a huge list of files, I need to copy that last 50 to another directory say
Code:
 /home/work/

later, I will do my ETL process and then again I need to copy from 51 to 100 and so on.

What is the command to copy files specifying 1 to 50 ordered by date timestamp desc.
# 2  
Old 11-21-2014
There is no (single) command to satisfy your requirement, I'm afraid.

Also not sure if I fully understood the requirement...

All three commands should be executed in the /home/data/orders directory.

Code:
# Invoked once only: create file list
# "ls -lrt" shows the newest files at the bottom
# For easier handling with sed, the created file list will have them at the top
# find . -maxdepth 1 -type f ! -name '.*' -printf '%T@ "%p"\n' | sort -r | cut -d' ' -f2- >filelistdesc.dat
ls -Ft | sed '/\/$/d; s/.*/"&"/' >filelistdesc.dat

Code:
# Grab 50 files from the file list and copy them to /home/work
sed 50q filelistdesc.dat | xargs cp -vt /home/work

# Delete 50 files in the file list
sed -i '1,50d' filelistdesc.dat

Usage:
ls ... # once only
sed 50q ...
sed -i ...
ETL process
sed 50q ...
sed -i ...
ETL process
sed 50q ...
sed -i ...
and so on.

Hope this helps.
Last edited by junior-helper; 11-22-2014 at 04:09 PM.. Reason: commented out the original find command, added "ls" based alternative; Thank you Don and RudiC
# 3  
Old 11-21-2014
Hi junior-helper,
I don't know why you would use ls -l when you only want names or why you would use ls -r when you want files in forward order.

Why not just ls -t > filelist.txt to get your list of files in the order you want?
This User Gave Thanks to Don Cragun For This Post:
junior-helper
# 4  
Old 11-22-2014
How about
Code:
ls -Ft /home/data/orders | grep -v "/$" | split -l50 -d - workfile
for W in work*; do ETL process; done

# 5  
Old 11-22-2014
Hello Don,
I offer two answers to your first question, although I think it was a rhetorical one in first place or was no question at all, hehe.
  1. True.
  2. Well, while it's true that I mentioned ls in the comment, I didn't use ls at all. Actually, the OP mentioned ls -ltr first and the comment above the find command was placed just in case the OP wonders why the files are processed top-down and not bottom-up.
ls -t > filelist.txt This looks and *is* very attractive and is *THE* way to create the required file list, but what if there are sub-directories in the particular directory (OK, they could be grepped out like RudiC did), but what if there are spaces in some files (my xargs command would fail then... thats why I told the printf option of the find command to put the file names inside quotes).

Ultimately,
Code:
ls -Ft | grep -v '/$' | sed 's/.*/"&"/' >filelist.txt

is far better than the stunt I performed with find.
# 6  
Old 11-22-2014
Then, why not
Code:
ls -Ft | sed '/\/$/d; s/.*/"&"/' >filelist.txt

 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

I'm trying 2 copy files after using the grep to get a list

for XmlFileName in ${xmlFileNames} do XmlFileName=$(echo $XmlFileName | sed 's|./||') # Remove leading ./ path that find command prefixes to filenames cp $XmlFileName $NEW_DIR/ done (1 Reply)
Discussion started by: emc^24sho
1 Replies

2. Shell Programming and Scripting

Copy files to a dir using from a list

Hi all, I'd very grateful for some help with the following: I have a directory with several subdirectories with files in them. All files are named different, even between different subdirectories. I also have a list with some of those file names in a txt file (without the path, just the file... (5 Replies)
Discussion started by: foracoffee
5 Replies

3. Shell Programming and Scripting

Copy list of files from a keyword list to another directory

Hello, I have a folder with a massive amount of files, and I want to copy out a specific subset of the files to a new directory. I would like to use a text file with the filenames listed, but can't get it to work. The thing I'm hung up on is that the folder names in the path can and do have... (5 Replies)
Discussion started by: twjolson
5 Replies

4. Shell Programming and Scripting

Copy files on a list to another directory

Hi. I have a list with file names like testfile1.wav testfile2.wav testfile3.wav and a folder that contains a large number of wav files (not only the ones on the list). I would like to copy the files whose names are on the list from the wav file directory to a new directory. I... (5 Replies)
Discussion started by: Bloomy
5 Replies

5. Shell Programming and Scripting

copy all files that ls -l list

Hi, I need to copy all files from this statement to another folder: ls –l ./apps | grep –E “^l|^v” How can i do that? best regards! (8 Replies)
Discussion started by: xus
8 Replies

6. Shell Programming and Scripting

how to copy files followed by list of names of all the files in /etc?

....... (2 Replies)
Discussion started by: pcbuilder
2 Replies

7. Shell Programming and Scripting

how can i copy a list of files with different names into others directory have the same name

dear all. how can i copy a list of files with different names into others directory have the same name like i have 3 files 10_10 10_10_11 10_10_11_12 and i have 3 directories 10_10 10_10_11 10_10_11_12 how can i make a loop to cp this files into the directory have the same name like... (0 Replies)
Discussion started by: t17
0 Replies

8. Shell Programming and Scripting

find list of files from a list and copy to a directory

I will be very grateful if someone can help me with bash shell script that does the following: I have a list of filenames: A01_155716 A05_155780 A07_155812 A09_155844 A11_155876 that are kept in different sub directories within my current directory. I want to find these files and copy... (3 Replies)
Discussion started by: manishabh
3 Replies

9. UNIX for Dummies Questions & Answers

How to List and copy the files containing a string

:confused: I have more than 8000 files in a dir, I need to copy to other dir which containing the "sample" I tried grep -il "1189609240791-1268115603299237276@216.109.111.119 ' | cp /tmp/inv Nothing is happening for long time for 100 file dir too, Any one can help me? (11 Replies)
Discussion started by: redlotus72
11 Replies

10. UNIX for Dummies Questions & Answers

copy and rename list of files

Hi all, I am a newbie in writng unix..I am using ksh shell..Does anyone know how to copy a list o files from directory A to directory B with differnt names? i.e in Dir A, I have RPT101.555.TXT RPT102.666.TXT and I want to copy those files to dir B with new naming convention.. in Dir B,... (7 Replies)
Discussion started by: kinmak
7 Replies
Login or Register to Ask a Question