Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Merging few files into one, duplicates are removed

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Merging few files into one, duplicates are removed
# 1  
Old 01-10-2013
Merging few files into one, duplicates are removed
Hello,

I have few file such as below:

Code:
abc.txt
def.txt
ghi.txt
jkl.txt
n.txt

I would like to merge all these files together into one file. At the same time, any duplicates will be removed.
Last edited by Scrutinizer; 01-10-2013 at 05:13 AM.. Reason: code tags
# 2  
Old 01-10-2013
Code:
 
cat *.txt | sort -u > output.txt

# 3  
Old 01-10-2013
You can also try:
Code:
awk '!a[$0]++' abc.txt def.txt ghi.txt jkl.txt n.txt > new_file

# 4  
Old 01-10-2013
what if those files are located on few different directories?

./2013/Jan/Fri/2013-01-11.log
./2013/Jan/Sat/2013-01-12.log
./2013/Jan/Tue/2013-01-15.log
./2013/Jan/Sun/2013-01-13.log
./2013/Jan/Thu/2013-01-10.log
./2013/Jan/Mon/2013-01-14.log
./2013/Jan/Wed/2013-01-16.log

im trying to make it as a variabe.
# 5  
Old 01-10-2013
Why not use command substitution like
Code:
awk '...' $(ls ./2013/Jan/*/*.log)

?
# 6  
Old 01-10-2013
thats what i want. Thanks RudiC
Login or Register to Ask a Question

Previous Thread | Next Thread

8 More Discussions You Might Find Interesting

1. Red Hat

Can all files under /tmp be safely removed

I wanted to know whether all files under /tmp can be safely removed. I guess that /tmp may also have temporary files for applications currently being worked on, so at the most those applications may just shut down. I hope that my question is clear whether all files under /tmp can be safely... (5 Replies)
Discussion started by: RHCE
5 Replies

2. UNIX for Advanced & Expert Users

How to find duplicates contents in a files by comparing other files?

Hi Guys , we have one directory ...in that directory all files will be set on each day.. files must have header ,contents ,footer.. i wants to compare the header,contents,footer ..if its same means display an error message as 'files contents same' (7 Replies)
Discussion started by: Venkatesh1
7 Replies

3. UNIX for Advanced & Expert Users

can I view removed/deleted files

I have a script that have deleted some files, and I need to know which files were deleted. Is there a log file in linux that shows deleted files? I use ubuntu. (2 Replies)
Discussion started by: locoroco
2 Replies

4. Shell Programming and Scripting

Duplicates to be removed

Hi, I have a text file with 2000 rows and 2000 columns (number of columns might vary from row to row) and "comma" is the delimiter. In every row, there maybe few duplicates and we need to remove those duplicates and "shift left" the consequent values. ex: 111 222 111 555 444 999 666... (6 Replies)
Discussion started by: prvnrk
6 Replies

5. Shell Programming and Scripting

Find duplicates from multuple files with 2 diff types of files

I need to compare 2 diff type of files and find out the duplicate after comparing each types of files: Type 1 file name is like: file1.abc (the extension abc could any 3 characters but I can narrow it down or hardcode for 10/15 combinations). The other file is file1.bcd01abc (the extension... (2 Replies)
Discussion started by: ricky007
2 Replies

6. UNIX for Dummies Questions & Answers

recovering files removed with rm

Hello, I was reading the manual on rm and it states that when you use 'rm' the files are usual recoverable, how is this done? Does it assume that a backup system is in place? Cheers Jack (4 Replies)
Discussion started by: jack1981
4 Replies

7. UNIX for Dummies Questions & Answers

Will Old Files Be Removed

I have windows Xp installed, and decided to install Solaris Sun Unix 10. The hard disk was previousely partitioned into 5 partition. C: = Win98 D = WinXP and e,f,g,h are applications and so on. When istalling Sun Unix, will all the drives be removed, or I will specify where to install it. Thanks... (5 Replies)
Discussion started by: sunsation
5 Replies

8. UNIX for Dummies Questions & Answers

Bring back removed files

Dear People I have removed some of my files and directories( by using rm and rmdir commands) by mistake. I wish to bring them back. How is it possible?( I am using solaris 2.6) best regards Reza Nazarian:( (2 Replies)
Discussion started by: Reza Nazarian
2 Replies
Login or Register to Ask a Question