Sponsored Content
Top Forums Shell Programming and Scripting Problem comparing 2 files with lot of data Post 302128286 by lorcan on Monday 23rd of July 2007 11:41:04 PM
Old 07-24-2007
Hope this helps

You can try something like the below

your_path is the path where there is enough space to execute the sort command for your huge files.

Code:
cat FILE1 FILE2 | sort -T your_path | uniq -u 

(or) to avoid UUOC

sort -T your_path FILE1 FILE2 | uniq -u

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

rename a lot of files again

here I go again...kinda hard to explain so I apologize. I need to rename a bunch of files in a directory. I need to remove the first three characters of the filename, and then toward the end of the filename there is constant text inside of brackets. here is a demo (not for real) 'ls -1' of the... (11 Replies)
Discussion started by: ajp7701
11 Replies

2. Shell Programming and Scripting

Last field problem while comparing two csv files

Hi All, I've two .csv files as below file1.csv abc, tdf, 223, tpx jgsd, tex, 342, rpy a, jdjdsd, 423, djfkld Where as file2.csv is the new version of file1.csv with some added fields in the end of each line and some additional lines. lfj, eru, 98, jkldj, 39, jdkj9 abc, tdf, 223, tpx,... (3 Replies)
Discussion started by: ganapati
3 Replies

3. UNIX for Dummies Questions & Answers

Sorting with unique piping for a lot of files

Hi power user, if I have this file: file1.txt: 1111 1111 2222 2222 3333 3333 3333 4444 4444 4444 when I run the sort file1.txt | uniq > data1.txt the result is (2 Replies)
Discussion started by: anjas
2 Replies

4. Shell Programming and Scripting

chmod a lot of files

So i have about 600gb of data.. in which there are alot of directories and alot of files.. Im trying to put this on a ftp server.. So i want to set the permissions on the directories to be 755 and the permission on the files to be 644. So i used: find . -type d -exec chmod 755 {}\; and find .... (6 Replies)
Discussion started by: supermiguel
6 Replies

5. Shell Programming and Scripting

Rename a lot of files using shells script

Hi This is the list file that i have : The files is more than this. I will rename one by one file become like this : So just change the time stamp 200906 become 200905. Is it possible using script ? Thanks (3 Replies)
Discussion started by: justbow
3 Replies

6. Shell Programming and Scripting

Problem in comparing 2 fields from 2 files

I've 2 files. Need to compare File1.Field1,File1.Field2 with File2.Field1,File2.Field2. If matches then create a new file. File1 10 A|ADB|967143.24|1006101.5 3E HK|DHB|24294.76|242513.89 ABN ACU|ADB|22104.69|51647.14 ABN BU|DBA|39137.14|109128.38 ABN|ADB|64466.89|167936.55 ABOC... (2 Replies)
Discussion started by: buster
2 Replies

7. Shell Programming and Scripting

Need to modify a lot of html files

Hello, I have about 3400 files in a tree structure (about 80% are html files). 1. I need to modify every html file to remove <p> style and old things like font attribute and add another style. 2. I need to change the root of all links that are in the html. e.g. change /old/path/ to /new/path... (1 Reply)
Discussion started by: Yaazkal
1 Replies

8. UNIX for Dummies Questions & Answers

Lot of warn files filling /

hi guys I have suse 11 sp1 and I have a lot of warn file filling / these are under /var/log there's this big one -rw-r----- 1 root root 3.9G Feb 1 10:28 warn warn: ASCII text and the others that are about 2.5 to 3MB - they are about 130 warn-*.bz2 -rw-r----- 1 root root 3.9G Feb... (2 Replies)
Discussion started by: karlochacon
2 Replies

9. Shell Programming and Scripting

Comparing the data in a 2 files

Hi Friends, I have a file 1 CREATE MULTISET TABLE TEYT_Q9_T.TEST ,NO FALLBACK , NO BEFORE JOURNAL, NO AFTER JOURNAL, CHECKSUM = DEFAULT, DEFAULT MERGEBLOCKRATIO ( XYZ DECIMAL(10,0), ABC VARCHAR(5) CHARACTER SET LATIN NOT CASESPECIFIC, PQR... (3 Replies)
Discussion started by: i150371485
3 Replies

10. Shell Programming and Scripting

Convert a lot of files in subdirectories automatically

Hi, I have a huge structure of directories and subdirectories contsining some data. The lowest folders contain a file "image.png" which need to be converted to "folder.jpg". But how can I do that for all these files automatically? That's what I alredy have find /path -type f -name... (1 Reply)
Discussion started by: KarlKarpfen
1 Replies
COMBINE(1)																COMBINE(1)

NAME
combine - combine sets of lines from two files using boolean operations SYNOPSIS
combine file1 and file2 combine file1 not file2 combine file1 or file2 combine file1 xor file2 _ file1 and file2 _ _ file1 not file2 _ _ file1 or file2 _ _ file1 xor file2 _ DESCRIPTION
combine combines the lines in two files. Depending on the boolean operation specified, the contents will be combined in different ways: and Outputs lines that are in file1 if they are also present in file2. not Outputs lines that are in file1 but not in file2. or Outputs lines that are in file1 or file2. xor Outputs lines that are in either file1 or file2, but not in both files. "-" can be specified for either file to read stdin for that file. The input files need not be sorted, and the lines are output in the order they occur in file1 (followed by the order they occur in file2 for the two "or" operations). Bear in mind that this means that the operations are not commutative; "a and b" will not necessarily be the same as "b and a". To obtain commutative behavior sort and uniq the result. Note that this program can be installed as "_" to allow for the syntactic sugar shown in the latter half of the synopsis (similar to the test/[ command). It is not currently installed as "_" by default, but you can alias it to that if you like. SEE ALSO
join(1) AUTHOR
Copyright 2006 by Joey Hess <joey@kitenet.net> Licensed under the GNU GPL. moreutils 2012-04-09 COMBINE(1)
All times are GMT -4. The time now is 12:42 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy