Sponsored Content
Top Forums Shell Programming and Scripting Compare lines in two files (Memory getting exhausted) Post 302337151 by Corona688 on Thursday 23rd of July 2009 01:11:32 PM
Old 07-23-2009
How huge is file1? If it can't load all the strings into memory, then it just won't go. You may have to process it in smaller batches then merge them.
 

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Memory exhausted in awk

Dear All, I have executed a awk script in linux box which consists of 21 Million records.And i have two mapping files of 500 and 5200 records.To my surprise i found an error awk: cmd. line:19: (FILENAME=/home/FILE FNR=21031272) fatal: Memory exhausted. Is there any limitation for records... (3 Replies)
Discussion started by: cskumar
3 Replies

2. Shell Programming and Scripting

Trying to compare lines in 2 files

Hello, I am new to scripting and need some help. In looking at other posts on this forum, I came up with the following logic. I cannot figure out why I am getting names of files of the current directory in my echo output. Scenario: message file has a line containing the version. Version.txt... (2 Replies)
Discussion started by: brdholman
2 Replies

3. Shell Programming and Scripting

compare two files and to remove the matching lines on both the files

I have two files and need to compare the two files and to remove the matching lines from both the files (4 Replies)
Discussion started by: shellscripter
4 Replies

4. Shell Programming and Scripting

compare files and then remove some lines

Hi everyone I have a dilemma and I'm hoping someone has an answer for me. I have two files: # cat masterfile line3 line4 line5 line6 line7 # cat tempfile line1 line2 line3 line4 I want to compare tempfile with masterfile. (3 Replies)
Discussion started by: soliberus
3 Replies

5. Web Development

PHP Fatal error: Allowed memory size of 134217728 bytes exhausted

Any clues on how to get rid of this PHP error? PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 71 bytes) in /website/www/includes/functions_manpages.php on line 58 PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 71... (4 Replies)
Discussion started by: Neo
4 Replies

6. Shell Programming and Scripting

Error PHP Fatal error: Allowed memory size of 67108864 bytes exhausted(tried to allocate 401 bytes)

While running script I am getting an error like Few lines in data are not being processed. After googling it I came to know that adding such line would give some memory to it ini_set("memory_limit","64M"); my input file size is 1 GB. Is that memory limit is based on RAM we have on... (1 Reply)
Discussion started by: elamurugu
1 Replies

7. Shell Programming and Scripting

Compare files by lines and columns

Inspired by the extremely short awk code from Ygor on this post I wanted to compare two files on only one field. I can't get it to work. Can anybody help on explaining the code and fix the code? My code which does not work: awk 'BEGIN{a=1};a!=1' file1.txt file2.txt >outfile.txt file1.txt... (1 Reply)
Discussion started by: sdf
1 Replies

8. UNIX for Dummies Questions & Answers

Compare lines in 2 files

I have 2 files with exactly the same information (with header and separated by ";") and what I would like to do is print (for both files!) the columns that are different and also print the "key" column that is equal in the 2 files For example, if File1: key1;aaa;bbb;ccc key2;ddd;eee;fff... (4 Replies)
Discussion started by: mvalonso
4 Replies

9. Shell Programming and Scripting

Compare lines between two files

I have two files I need to compare these two files and take the lines that are common in both the files and consider the line present in second file for my further processing I have used "Awk" along with "FNR and NR" but that is not working gawk -F= ' > FNR==NR {a=$1; next}; > ... (2 Replies)
Discussion started by: Priya Amaresh
2 Replies
MERGE(1)						      General Commands Manual							  MERGE(1)

NAME
merge - three-way file merge SYNOPSIS
merge [ options ] file1 file2 file3 DESCRIPTION
merge incorporates all changes that lead from file2 to file3 into file1. The result ordinarily goes into file1. merge is useful for com- bining separate changes to an original. Suppose file2 is the original, and both file1 and file3 are modifications of file2. Then merge combines both changes. A conflict occurs if both file1 and file3 have changes in a common segment of lines. If a conflict is found, merge normally outputs a warning and brackets the conflict with <<<<<<< and >>>>>>> lines. A typical conflict will look like this: <<<<<<< file A lines in file A ======= lines in file B >>>>>>> file B If there are conflicts, the user should edit the result and delete one of the alternatives. OPTIONS
-A Output conflicts using the -A style of diff3(1), if supported by diff3. This merges all changes leading from file2 to file3 into file1, and generates the most verbose output. -E, -e These options specify conflict styles that generate less information than -A. See diff3(1) for details. The default is -E. With -e, merge does not warn about conflicts. -L label This option may be given up to three times, and specifies labels to be used in place of the corresponding file names in conflict reports. That is, merge -L x -L y -L z a b c generates output that looks like it came from files x, y and z instead of from files a, b and c. -p Send results to standard output instead of overwriting file1. -q Quiet; do not warn about conflicts. -V Print RCS's version number. DIAGNOSTICS
Exit status is 0 for no conflicts, 1 for some conflicts, 2 for trouble. IDENTIFICATION
Author: Walter F. Tichy. Manual Page Revision: 5.8.1; Release Date: 2012-06-06. Copyright (C) 2010-2012 Thien-Thi Nguyen. Copyright (C) 1990, 1991, 1992, 1993, 1994, 1995 Paul Eggert. Copyright (C) 1982, 1988, 1989 Walter F. Tichy. SEE ALSO
diff3(1), diff(1), rcsmerge(1), co(1). BUGS
It normally does not make sense to merge binary files as if they were text, but merge tries to do it anyway. GNU RCS 5.8.1 2012-06-06 MERGE(1)
All times are GMT -4. The time now is 10:52 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy