Comparing two fixed width file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Comparing two fixed width file
# 1  
Old 05-18-2011
Error Comparing two fixed width file

Hi Guys

I am checking the treads to get the answer but i am not able to get the answer for my question.

I have two files. First file is a pattern file and the second file is the file i want to search in it. Output will be the lines from file2.
File1:
Code:
P2797f12af                    44751228
P2b1204d0f                    33470964
P2b1205f76                    35815429
P2797f0250                    8219027

File 2:
Code:
P2797ea6c0                    1942611  SAN   SAN
P2797f12af                    44751228 SAN   SAN
P2b1204d0f                    33470964 SAN   SAN
P2b1205f76                    35815429 SAN   SAN
P2797f0250                    8219027  SAN   SAN

Output:
Code:
P2797f12af                    44751228 SAN   SAN
P2b1204d0f                    33470964 SAN   SAN
P2b1205f76                    35815429 SAN   SAN
P2797f0250                    8219027  SAN   SAN

I am able to do this using below command:
Code:
fgrep -f file1 file2

But it is giving me an error of out or memory as my file size is more than 1 million.
I also tried splitting it:

Code:
split -l 10000 file1 file1.split.
for CHUNK in file1.split.* ; do
        fgrep -f "$CHUNK" file2
done
rm file1.split.*

It is also taking a lot of time. First loop is done really quick but for the next loop to start it is taking long time. Smilie

Can you please let me know if i am doing something wrong here. Or can you please provide me any awk command to do this stuff.

You guys are great... looking forward for your reply.

Last edited by radoulov; 05-18-2011 at 10:11 AM.. Reason: Code tags, please!
# 2  
Old 05-18-2011
Ooops

---------- Post updated at 03:02 PM ---------- Previous update was at 02:58 PM ----------

what is your file size ?
# 3  
Old 05-18-2011
fgrep not working

Fgrep not working as files are huge.
Getting below error:
fgrep: not enough memory.
# 4  
Old 05-18-2011
What are the size of your file1 and file2 ?
# 5  
Old 05-18-2011
both file will be containing around 400,000 lines.
# 6  
Old 05-18-2011
Could you please provide their size ? (not their number of lines)
# 7  
Old 05-18-2011
file1: 99417680
file2: 20430220
In bytes
File 2 may also be larger that the file 1.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Answers to Frequently Asked Questions

Fixed width file issue and resolutions

I have a fixed width file with no separators , but the file has JUNK characters in it and i know how to remove it. but in few cases these junk characters having created a mess by splitting single row into 2 or 3 rows. I need to put them back to a single line like rest of the rows. This fixed... (2 Replies)
Discussion started by: santoshkm
2 Replies

2. Shell Programming and Scripting

Alter Fixed Width File

Thank u so much .Its working fine as expected. ---------- Post updated at 03:41 PM ---------- Previous update was at 01:46 PM ---------- I need one more help. I have another file(fixed length) that will get negative value (ex:-00000000003000) in postion (98 - 112) then i have to... (6 Replies)
Discussion started by: vinus
6 Replies

3. UNIX for Dummies Questions & Answers

Length of a fixed width file

I have a fixed width file of length 53. when is try to get the lengh of the record of that file i get 2 different answers. awk '{print length;exit}' <File_name> The above code gives me length 50. wc -L <File_name> The above code gives me length 53. Please clarify on... (2 Replies)
Discussion started by: Amrutha24
2 Replies

4. Shell Programming and Scripting

sorting a fixed width seq file

I have a file like this... 2183842512010-11-25 15379043 453130325 2386225062010-11-30 4946518 495952336 2386225062010-11-30 4946518 495952345 2386225062010-11-25 262066688 -516224026 2679350512010-11-25 262066688 -516224124 3196089062010-11-25 262066688 203238229... (5 Replies)
Discussion started by: issaq84mohd
5 Replies

5. Shell Programming and Scripting

Fixed-Width file from Oracle

Hi All, I have created a script which generates FIXED-WIDTH file by executing Oracle query. SELECT RPAD(NVL(col1,CHR(9)),20)||NVL(col2,CHR(9))||NVL(col3,CHR(9) FROM XYZ It generates the data file with proper alignment. But if same file i transfer to windows server or Mainframe... (5 Replies)
Discussion started by: Amit.Sagpariya
5 Replies

6. Shell Programming and Scripting

Comparing column of variable length anf fixed width file

Hi, I have two input files. File1: ID Name Place 1-234~name1~Newyork 1-34~name2~Boston 1-2345~name3~Hungary File1 is a variable length file where each column is seperated by delimitter "~". File2: ID Country 1-34<<11 SPACES>>USA<<7 spaces>> 1-234<<10 SPACES>>UK<<8... (5 Replies)
Discussion started by: manneni prakash
5 Replies

7. UNIX Desktop Questions & Answers

Help with Fixed width File Parsing

I am trying to parse a Fixed width file with data as below. I am trying to assign column values from each record to variables. When I parse the data, the spaces in all coumns are dropped. I would like to retain the spaces as part of the dat stored in the variables. Any help is appreciated. I... (4 Replies)
Discussion started by: sate911
4 Replies

8. Shell Programming and Scripting

Changing particular field in fixed width file

I have a fixed width file and i need to change 36th field to "G" in for about random 20 records? How can I do it? (4 Replies)
Discussion started by: dsravan
4 Replies

9. Shell Programming and Scripting

adding delimiter to a fixed width file

Hi , I have a file : CSCH74000.00 CSCH74000.00 CSCH74100.00 CSCH74000.00 CSCH74100.00 CSCH74000.00 CSCH74000.00 CSCH74100.00 CSCH74100.00 CSCH74100.00 I have to put a delimiter( say comma) in between after 6th character: CSCH74,000.00 CSCH74,000.00 CSCH74,100.00 (2 Replies)
Discussion started by: sumeet
2 Replies

10. UNIX for Dummies Questions & Answers

Fixed Width file using AWK

I am using the following command at the Unix prompt to make my 'infile' into a fixed width file of 100 characters. awk '{printf "%-100s\n",$0}' infile > outfile However, there are some records with a special character "©" These records are using 3 characters in place of one and my record... (2 Replies)
Discussion started by: alok.benjwal
2 Replies
Login or Register to Ask a Question