Sponsored Content
Top Forums Shell Programming and Scripting Comparing two fixed width file Post 302523251 by anshul_er on Wednesday 18th of May 2011 08:53:22 AM
Old 05-18-2011
Error Comparing two fixed width file

Hi Guys

I am checking the treads to get the answer but i am not able to get the answer for my question.

I have two files. First file is a pattern file and the second file is the file i want to search in it. Output will be the lines from file2.
File1:
Code:
P2797f12af                    44751228
P2b1204d0f                    33470964
P2b1205f76                    35815429
P2797f0250                    8219027

File 2:
Code:
P2797ea6c0                    1942611  SAN   SAN
P2797f12af                    44751228 SAN   SAN
P2b1204d0f                    33470964 SAN   SAN
P2b1205f76                    35815429 SAN   SAN
P2797f0250                    8219027  SAN   SAN

Output:
Code:
P2797f12af                    44751228 SAN   SAN
P2b1204d0f                    33470964 SAN   SAN
P2b1205f76                    35815429 SAN   SAN
P2797f0250                    8219027  SAN   SAN

I am able to do this using below command:
Code:
fgrep -f file1 file2

But it is giving me an error of out or memory as my file size is more than 1 million.
I also tried splitting it:

Code:
split -l 10000 file1 file1.split.
for CHUNK in file1.split.* ; do
        fgrep -f "$CHUNK" file2
done
rm file1.split.*

It is also taking a lot of time. First loop is done really quick but for the next loop to start it is taking long time. Smilie

Can you please let me know if i am doing something wrong here. Or can you please provide me any awk command to do this stuff.

You guys are great... looking forward for your reply.

Last edited by radoulov; 05-18-2011 at 10:11 AM.. Reason: Code tags, please!
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Fixed Width file using AWK

I am using the following command at the Unix prompt to make my 'infile' into a fixed width file of 100 characters. awk '{printf "%-100s\n",$0}' infile > outfile However, there are some records with a special character "©" These records are using 3 characters in place of one and my record... (2 Replies)
Discussion started by: alok.benjwal
2 Replies

2. Shell Programming and Scripting

adding delimiter to a fixed width file

Hi , I have a file : CSCH74000.00 CSCH74000.00 CSCH74100.00 CSCH74000.00 CSCH74100.00 CSCH74000.00 CSCH74000.00 CSCH74100.00 CSCH74100.00 CSCH74100.00 I have to put a delimiter( say comma) in between after 6th character: CSCH74,000.00 CSCH74,000.00 CSCH74,100.00 (2 Replies)
Discussion started by: sumeet
2 Replies

3. Shell Programming and Scripting

Changing particular field in fixed width file

I have a fixed width file and i need to change 36th field to "G" in for about random 20 records? How can I do it? (4 Replies)
Discussion started by: dsravan
4 Replies

4. UNIX Desktop Questions & Answers

Help with Fixed width File Parsing

I am trying to parse a Fixed width file with data as below. I am trying to assign column values from each record to variables. When I parse the data, the spaces in all coumns are dropped. I would like to retain the spaces as part of the dat stored in the variables. Any help is appreciated. I... (4 Replies)
Discussion started by: sate911
4 Replies

5. Shell Programming and Scripting

Comparing column of variable length anf fixed width file

Hi, I have two input files. File1: ID Name Place 1-234~name1~Newyork 1-34~name2~Boston 1-2345~name3~Hungary File1 is a variable length file where each column is seperated by delimitter "~". File2: ID Country 1-34<<11 SPACES>>USA<<7 spaces>> 1-234<<10 SPACES>>UK<<8... (5 Replies)
Discussion started by: manneni prakash
5 Replies

6. Shell Programming and Scripting

Fixed-Width file from Oracle

Hi All, I have created a script which generates FIXED-WIDTH file by executing Oracle query. SELECT RPAD(NVL(col1,CHR(9)),20)||NVL(col2,CHR(9))||NVL(col3,CHR(9) FROM XYZ It generates the data file with proper alignment. But if same file i transfer to windows server or Mainframe... (5 Replies)
Discussion started by: Amit.Sagpariya
5 Replies

7. Shell Programming and Scripting

sorting a fixed width seq file

I have a file like this... 2183842512010-11-25 15379043 453130325 2386225062010-11-30 4946518 495952336 2386225062010-11-30 4946518 495952345 2386225062010-11-25 262066688 -516224026 2679350512010-11-25 262066688 -516224124 3196089062010-11-25 262066688 203238229... (5 Replies)
Discussion started by: issaq84mohd
5 Replies

8. UNIX for Dummies Questions & Answers

Length of a fixed width file

I have a fixed width file of length 53. when is try to get the lengh of the record of that file i get 2 different answers. awk '{print length;exit}' <File_name> The above code gives me length 50. wc -L <File_name> The above code gives me length 53. Please clarify on... (2 Replies)
Discussion started by: Amrutha24
2 Replies

9. Shell Programming and Scripting

Alter Fixed Width File

Thank u so much .Its working fine as expected. ---------- Post updated at 03:41 PM ---------- Previous update was at 01:46 PM ---------- I need one more help. I have another file(fixed length) that will get negative value (ex:-00000000003000) in postion (98 - 112) then i have to... (6 Replies)
Discussion started by: vinus
6 Replies

10. Answers to Frequently Asked Questions

Fixed width file issue and resolutions

I have a fixed width file with no separators , but the file has JUNK characters in it and i know how to remove it. but in few cases these junk characters having created a mess by splitting single row into 2 or 3 rows. I need to put them back to a single line like rest of the rows. This fixed... (2 Replies)
Discussion started by: santoshkm
2 Replies
PURITY(6)							   Games Manual 							 PURITY(6)

NAME
purity - a general purpose purity test SYNOPSIS
/usr/games/purity [ flags ] [ testname ] DESCRIPTION
Purity is an interactive purity test program with a simple, user interface and datafile format. For each test, questions are printed to the your terminal, and you are prompted for an answer to the current question. At a prompt, these are your choices: y Answer "yes" to the question. n Answer "no" to the question. b Backup one question, if you answered it incorrectly, or someone is watching you take the test, and you don't (or do) want to admit a different answer. r Redraw the current question. q Quit the test, and print the current score. ? Print a help screen for the current prompt. k Kill a section of the test. This skips all the questions of the test until the next subject heading. a Toggle answer mode between real answers and obfuscated answers. Real answers print "yes" and "no", while obfuscated answers are "Maybe" and "maybe". Obfuscated answers are preferred if you are shy, and don't want people to be able to read your answers over your shoulder as you take the test. d Toggle dERanGe output. s Print your current score on the test you are taking. l Toggle score logging. At the end of the test, your score is printed out. For most purity tests, lower scores denote more "experience" of the test material. FLAGS
These are the command line flags for the test. -a Show real answers (i.e. "yes" and "no") instead of obfuscated ones (i.e. "Maybe" and "maybe") as you answer the questions. -d PrINt THe tESt in DerANgeD pRInT. -f Take the test in fast mode. Only the questions are printed, and not any other text blocks, like the introdution, subject headers, and the conclusion. -l Take the test without having your score logged. -p Print the test without prompting for answers. This is useful for making hard copies of the tests without having to edit out the prompts by hand. -r Decrypt the test using the Rot 13 algorithm. This is done as a form of "protection", such that if you read a rot13 test and it offends you, it's your own fault. -z zoom through more prompts in large text blocks. The default is to prompt the user for more when a screenful of text has been printed without any user input. DATAFILE FORMAT
The format of the datafiles is a very simple format, intended such that new tests can quickly and easily be converted to run with the test. There are four types of text in a purity test datafile. Each type is contained in a bracket type of punctuation. The definitions are as follows: the styles of text blocks are: { plain text block } [ subject header ] ( test question ) and < conclusion > Plain text blocks are printed out character for character. Subject headers are preceded by their subject numbers, starting at 1, and then printed as text blocks. Questions are preceded by their numbers, and then prompt the user to answer the question, keeping track of the user's current score. Conclusions first calculate and print the user's score for the test, then print out the conclusion as a text block. If you wish to include any of the various bracket punctuation in your text, the backslash ("") character will escape the next character. To print a question with parentheses, you would use the following format: (have you ever written a purity test (like this one)?) the output would be this: 1. have you ever written a purity test (like this one)? and then it would have asked the user for her/his answer. For a generic datafile, use the "sample" datafile for the test. FILES
/var/games/purity.scores the score logfile /usr/share/games/purity/* test data files AUTHOR
Eric Lechner, lechner@ucscb.ucsc.edu 18 December 1989 PURITY(6)
All times are GMT -4. The time now is 10:56 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy