Sponsored Content
Top Forums Shell Programming and Scripting Efficient way to search array in text file by awk Post 302958734 by Don Cragun on Monday 26th of October 2015 04:14:12 AM
Old 10-26-2015
Hi siramitsharma,
Forum rules prohibit sending private message asking people to respond to your posts.

Note that with the two lines in SPLNO.TXT:
Code:
918542054921|30|1|2
854215595|12|1|2

any of the following lines in MDN.TXT:
Code:
918542054921
91854205492
9185420549
918542054
91854205
9185420
918542
91854
9185
918
91
9
8542054921
854205492
85420549
8542054
854205
85420
8542
854
85
8

would match the 1st line. And any of the following lines in MDN.TXT:
Code:
854215595
85421559
8542155
854215
85421
8542
854
85
8

would match the 2nd line.
So maybe you could pre-build a table of the values that could match entries from the first file and instead of performing lots of matches and comparisons in a loop, you could just test if((mdn in table) || ("91"mdn in table)) instead of the four slower tests you are current using to see if there is a match.

Since every entry in the input has the final two fields with the values 1 and 2, respectively, all we need to know is whether or not there is a match; not what values appear if there is a match. This is important because both entries above match lines in MDN.TXT containing the values:
Code:
8542
854
85
8

You also need to explain why the 2nd field in the 1st line above has the value 30. Since field 1 is 12 characters (918542054921), the longest possible string that can be matched is 12 characters. And, on the 2nd line we have a 1st field (854215595) with length 9 and a second field containing 12. So, I repeat, what use is the 2nd field in SPLNO.TXT other than to give you two more tests to slow down your loop?
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

text file search and replace with awk

hello all greeting for the day i have a text file as the following text.xml abcd<FIELD>123.456</FIELD>efgh i need to replace the value between <FIELD> and </FIELD> by using awk command. please throw some light on this. thank you very very much Erik (5 Replies)
Discussion started by: erikshek
5 Replies

2. Shell Programming and Scripting

search needed part in text file (awk?)

Hello! I have text file: From aaa@bbb Fri Jun 1 10:04:29 2010 --____OSPHWOJQGRPHNTTXKYGR____ Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline My code '234565'. ... (2 Replies)
Discussion started by: candyme
2 Replies

3. Shell Programming and Scripting

search text file in file if this file contains necessary text (awk,grep)

Hello friends! Help me pls to write correct awk and grep statements for my task: I have got files with name filename.txt It has such structure: Start of file FROM: address@domen.com (12...890) abc DATE: 11/23/2009 on Std SUBJECT: any subject End of file So, I must check, if this file... (4 Replies)
Discussion started by: candyme
4 Replies

4. Shell Programming and Scripting

Better and efficient way to reverse search a file for first matched line number.

How to reverse search for a matched string in a file. Get line# of the first matched line. I am getting '2' into 'lineNum' variable. But it feels like I am using too many commands. Is there a better more efficiant way to do this on Unix? abc.log aaaaaaaaaaaaa bbbbbbbbbbbbb... (11 Replies)
Discussion started by: kchinnam
11 Replies

5. Shell Programming and Scripting

Need an efficient way to search for a tag in an xml file having millions of rows

Hi, I have an XML file with around 1 billion rows in it and i am trying to find the number of times a particular tag occurs in it. The solution i am using works but takes a lot of time (~1 hr) .Please help me with an efficient way to do this. Lets say the input file is <Root> ... (13 Replies)
Discussion started by: Sheel
13 Replies

6. Homework & Coursework Questions

Efficient Text File Writing

Use and complete the template provided. The entire template must be completed. If you don't, your post may be deleted! 1. The problem statement, all variables and given/known data: Write a template main.c file via shell script to make it easier for yourself later. The issue here isn't writing... (2 Replies)
Discussion started by: george3isme
2 Replies

7. Shell Programming and Scripting

Efficient population of array from text file

Hi, I am trying to populate an array with data from a text file. I have a working method using awk but it is too slow and inefficent. See below. The text file has 70,000 lines. As awk is a line editor it reads each line of the file until it gets to the required line and then processes it.... (3 Replies)
Discussion started by: carlr
3 Replies

8. Shell Programming and Scripting

Search and replace from file in awk using a 16 bit text file

Hello, Some time ago a helpful awk file was provided on the forum which I give below: NR==FNR{A=$0;next}{for(j in A){split(A,P,"=");for(i=1;i<=NF;i++){if($i==P){$i=P}}}}1 While it works beautifully on English and Latin characters i.e. within the ASCII range of 127, the moment a character beyond... (6 Replies)
Discussion started by: gimley
6 Replies

9. Shell Programming and Scripting

Search text beween tags and write to file using awk

Hi Friends, I have a very big text file, that has code for multiple functions. I have scan through the file and write each function in seperate file. All functions starts with BEGIN DSFNC Identifier "ABCDDataValidationfnc" and ends with END DSFNC I need create a file(using identifier)... (2 Replies)
Discussion started by: anandapani
2 Replies

10. Shell Programming and Scripting

Read in search strings from text file, search for string in second text file and output to CSV

Hi guys, I have a text file named file1.txt that is formatted like this: 001 , ID , 20000 002 , Name , Brandon 003 , Phone_Number , 616-234-1999 004 , SSNumber , 234-23-234 005 , Model , Toyota 007 , Engine ,V8 008 , GPS , OFF and I have file2.txt formatted like this: ... (2 Replies)
Discussion started by: An0mander
2 Replies
magic(4)							   File Formats 							  magic(4)

NAME
magic - file command's magic number file SYNOPSIS
/etc/magic DESCRIPTION
The file(1) command identifies the type of a file using, among other tests, a test for whether the file begins with a certain magic number. The /etc/magic file, or a file specified as an option-argument to the -m or -M options of file(1), specifies what magic numbers are to be tested for, what message to print if a particular magic number is found, and additional information to extract from the file. Each line of the file specifies a position-sensitive test to perform. A test compares the data starting at a particular offset in the file with a 1-byte, 2-byte, 4-byte, or 8-byte numeric value or string. If the test succeeds, a message is printed. The line consists of the fol- lowing fields (separated by tabs): offset type value message offset A number specifying the offset, in bytes, into the file of the data which is to be tested. type The type of the data to be tested. The possible values are: byte, d1, dC A one-byte signed value. short, d2, dS A 2-byte signed value. long, d4, dI, dL, d A 4-byte signed value. llong, d8 An 8-byte signed value ubyte, u1, uC A one-byte unsigned value. ushort, u2, uS A 2-byte unsigned value. ulong, u4, uI, uL, u A 4-byte unsigned value. ullong, u8 An 8-byte unsigned value. string, s A string of bytes. All type specifiers, except for string and s, may be followed by a mask specifier of the form &number. If a mask specifier is given, the value is AND'ed with the number before any comparisons are done. The number is specified in C form. For instance, 13 is decimal, 013 is octal, and 0x13 is hexadecimal. value The value to be compared with the value from the file. If the type is numeric, this value is specified in C form. If it is a string, it is specified as a C string with the usual escapes permitted (for instance, for NEWLINE). Numeric values may be preceded by a character indicating the operation to be performed, as follows: = The value from the file must equal the specified value. < The value from the file must be less than the specified value. > The value from the file must be greater than the specified value. & All the bits in the specified value must be set in the value from the file. ^ At least one of the bits in the specified value must not be set in the value from the file. x Any value will match. If the character is omitted, it is assumed to be "=". For comparison of numeric values, the sign and size of both the value in the file and the value from the value field of the magic entry will match that of the corresponding type field. If there is a non-zero mask (&) in the type field, the compar- ison will be unsigned. For string values, the byte string from the file must match the specified byte string. The byte string from the file which is matched is the same length as the specified byte string. If the value is a string, it can contain the following sequences: character The backslash-escape sequences \, a, , f, , , , v. octal Octal sequences that can be used to represent characters with specific coded values. An octal sequence consists of a backslash followed by the longest sequence of one, two, or three octal-digit characters(01234567). message The message to be printed if the comparison succeeds. If the string contains a printf(3C) format specification, the value from the file (with any specified masking performed) is printed using the message as the format string. Some file formats contain additional information which is to be printed along with the file type. A line which begins with the character ">" indicates additional tests and messages to be printed. If the test on the line preceding the first line with a ">" succeeds, the tests specified in all the subsequent lines beginning with ">" are performed, and the messages are printed if the tests succeed. The next line which does not begin with a ">" terminates this. FILES
/etc/magic SEE ALSO
file(1), file(1B), printf(3C) NOTES
In Solaris 9 and prior releases, the file utility may have performed unsigned comparisons for types byte, short, and long. Old user-defined magic files, which were specified with the -m option, will need modification of byte, short, and long entries to their corresponding unsigned types (ubyte, ushort, or ulong) for those entries for which all of the following are true: o The entry uses the "<" or the ">" operator. o The type field does not contain a non-zero mask. o The intention of the entry is to test unsigned values. For example, if the following entry is expected to match any non-zero, one-byte value from the file, including values for which the sign bit is on: #offset type value message 0 byte >0 this matches any non-zero value then that entry should be changed to: 0 ubyte >0 this matches any non-zero value In Solaris 7 through Solaris 9, when applying tests for magic file entries whose type field is the numeric type "short" or "long", the file utility in the x86 environment would switch the byte order of the numeric values read. Starting in Solaris 10, the byte order will not be switched on x86. A test for a numeric value whose byte order is identical in both little- and big-endian architectures may require two magic file entries, to ensure that the test correctly identifies files in both environments. For example, a magic file entry that will match on a big-endian system may look like this: 0 long 0xf00000ff extended accounting file Its corresponding magic file entry that will match the same value on a little-endian system would look like this: 0 long 0xff0000f0 extended accounting file BUGS
There should be more than one level of subtests, with the level indicated by the number of `>' at the beginning of the line. SunOS 5.10 6 Feb 2004 magic(4)
All times are GMT -4. The time now is 06:33 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy