Sponsored Content
Top Forums Shell Programming and Scripting Extract data based on specific search criteria Post 302460284 by jaygamini on Wednesday 6th of October 2010 11:05:26 AM
Old 10-06-2010
Extract data based on specific search criteria

I have a huge file (about 2 millions records) contains data separated by “,” (comma). As part of the requirement, I can't change the format. The objective is to remove some of the records with the following condition. If the 23rd field on each line start with 302 , I need to remove that from the original file. Simple grep command like “grep –v ^302” but here 302 is actually 23rd field and separated by comma. Please see the sample input and expected out. Your immediate help is really appreciated.
Code:
Data,4l4680,71130,2010,277,01/03/2011,1,1,2,,,2,0,01/11/2010,,,,0,0,,0,,302619988771130,0,4l4680,Call,302619988771130,99988771130,1,
Data,4l4680,1132,2010,176,01/03/2011,1,1,2,,,2,0,01/11/2010,,,,0,0,,0,,302619988771132,0,14680,Call,302619988771132,99988771132,1,
Data,4l3689,1133,2010,1574,,1,1,1,,,2,0,,,,,0,0,,0,,302619988871133,0,12689,_Call,302619988871133,99988871133,1,
Data,05678,9131,2010,18,17/01/2011,2,1,2,DPE,TEST,2,0,18/12/2010,,,,1286200,0,09/08/2010,-2949,,1131,00,1678,all,131,99998881131,1,
Data,6909,289,2010,031,,1,1,1,Irvin,Andé,2,0,,520007980,ON,BH,0,0,,0,,000569,0,1909,CEST,56909,932356909,1,


Output:
Code:
Data,4l4680,71130,2010,277,01/03/2011,1,1,2,,,2,0,01/11/2010,,,,0,0,,0,,302619988771130,0,4l4680,Call,302619988771130,99988771130,1,
Data,4l4680,1132,2010,176,01/03/2011,1,1,2,,,2,0,01/11/2010,,,,0,0,,0,,302619988771132,0,14680,Call,302619988771132,99988771132,1,
Data,4l3689,1133,2010,1574,,1,1,1,,,2,0,,,,,0,0,,0,,302619988871133,0,12689,_Call,302619988871133,99988871133,1,



Moderator's Comments:
Mod Comment Please use code tags for your data en code

Last edited by Franklin52; 10-06-2010 at 01:20 PM..
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Select records based on search criteria on first column

Hi All, I need to select only those records having a non zero record in the first column of a comma delimited file. Suppose my input file is having data like: "0","01/08/2005 07:11:15",1,1,"Created",,"01/08/2005" "0","01/08/2005 07:12:40",1,1,"Created",,"01/08/2005"... (2 Replies)
Discussion started by: shashi_kiran_v
2 Replies

2. Shell Programming and Scripting

extract data from a data matrix with filter criteria

Here is what old matrix look like, IDs X1 X2 Y1 Y2 10914061 -0.364613333 -0.362922333 0.001691 -0.450094667 10855062 0.845956333 0.860396667 0.014440333 1.483899333... (7 Replies)
Discussion started by: ssshen
7 Replies

3. Shell Programming and Scripting

Append specific lines to a previous line based on sequential search criteria

I'll try explain this as best I can. Let me know if it is not clear. I have large text files that contain data as such: 143593502 09-08-20 09:02:13 xxxxxxxxxxx xxxxxxxxxxx 09-08-20 09:02:11 N line 1 test line 2 test line 3 test 143593503 09-08-20 09:02:13... (3 Replies)
Discussion started by: jesse
3 Replies

4. Shell Programming and Scripting

Delete new lines based on search criteria

Hi all! A bit of background: I am trying to create a script that formats SQL statements. I have gotten so far as to add new lines based on certain match criteria like commas, keywords etc. In the process, I end up adding newlines where I don't want. For example: substr(colName, 1, 10)... (3 Replies)
Discussion started by: jayarkay
3 Replies

5. Shell Programming and Scripting

Search for a specific data in a file based on a date range

Hi, Currently I am working on a script to automate the process of converting the log file from binary into text format. To achieve this, partly I am depending on my application’s utility for this conversion and the rest I am relying on shell commands to search for directory, locate the file and... (5 Replies)
Discussion started by: svajhala
5 Replies

6. Shell Programming and Scripting

Extract error records based on specific criteria from Unix file

Hi, I look for a awk one liner for below issue. input file ABC 1234 abc 12345 ABC 4567 678 XYZ xyz ght 678 ABC 787 yyuu ABC 789 7890 777 zxr hyip hyu mno uii 678 776 ABC ty7 888 All lines should be started with ABC as first field. If a record has another value for 1st... (7 Replies)
Discussion started by: ratheesh2011
7 Replies

7. Shell Programming and Scripting

Need To Delete Lines Based On Search Criteria

Hi All, I have following input file. I wish to retain those lines which match multiple search criteria. The search criteria is stored in a variable seperated from each other by comma(,). SEARCH_CRITERIA = "REJECT, DUPLICATE" Input File: ERROR,MYFILE_20130214_11387,9,37.75... (3 Replies)
Discussion started by: angshuman
3 Replies

8. Shell Programming and Scripting

Copying section of file based on search criteria

Hi Guru's, I am new to unix scripting. I have a huge file with user details in it(file2) and I have another file with a list of users(file1). Script has to search a user from file1 and get all the associated lines from file2. Example: fiel1: cn=abc cn=DEF cn=xyx File 2: dn:... (10 Replies)
Discussion started by: Samingla
10 Replies

9. Shell Programming and Scripting

awk to print specific line in file based on criteria

In the file below I am trying to extract a specific instance of path, if the adjacent plugin": "/rundb/api/v1/plugin/49/. Thank you :). file "path": "/results/analysis/output/Home/Auto_user_S5-00580-4-Medexome_65_028/plugin_out/FileExporter_out.52", "plugin": "/rundb/api/v1/plugin/49/",... (8 Replies)
Discussion started by: cmccabe
8 Replies

10. Shell Programming and Scripting

Need a Linux command for find/replace column based on specific criteria.

I'm new to shell programming, I have a huge text file in the following format, where columns are separated by single space: ACA MEX 4O_ $98.00 $127.40 $166.60 0:00 0:00 0 ; ACA YUL TS_ $300.00 $390.00 $510.00 0:00 0:00 0 ; ACA YYZ TS_ $300.00 $390.00 $510.00 0:00 0:00 0 ; ADZ YUL TS_ $300.00... (3 Replies)
Discussion started by: transat
3 Replies
AUDITREDUCE(1)						    BSD General Commands Manual 					    AUDITREDUCE(1)

NAME
auditreduce -- select records from audit trail files SYNOPSIS
auditreduce [-A] [-a YYYYMMDD[HH[MM[SS]]]] [-b YYYYMMDD[HH[MM[SS]]]] [-c flags] [-d YYYYMMDD] [-e euid] [-f egid] [-g rgid] [-j id] [-m event] [-o object=value] [-r ruid] [-u auid] [-v] [file ...] DESCRIPTION
The auditreduce utility selects records from the audit trail files based on the specified criteria. Matching audit records are printed to the standard output in their raw binary form. If no file argument is specified, the standard input is used by default. Use the praudit(1) utility to print the selected audit records in human-readable form. The options are as follows: -A Select all records. -a YYYYMMDD[HH[MM[SS]]] Select records that occurred after or on the given datetime. -b YYYYMMDD[HH[MM[SS]]] Select records that occurred before the given datetime. -c flags Select records matching the given audit classes specified as a comma separated list of audit flags. See audit_control(5) for a description of audit flags. -d YYYYMMDD Select records that occurred on a given date. This option cannot be used with -a or -b. -e euid Select records with the given effective user ID or name. -f egid Select records with the given effective group ID or name. -g rgid Select records with the given real group ID or name. -j id Select records having a subject token with matching ID, where ID is a process ID. -m event Select records with the given event name or number. This option can be used more then once to select records of multiple event types. See audit_event(5) for a description of audit event names and numbers. -o object=value file Select records containing path tokens, where the pathname matches one of the comma delimited extended regular expression con- tained in given specification. Regular expressions which are prefixed with a tilde ('~') are excluded from the search results. These extended regular expressions are processed from left to right, and a path will either be selected or deslected based on the first match. Since commas are used to delimit the regular expressions, a backslash ('') character should be used to escape the comma if it is a part of the search pattern. msgqid Select records containing the given message queue ID. pid Select records containing the given process ID. semid Select records containing the given semaphore ID. shmid Select records containing the given shared memory ID. -r ruid Select records with the given real user ID or name. -u auid Select records with the given audit ID. -v Invert sense of matching, to select records that do not match. EXAMPLES
To select all records associated with effective user ID root from the audit log /var/audit/20031016184719.20031017122634: auditreduce -e root /var/audit/20031016184719.20031017122634 To select all setlogin(2) events from that log: auditreduce -m AUE_SETLOGIN /var/audit/20031016184719.20031017122634 Output from the above command lines will typically be piped to a new trail file, or via standard output to the praudit(1) command. Select all records containing a path token where the pathname contains /etc/master.passwd: auditreduce -o file="/etc/master.passwd" /var/audit/20031016184719.20031017122634 Select all records containing path tokens, where the pathname is a TTY device: auditreduce -o file="/dev/tty[a-zA-Z][0-9]+" /var/audit/20031016184719.20031017122634 Select all records containing path tokens, where the pathname is a TTY except for /dev/ttyp2: auditreduce -o file="~/dev/ttyp2,/dev/tty[a-zA-Z][0-9]+" /var/audit/20031016184719.20031017122634 SEE ALSO
praudit(1), audit_control(5), audit_event(5) HISTORY
The OpenBSM implementation was created by McAfee Research, the security division of McAfee Inc., under contract to Apple Computer Inc. in 2004. It was subsequently adopted by the TrustedBSD Project as the foundation for the OpenBSM distribution. AUTHORS
This software was created by McAfee Research, the security research division of McAfee, Inc., under contract to Apple Computer Inc. Addi- tional authors include Wayne Salamon, Robert Watson, and SPARTA Inc. The Basic Security Module (BSM) interface to audit records and audit event stream format were defined by Sun Microsystems. BSD
January 24, 2004 BSD
All times are GMT -4. The time now is 04:35 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy