Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Fetching record based on Uniq Key from huge file.

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Fetching record based on Uniq Key from huge file.
# 1  
Old 10-18-2012
Fetching record based on Uniq Key from huge file.
Hi i want to fetch 100k record from a file which is looking like as below.

Code:
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
                              EMDS/CREDIT SERVICES                     RunDate: 10/25/2011                    REC#      1
             SEQUENCE NUMBER: 06000000001           L90 KEY :060882D85800E201684C0000            CID KEY:000000000000000000
DEBT RATIO=  0%  INQ W/I 6=  0, W/I 12=  0, W/I 18=  0, W/I 24=  0 ,PRM W/I 6=0; Trades RPTed W/I 24 =  0, No. of Trades =   3
*** FACT FRAUD CODE =  ,FACT ALERT DATA CODE = 
XXXXXX                    XXXXX                                                                   SSN-XXX-XX-XXXX C
XXXXXX,XXXXX.                                                                            FAD:  0/00/0000  FILE SINCE:  0/00/0000
XXXXX XXXXXXXX XX  XXXXXXXCA,92236
,,,,,,,,0
,,,,,,,,0
,,,,,,,,0

Now i want to fetch based on SEQUENCE NUMBER: 06000000001 which is uniq for whole files.

can any one help me?
thanks in advance.
prashant
Last edited by Scrutinizer; 10-18-2012 at 06:44 AM.. Reason: code tags
# 2  
Old 10-18-2012
What is the expected output?
Please use code tags for code and data samples.
# 3  
Old 10-18-2012
output will be 100k records.

The one which i pinged is one record and i want to fetch 100k record from a file having 1000k file.


Code:
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
                              EMDS/CREDIT SERVICES                     RunDate: 10/25/2011                    REC#      1
             SEQUENCE NUMBER: 06000000001           L90 KEY :060882D85800E201684C0000            CID KEY:000000000000000000
DEBT RATIO=  0%  INQ W/I 6=  0, W/I 12=  0, W/I 18=  0, W/I 24=  0 ,PRM W/I 6=0; Trades RPTed W/I 24 =  0, No. of Trades =   3
*** FACT FRAUD CODE =  ,FACT ALERT DATA CODE = 
XXXXXX                    XXXXX                                                                   SSN-XXX-XX-XXXX C
XXXXXX,XXXXX.                                                                            FAD:  0/00/0000  FILE SINCE:  0/00/0000
XXXXX XXXXXXXX XX  XXXXXXXCA,92236
,,,,,,,,0
,,,,,,,,0
,,,,,,,,0
 
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
                              EMDS/CREDIT SERVICES                     RunDate: 10/25/2011                    REC#      2
             SEQUENCE NUMBER: 06000000002           L90 KEY :060882D85800E201684C0000            CID KEY:000000000000000000
DEBT RATIO=  0%  INQ W/I 6=  0, W/I 12=  0, W/I 18=  0, W/I 24=  0 ,PRM W/I 6=0; Trades RPTed W/I 24 =  0, No. of Trades =   3
*** FACT FRAUD CODE =  ,FACT ALERT DATA CODE = 
XXXXXX                    XXXXX                                                                   SSN-XXX-XX-XXXX C
XXXXXX,XXXXX.                                                                            FAD:  0/00/0000  FILE SINCE:  0/00/0000
XXXXX XXXXXXXX XX  XXXXXXXCA,92236
,,,,,,,,0
,,,,,,,,0
,,,,,,,,0

like 100k.

thanks,
prashant
Last edited by Scrutinizer; 10-18-2012 at 06:45 AM.. Reason: code tags
# 4  
Old 10-18-2012
OK. So, I'll assume things and give you a solution:
Code:
awk '/SEQUENCE NUMBER: 06000000001/' RS= file

This is for fetching 1 record. You haven't mentioned how (criterion) you need to fetch the 100k records from the 1000k-record file.
# 5  
Old 10-18-2012
i tried that but i want to fetch 100k record which is uniq for this file as i told you the sequnce can be.
SEQUENCE NUMBER: 06000000001
SEQUENCE NUMBER: 06000000002
SEQUENCE NUMBER: 06000000003
.
.
SEQUENCE NUMBER: 06000100000
.
.
SEQUENCE NUMBER: 06010000000
hope you are geting what i want to tell.
out of all this records i want from
SEQUENCE NUMBER: 06000000001 to SEQUENCE NUMBER: 06000100000.
# 6  
Old 10-18-2012
Code:
awk '$2+0>=6000000001 && $2+0<=6000100000' FS='SEQUENCE NUMBER: ' RS= file

By the way, you could've avoided 2 posts if you'd given your requirements clearly in the first post. And then, you're still not clear; I've still assumed things.
Last edited by elixir_sinari; 10-18-2012 at 06:45 AM..
# 7  
Old 10-18-2012
Code:
awk '/SEQUENCE NUMBER/{if($3<=06000100000){print $1,$2,$3;}else{exit;}}' file
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

EBCDIC File Split Based On Record Key

I was wondering if anyone could explain to me how to split a variable length EBCDIC file into seperate files based on the record key. I have the COBOL layout, and so I need to split the file into 13 different EBCDIC files so that I can run each one through a C++ converter I have, and get the... (11 Replies)
Discussion started by: hanshot1stx
11 Replies

2. Shell Programming and Scripting

Fetching values in CSV file based on column name

input.csv: Field1,Field2,Field3,Field4,Field4 abc ,123 ,xyz ,000 ,pqr mno ,123 ,dfr ,111 ,bbb output: Field2,Field4 123 ,000 123 ,111 how to fetch the values of Field4 where Field2='123' I don't want to fetch the values based on column position. Instead want to... (10 Replies)
Discussion started by: bharathbangalor
10 Replies

3. UNIX for Dummies Questions & Answers

Split a huge 7 GB File Based on Pattern into 4 files

Hi, I have a Huge 7 GB file which has around 1 million records, i want to split this file into 4 files to contain around 250k messages each. Please help me as Split command cannot work here as it might miss tags.. Format of the file is as below <!--###### ###### START-->... (6 Replies)
Discussion started by: KishM
6 Replies

4. Shell Programming and Scripting

Removing Dupes from huge file- awk/perl/uniq

Hi, I have the following command in place nawk -F, '!a++' file > file.uniq It has been working perfectly as per requirements, by removing duplicates by taking into consideration only first 3 fields. Recently it has started giving below error: bash-3.2$ nawk -F, '!a++'... (17 Replies)
Discussion started by: makn
17 Replies

5. Shell Programming and Scripting

Need help splitting huge single record file

I was given a data file that I need to split into multiple lines/records based on a key word. The problem is that it is 2.5GB or bigger and everything I try in perl or sed causes a Segmentation fault. Can someone give me some other ideas. The data is of the form:... (5 Replies)
Discussion started by: leolson
5 Replies

6. Shell Programming and Scripting

Keep the last uniq record only

Hi folks, Below is the content of a file 'tmp.dat', and I want to keep the uniq record (key by first column). However, the uniq record should be the last record. 302293022|2|744124889|744124889 302293022|3|744124889|744124889 302293022|4|744124889|744124889 302293022|5|744124889|744124889... (4 Replies)
Discussion started by: ChicagoBlues
4 Replies

7. Shell Programming and Scripting

filter the uniq record problem

Anyone can help for filter the uniq record for below example? Thank you very much Input file 20090503011111|test|abc 20090503011112|tet1|abc|def 20090503011112|test1|bcd|def 20090503011131|abc|abc 20090503011131|bbc|bcd 20090503011152|bcd|abc 20090503011151|abc|abc... (8 Replies)
Discussion started by: bleach8578
8 Replies

8. Shell Programming and Scripting

Logic for file fetching based on date

Dear friends, I receive the following files into a FTP location on a daily basis -rw-r----- 1 guest ftp1 5021 Aug 19 09:03 CHECK_TEST_Extracts_20080818210000.zip -rw-r----- 1 guest ftp1 2437 Aug 20 05:15 CHECK_TEST_Extracts_20080819210000.zip -rw-r----- 1 guest ... (2 Replies)
Discussion started by: sureshg_sampat
2 Replies

9. Shell Programming and Scripting

Compare 2 huge files wrt to a key using awk

Hi Folks, I need to compare two very huge file ( i.e the files would contain a minimum of 70k records each) using awk or sed. The comparison needs to be done with respect to a 'key'. For example : File1 ********** 1234|TONY|Y75634|20/07/2008 1235|TINA|XCVB56|30/07/2009... (13 Replies)
Discussion started by: Ranjani
13 Replies

10. UNIX for Dummies Questions & Answers

Parsing out records from one huge record

Hi, I have one huge record and know that each record in the file is 550 bytes long. How do I parse out individual records from the single huge record. Thanks, (4 Replies)
Discussion started by: bwrynz1
4 Replies
Login or Register to Ask a Question