Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Getting Segmention fault error while searching string in csv file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Getting Segmention fault error while searching string in csv file
# 1  
Old 01-19-2011
Question Getting Segmention fault error while searching string in csv file
I am using grep command to search in a particular file whose size is 11 GB and i am getting Segmentation fault error as an output.

My command and output is as follows:
Code:
[sdpuser@gnnsdp40 test]$ cat new* | grep 8858406465
Segmentation fault

My linux version is as follows:
Code:
[sdpuser@gnnsdp40 test]$ uname -a
Linux gnnsdp40 2.6.18-164.el5 #1 SMP Tue Aug 18 15:51:48 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux

Please guide me how i can parse the complete file for searching string.
I have also used split command of linux which splits the file of 11 GB to 11 files of 1 GB each respectively.
But still getting the same "Segmentation fault" error while using grep.
Please guide me for the above.

Thanks in advance.
Last edited by poweroflinux; 01-19-2011 at 01:53 AM.. Reason: edit
# 2  
Old 01-19-2011
Try this,
sed can process data rather quickly. It can handle large files well.
Code:
sed -n '/<pattern>/p' datafile

# 3  
Old 01-19-2011
Question Reply to Pravin27
Quote:
Originally Posted by pravin27
Try this,
sed can process data rather quickly. It can handle large files well.
Code:
sed -n '/<pattern>/p' datafile

Hi Pravin,

No output received using the above command.
Following shows for your reference:
Code:
[sdpuser@gnnsdp40 test]$ tail -f DCILog_DEBUG.csv.2010-12-21
21 Dec 2010 23:59:59,760 INFO  -  [dci_plugin_InstanceOne#wlng_nt_payment_dci#1.0] ******
21 Dec 2010 23:59:59,760 INFO  -  [dci_plugin_InstanceOne#wlng_nt_payment_dci#1.0] getBalanceWrapper::getBalance:
21 Dec 2010 23:59:59,760 INFO  -  [dci_plugin_InstanceOne#wlng_nt_payment_dci#1.0] Inside parsing Balance Buckets-[2;, 1;, 27;]
21 Dec 2010 23:59:59,760 INFO  -  [dci_plugin_InstanceOne#wlng_nt_payment_dci#1.0] Current Bucket Lookedup  :1;
21 Dec 2010 23:59:59,760 INFO  -  [dci_plugin_InstanceOne#wlng_nt_payment_dci#1.0] Current Bucket Lookedup  :13;
21 Dec 2010 23:59:59,760 INFO  -  [dci_plugin_InstanceOne#wlng_nt_payment_dci#1.0] Current Bucket Lookedup  :20;
21 Dec 2010 23:59:59,760 INFO  -  [dci_plugin_InstanceOne#wlng_nt_payment_dci#1.0] Current Bucket Lookedup  :24;
21 Dec 2010 23:59:59,760 INFO  -  [dci_plugin_InstanceOne#wlng_nt_payment_dci#1.0] Parse Bucket Details-->{1;=20.50, STR_BUCKET_TOTAL=20.50}
21 Dec 2010 23:59:59,760 INFO  -  [dci_plugin_InstanceOne#wlng_nt_payment_dci#1.0] Bucket details after summation : 1;20091003;20110619;20.50;|13;20101210;20110109;102400.00;|20;20101210;20110109;99854.00;|24;20101208;20110107;334.00;|
21 Dec 2010 23:59:59,760 INFO  -  [dci_plugin_InstanceOne#wlng_nt_payment_dci#1.0] the balance bucket is : 1;20091003;20110619;20.50;|13;20101210;20110109;102400.00;|20;20101210;20110109;99854.00;|24;20101208;20110107;334.00;|

[sdpuser@gnnsdp40 test]$
[sdpuser@gnnsdp40 test]$ sed -n '/getBalanceWrapper/p' DCILog_DEBUG.csv.2010-12-21
[sdpuser@gnnsdp40 test]$ sed -n '/Parse/p' DCILog_DEBUG.csv.2010-12-21
[sdpuser@gnnsdp40 test]$ sed -n '/8858406465/p' DCILog_DEBUG.csv.2010-12-21

# 4  
Old 01-19-2011
I have created file of 1.5 GB with your data and some other information(junk data) and you can see the result below.

Code:
# du -h DCILog_DEBUG.csv.2010-12-21
1.5G    DCILog_DEBUG.csv.2010-12-21
# sed -n '/getBalanceWrapper/p' DCILog_DEBUG.csv.2010-12-21
21 Dec 2010 23:59:59,760 INFO  -  [dci_plugin_InstanceOne#wlng_nt_payment_dci#1.0] getBalanceWrapper::getBalance:
21 Dec 2010 23:59:59,760 INFO  -  [dci_plugin_InstanceOne#wlng_nt_payment_dci#1.0] getBalanceWrapper::getBalance:
21 Dec 2010 23:59:59,760 INFO  -  [dci_plugin_InstanceOne#wlng_nt_payment_dci#1.0] getBalanceWrapper::getBalance:
21 Dec 2010 23:59:59,760 INFO  -  [dci_plugin_InstanceOne#wlng_nt_payment_dci#1.0] getBalanceWrapper::getBalance:

# 5  
Old 01-19-2011
Question Reply to Pravin27
Quote:
Originally Posted by pravin27
I have created file of 1.5 GB with your data and some other information(junk data) and you can see the result below.

Code:
# du -h DCILog_DEBUG.csv.2010-12-21
1.5G    DCILog_DEBUG.csv.2010-12-21
# sed -n '/getBalanceWrapper/p' DCILog_DEBUG.csv.2010-12-21
21 Dec 2010 23:59:59,760 INFO  -  [dci_plugin_InstanceOne#wlng_nt_payment_dci#1.0] getBalanceWrapper::getBalance:
21 Dec 2010 23:59:59,760 INFO  -  [dci_plugin_InstanceOne#wlng_nt_payment_dci#1.0] getBalanceWrapper::getBalance:
21 Dec 2010 23:59:59,760 INFO  -  [dci_plugin_InstanceOne#wlng_nt_payment_dci#1.0] getBalanceWrapper::getBalance:
21 Dec 2010 23:59:59,760 INFO  -  [dci_plugin_InstanceOne#wlng_nt_payment_dci#1.0] getBalanceWrapper::getBalance:


See my output below:
File size is:
[sdpuser@gnnsdp40 test]$ ll -h DCILog_DEBUG.csv.2010-12-21
-rw-r--r-- 1 sdpuser sdpadmin 11G Dec 22 13:07 DCILog_DEBUG.csv.2010-12-21
Code:
[sdpuser@gnnsdp40 test]$ ll
total 21776828
-rw-r--r-- 1 sdpuser sdpadmin 11138823326 Dec 22 13:07 DCILog_DEBUG.csv.2010-12-21
-rw-r--r-- 1 sdpuser sdpadmin  1073741824 Jan 19 11:37 newaa
-rw-r--r-- 1 sdpuser sdpadmin  1073741824 Jan 19 11:37 newab
-rw-r--r-- 1 sdpuser sdpadmin  1073741824 Jan 19 11:37 newac
-rw-r--r-- 1 sdpuser sdpadmin  1073741824 Jan 19 11:37 newad
-rw-r--r-- 1 sdpuser sdpadmin  1073741824 Jan 19 11:37 newae
-rw-r--r-- 1 sdpuser sdpadmin  1073741824 Jan 19 11:37 newaf
-rw-r--r-- 1 sdpuser sdpadmin  1073741824 Jan 19 11:37 newag
-rw-r--r-- 1 sdpuser sdpadmin  1073741824 Jan 19 11:37 newah
-rw-r--r-- 1 sdpuser sdpadmin  1073741824 Jan 19 11:37 newai
-rw-r--r-- 1 sdpuser sdpadmin  1073741824 Jan 19 11:37 newaj
-rw-r--r-- 1 sdpuser sdpadmin   401405086 Jan 19 11:37 newak
[sdpuser@gnnsdp40 test]$ sed -n '/getBalanceWrapper/p' DCILog_DEBUG.csv.2010-12-21
[sdpuser@gnnsdp40 test]$

# 6  
Old 01-19-2011
Can you try to do pattern search on files which you have created after split and let us know the result. May be Experts on this forum can provide you solution which meets your requirement.

Try this,
104857600 =100 MB, You can make it 1 GB. Hope this will help you
Code:
split -b 104857600  DCILog_DEBUG.csv.2010-12-21 new_ ; for i in "new_*"; do sed -n '/getBalanceWrapper/p' $i; done

Last edited by pravin27; 01-19-2011 at 03:43 AM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Removing string from CSV file by provide removal string from other file

What I need is to remove the text from Location_file.txt from each line matching all entries from Remove_location.txt Location_file.txt FlowPrePaid, h3nmg1cm2,Jamaica_MTAImageFileFlowPrePaid,h0nmg1cm1, Flow_BeatTest,FlowRockTest FlowNewTest,FlowNewTest,h0nmg1cm1 PartiallySubscribed,... (3 Replies)
Discussion started by: ketanraut
3 Replies

2. Shell Programming and Scripting

Searching a string in a particular file name

Hello, I have a file name like FIRST_DPF_DAILY_CUST_0826152322.txt i need to extract the string after the third "_" underscore upto timestamp ends i.e CUST_0826152322 can anyone help me with the code Thank you! Regards Srikanth Sagi (3 Replies)
Discussion started by: srikanth_sagi
3 Replies

3. Shell Programming and Scripting

Searching a string stored in other file

I need to design a script which can do the following: I have two files abc.txt and constant.hmtl abc.txt contains some 5 string that I need to see if they exist in contants.html. Constants.hmtl is a very large file around 800 lines. I want to search all the strings present in file... (8 Replies)
Discussion started by: hemasid
8 Replies

4. Shell Programming and Scripting

Searching for a specific string in a file

Hi I am trying to search for a certain set of patterns within a file, and then perform other commands based on output. testfile contents: password requisite pam_cracklib.so lcredit=-1 ucredit=-1 ocredit=-1 script: D="dcredit=-1" if then echo $D exists else echo $D doesnt... (8 Replies)
Discussion started by: bludhemn
8 Replies

5. Shell Programming and Scripting

searching each file for a string

Hi Guys... I want to search for each file that contains a particular string. e.g find . -print | xargs grep -i string_name Now my issue is the files that I search in are gzipped. Will I be able to find the string, using the above commands, even if the files are gzipped? Please... (2 Replies)
Discussion started by: Phuti
2 Replies

6. Shell Programming and Scripting

Searching a string in a file

Hi, I am new to unix shell scripting. I have a requirement. Could anyone help me writing the script for the same? Here goes the requirement: I have a config file let's say temp.config. Here is the data in the config file temp.config : ------------- name=victor age=42 state=texas... (5 Replies)
Discussion started by: badrimohanty
5 Replies

7. UNIX for Dummies Questions & Answers

searching for a string in a file

I need to search for a specific string in a file and if this string exist I need to replace it with something else. I am not sure how I could do this, using an if statement. (2 Replies)
Discussion started by: ROOZ
2 Replies

8. Shell Programming and Scripting

Extracting a string from one file and searching the same string in other files

Hi, Need to extract a string from one file and search the same in other files. Ex: I have file1 of hundred lines with no delimiters not even space. I have 3 more files. I should get 1 to 10 characters say substring from each line of file1 and search that string in rest of the files and get... (1 Reply)
Discussion started by: mohancrr
1 Replies

9. Shell Programming and Scripting

Perl: searching for a string in a file...

Hi All, I need to search for a string in a file that I've opened and base a decision on the result. The logic is this: "if the word 'Shared' appears on the first line then do this on the whole file else do this on the whole file " The code I currently have isn't working:... (4 Replies)
Discussion started by: pondlife
4 Replies

10. UNIX for Dummies Questions & Answers

searching for a string though file system

Is there a way to search an entire file system for the occurance of a string..... other than grep. I have a large directory structure and I'm not certain that grep <string> */*/*/*... is all that effective - especially as I can't be sure of the number of levels to go down without heaps of... (3 Replies)
Discussion started by: peter.herlihy
3 Replies
Login or Register to Ask a Question