Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Replacing in huge text file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Replacing in huge text file
# 8  
Old 05-29-2011
Code:
awk '{for (i=1;i<=NF-2;i++) printf $i}' infile

# 9  
Old 05-29-2011
Quote:
Originally Posted by rdcwayx
Code:
awk '{for (i=1;i<=NF-2;i++) printf $i}' infile

Coud you explain the role of NF here... I'm kind of noob to awk...
# 10  
Old 05-29-2011
NF is a built-in variable whose value is the number of fields in the current record. awk automatically updates the value of NF each time it reads a record. No matter how many fields there are, the last field in a record can be represented by $NF. So, $NF is the same as $7, which is ‘example.’. If you try to reference a field beyond the last one (such as $8 when the record has only seven fields), you get the empty string. (If used in a numeric operation, you get zero.)

Fields - The GNU Awk User's Guide
# 11  
Old 05-29-2011
wow that was simple.....

Definitely doing it this way....
btw how to search for a particular term in a 15 GB text file ???
and for the creation of such a file how can i check the block size ??

for searching i was thinking of a multithreaded application using fork..??
# 12  
Old 05-30-2011
Code:
grep "KEY" infile
awk '/KEY/' infile

and what's meaning of "block size", the file size?
Code:
ls -l infile

# 13  
Old 05-30-2011
Quote:
Originally Posted by shantanuthatte
I have huge text files (~120 MB)x100 which equivalents to ~11GB of data. The files contain pure numbers, actually the value of "phi" to 10 billion digits!!

I know its huge!! Here are the last few lines of a file
Code:
0952899155 3233967444 3344925499 0276061529 7261968933 9683989044 3317145063 2771963944 5807139825 5785263278 : 999996
7076665287 1341193004 9994291160 2752806087 3098057018 7993954003 8272886989 6031743863 1213075239 5486559526 : 999997
4770078828 1376659981 9345095495 5822463216 7224348351 6200913437 5085852987 6060405404 9200077203 8324752051 : 999998
4334324783 5519682615 3340745027 7486245638 0533805208 0097461685 3057557984 4986386591 3281896020 9655014075 : 999999
6983266465 0958762067 5922249107 5144125222 8226019880 4186130718 6909500836 2519505480 1837059131 8941970031 : 1000000

each line consists of 10x10 digits and at the end the line number. What I want to do is to remove the spaces and the trailing line number and line break. I tried doing that using sed but I keep messing up. I want the output as:
If you have 1 million lines of 'phi' and each line is 100 characters long, that is only 100 million digits of 'phi' not 10 billion
# 14  
Old 05-30-2011
@AlphaLexman, don't forget he has 100 files so 100x100million

I wouldn't store the numbers as ASCII you would be better off storing them as 9 digit long integers. You could then write them to an indexed database for quick searching, along with an ordinal position within pi.

eg:
Code:
314159265,1
141592653,2
415926535,3
159265358,4
...
894197003,99999991
941970031,99999992

Still, with only a 10^10 digit pool, you'll have to be quite lucky to find any given number much longer than a phone number.
Last edited by Chubler_XL; 05-30-2011 at 10:06 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Filter records in a huge text file from a filter text file

Hi Folks, I have a text file with lots of rows with duplicates in the first column, i want to filter out records based on filter columns in a different filter text file. bash scripting is what i need. Data.txt Name OrderID Quantity Sam 123 300 Jay 342 498 Kev 78 2500 Sam 420 50 Vic 10... (3 Replies)
Discussion started by: tech_frk
3 Replies

2. Shell Programming and Scripting

Output only first 400 bytes of a huge text file

How do I output only the first 400 bytes of a huge text file to a new file. It has to be unmodified so no added invisible characters. Many thanks..... (3 Replies)
Discussion started by: garethsays
3 Replies

3. Shell Programming and Scripting

How to open a huge text file?

Hi. I have a 10 Gb text file.the default text editor in ubuntu doens't open it. Does anyone know how can i open it?? Thanks (4 Replies)
Discussion started by: stalaei
4 Replies

4. UNIX for Dummies Questions & Answers

Replacing a column in a text file

Say I had a text file that contained four columns, like the following: Mack Christopher:237 Avondale Blvd:970-791-6419:S Ben Macdonor:30 Dragon Rd:647-288-6395:B I'm making a loop that will replace the fourth column a line in the file with the contents of a variable 'access', but I have no... (6 Replies)
Discussion started by: Sotau
6 Replies

5. Shell Programming and Scripting

Replacing second line from huge files

I'm trying simple functionality of replacing the second line of files with some other string. Problem is these files are huge and there are too many files to process. Could anyone please suggest me a way to replace the second line of all files with another text in a fastest possible manner. ... (2 Replies)
Discussion started by: satish.pyboyina
2 Replies

6. UNIX for Dummies Questions & Answers

Help parsing and replacing text with file name

Hi everyone, I'm having trouble figuring this one out. I have ~100 *.fa files with multiple lines of fasta sequences like this: file1.fa >xyzsequence atcatgcacac...... ataccgagagg..... atataccagag..... >abcsequence atgagatatat..... acacacggd..... atcgaacac.... agttccagat.... The... (2 Replies)
Discussion started by: mycoguy
2 Replies

7. Shell Programming and Scripting

replacing text with contents from another file

I'm trying to change the ramfs size in kernel .config automatically. I have a ramfs_size file generated with du -s cat ramfs_size 64512 I want to replace the linux .config's ramdisk size with the above value CONFIG_BLK_DEV_RAM_SIZE=73728 Right now I'm doing something dumb like: ... (3 Replies)
Discussion started by: amoeba
3 Replies

8. Shell Programming and Scripting

replacing text in a file, but...

Hi all, Very first post on this forums, hope you can help me with this scripting task. I have a big text file with over 3000 lines, some of those lines contain some text that I need to replace, lets say for simplicity the text to be replaced in those lines is "aaa" and I need it to replace it... (2 Replies)
Discussion started by: Angelseph
2 Replies

9. Shell Programming and Scripting

Replacing Text in Text file

Hi Guys, I am needing some help writing a shell script to replace the following in a text file /opt/was/apps/was61 with some other path eg /usr/blan/blah/blah. I know that i can do it using sed or perl but just having difficulty writing the escape characters for it All Help... (3 Replies)
Discussion started by: cgilchrist
3 Replies

10. UNIX for Dummies Questions & Answers

How to remove FIRST Line of huge text file on Solaris

i need help..!!!! i have one big text file estimate data file size 50 - 100GB with 70 Mega Rows. on OS SUN Solaris version 8 How i can remove first line of the text file. Please suggest me for solutions. Thank you very much in advance:) (5 Replies)
Discussion started by: madoatz
5 Replies
Login or Register to Ask a Question