Read a large file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Read a large file
# 1  
Old 03-14-2018
Read a large file

Dear Experts,

Would like to know if there's a way if we can read a large file(like 1-2GB) using a shell command or a script and that can show the output as well of the contents of the file in the CLI?
Will appreciate your response. Thank you
# 2  
Old 03-14-2018
There are hundreds of ways to read a text file in a shell. The definition of a text file varies from operating system to operating system. The ways to read a text file vary from shell to shell.

What operating system are you using?

What shell are you using?

Is the file you want to read a text file? If it contains binary data or one or more lines containing more than LINE_MAX bytes, your options for reading the file and displaying its contents in a readable form will be much more limited.

Of course, you could write your own CLI in C or C++ and make it read any type of file and print it in whatever format you desire.
# 3  
Old 03-14-2018
The file is text file containing texts separated by new lines but only thing is file size is around more than a GIG, the OS is Red hat linux and shell is Bash, please do let us know how to approach on this?
# 4  
Old 03-14-2018
If you want to display it on your screen with a new line from the file being displayed every second, try something like:
Code:
while read -r line
do	printf '%s\n' "$line"
	sleep 1
done < filename

If you're a very fast reader with nimble fingers and you want to print the file to your terminal as quickly as possible stopping once-in-a-while when you see something interesting going by, use:
Code:
cat filename

and while it is printing use ctl-s (i.e. hit the control key and the s key at the same time) to temporarily stop the output and ctl-q to restart the output.

If you want to display the output a page at a time, use:
Code:
less filename

or:
Code:
more filename

If you want to skip around in the file searching for certain patterns and you're used to vi editing commands, use:
Code:
view filename

(note that view is a link to vi that opens its file operands read-only instead of read-write).

If you're looking for certain patterns in your file and only want to see lines that contain that pattern, use grep.

If you're trying to do something else, use or write a utility, that does what you want... ... ...
This User Gave Thanks to Don Cragun For This Post:
# 5  
Old 03-14-2018
Appreciate and thanks very much Don, let me try those suggestions.
# 6  
Old 03-14-2018
Could you please explicate what would be meant by
Quote:
Originally Posted by VKIRUPHAKARAN
. . . show the output as well of the contents of the file in the CLI? . . .
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

sed awk: split a large file to unique file names

Dear Users, Appreciate your help if you could help me with splitting a large file > 1 million lines with sed or awk. below is the text in the file input file.txt scaffold1 928 929 C/T + scaffold1 942 943 G/C + scaffold1 959 960 C/T +... (6 Replies)
Discussion started by: kapr0001
6 Replies

2. UNIX for Dummies Questions & Answers

When reading a csv file, counter to read 20 lines and wait for minute then read next 20 till end

Hello All, i am a newbie and need some help when reading a csv file in a bourne shell script. I want to read 10 lines, then wait for a minute and then do a reading of another 10 lines and so on in the same way. I want to do this till the end of file. Any inputs are appreciated ... (3 Replies)
Discussion started by: victor.s
3 Replies

3. Shell Programming and Scripting

Script to search a large file with a list of terms in another file

Hi- I am trying to search a large file with a number of different search terms that are listed one per line in 3 different files. Most importantly I need to be able to do a case insensitive search. I have tried just using egrep -f but it doesn't seam to be able to handle the -i option when... (3 Replies)
Discussion started by: dougzilla
3 Replies

4. Shell Programming and Scripting

PHP read large string & split in multidimensional arrays & assign fieldnames & write into MYSQL

Hi, I hope the title does not scare people to look into this thread but it describes roughly what I'm trying to do. I need a solution in PHP. I'm a programming beginner, so it might be that the approach to solve this, might be easier to solve with an other approach of someone else, so if you... (0 Replies)
Discussion started by: lowmaster
0 Replies

5. Programming

Cannot read a file with read(fd, buffer, buffersize) function

# include <stdio.h> # include <fcntl.h> # include <stdlib.h> # include <sys/stat.h> int main(int argc, char *argv) { int fRead, fPadded, padVal; int btRead; int BUFFSIZE = 512; char buff; if (argc != 4) { printf ("Please provide all of the... (3 Replies)
Discussion started by: naranja18she
3 Replies

6. Shell Programming and Scripting

Performance issue in UNIX while generating .dat file from large text file

Hello Gurus, We are facing some performance issue in UNIX. If someone had faced such kind of issue in past please provide your suggestions on this . Problem Definition: /Few of load processes of our Finance Application are facing issue in UNIX when they uses a shell script having below... (19 Replies)
Discussion started by: KRAMA
19 Replies

7. Programming

Read/Write a fairly large amount of data to a file as fast as possible

Hi, I'm trying to figure out the best solution to the following problem, and I'm not yet that much experienced like you. :-) Basically I have to read a fairly large file, composed of "messages" , in order to display all of them through an user interface (made with QT). The messages that... (3 Replies)
Discussion started by: emitrax
3 Replies

8. UNIX for Advanced & Expert Users

Shell script failing to read large Xml record-urgent critical help

Hi All, I have shell script running on AIX 5.3 box. It has 7 to 8 "sed" commands piped(|) together. It has a an Xml file as its input which has many records internally. There are certain record which which have more than hundered tags.The script is taking a huge amount of time more than 1.5 hrs... (10 Replies)
Discussion started by: aixjadoo
10 Replies

9. Shell Programming and Scripting

Split large file and add header and footer to each file

I have one large file, after every 200 line i have to split the file and the add header and footer to each small file? It is possible to add different header and footer to each file? (1 Reply)
Discussion started by: ashish4422
1 Replies

10. Filesystems, Disks and Memory

Strange difference in file size when copying LARGE file..

Hi, Im trying to take a database backup. one of the files is 26 GB. I am using cp -pr to create a backup copy of the database. after the copying is complete, if i do du -hrs on the folders i saw a difference of 2GB. The weird fact is that the BACKUP folder was 2 GB more than the original one! ... (1 Reply)
Discussion started by: 0ktalmagik
1 Replies
Login or Register to Ask a Question