Read a file with n records as one big string using linux


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Read a file with n records as one big string using linux
# 1  
Old 02-03-2012
Read a file with n records as one big string using linux

Hello!
Is there a way i can read a file with n records as one big string using linux shell script? I have a file in the below format -

REC1
REC2
REC3
.
.
.
REC4

Record length is 3000 bytes per record and with a newline char at the end. What i need to do is
- read this file as one big string(char by char) and not line by line.
- truncate every 3001'st char and write a new file.

I tried couple of things with awk and sed but none worked for me.

Appreciate your help!
Thanks

Last edited by mailme0205; 02-03-2012 at 01:43 PM..
# 2  
Old 02-03-2012
Like this?
Code:
tr '\n' ' ' < infile > outfile

--ahamed
# 3  
Old 02-03-2012
process substitution, assuming bash
Code:
#!/bin/bash
var=$(</path/to/file)
echo "$var"

# 4  
Old 02-03-2012
Thanks Ahamed!

Is there any other way. I cannot use just tr to replace the newline chars with space while i read data. My source data also have some newline chars in it in each records.

Also the Source data has spaces all special char, binary data etc...

Is it possible to read file as one big blob and then use a loop somehow to remove every 3001th char?

Thanks
# 5  
Old 02-03-2012
We can go around on this until you get your requirements clear. Basically there is no reasonable way to do this with shell commands -- you need some kind of language: perl, python, C.

Everybody with Linux has gcc by default, compile this to do what you want.
Code:
/* r3000.c read whole file of any character type
  step thru read buffer , write 2999 characters skip 3000th (remove every 3000th character)
  usage: ./r3000 inputfile outputfile
  to compile use gcc:  
      gcc r3000.c -o r3000
      
*/
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/stat.h>

size_t filesize(char *fname)
{
    struct stat st;
    if(stat(fname, &st)==-1)
    {
      perror("File open error on input");
      exit(1);
    }
    return st.st_size;
}
int main(int argc, char **argv)
{
    size_t  len=filesize(argv[1]);
    unsigned char *buf=malloc(len);
    unsigned char *p=buf;
    ssize_t n=0;
    FILE *in=fopen(argv[1], "rb");
    FILE *out=fopen(argv[2], "wb");

    if(out!=NULL && in !=NULL && buf!=NULL)
    {
       n=read(fileno(in), p, len);
       if(n==len)
       {
          int i=0;
          while(i<=len)
          {          
             write(fileno(out), p, 2999);
             i+=3000;
             p+=3000;
          }
       }
       else
       {
           perror("file read error");
           exit(1);
       }
    }
    else
    {
        perror("Fatal error");
        exit(1);
    }
    
    return 0;
}

# 6  
Old 02-03-2012
For limited size this might work:
Code:
awk '{while(s=substr($0,1,3000)){printf "%s",s;sub(substr($0,1,3001),x)}}END{print x}' RS= infile

-edit- it won't work if there are two consecutive newlines in the file...

Last edited by Scrutinizer; 02-03-2012 at 03:11 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Need help for faster file read and grep in big files

I have a very big input file <inputFile1.txt> which has list of mobile no inputFile1.txt 3434343 3434323 0970978 85233 ... around 1 million records i have another file as inputFile2.txt which has some log detail big file inputFile2.txt afjhjdhfkjdhfkd df h8983 3434343 | 3483 | myout1 |... (3 Replies)
Discussion started by: reldb
3 Replies

2. Shell Programming and Scripting

How to read a particular records from a file?

Hi All Can anybody let me know the code to read a particular record from a file For e.g. File Name: File.txt File content: Script_path=/abc/def/script/ File_path=/xyz/data/ Business Date=19990905 SERVER_NAME=Server DATABASE_NAME=Database Login=NewUser Password=NewPassword ... (3 Replies)
Discussion started by: Siddhartha9833
3 Replies

3. Shell Programming and Scripting

Read File and check records for length

I need a script that will run in unix to: 1) Read and input file with 1 column that contains for ex: 0123456789 1234567890 ...etc 2) Checks the first column if it is: a. Numeric from 0 - 9 b. if it is not less... (4 Replies)
Discussion started by: mrn6430
4 Replies

4. Shell Programming and Scripting

How to read records in a file and sort it?

I have a file which has number of pipe delimited records. I am able to read the records....but I want to sort it after reading. i=0 while IFS="|" read -r usrId dataOwn expire email group secProf startDt endDt smhRole RoleCat DataProf SysRole MesgRole SearchProf do print $usrId $dataOwn... (4 Replies)
Discussion started by: harish468
4 Replies

5. UNIX for Dummies Questions & Answers

Delete records from a big file based on some condition

Hi, To load a big file in a table,I have a make sure that all rows in the file has same number of the columns . So in my file if I am getting any rows which have columns not equal to 6 , I need to delete it . Delimiter is space and columns are optionally enclosed by "". This can be ... (1 Reply)
Discussion started by: hemantraijain
1 Replies

6. Shell Programming and Scripting

Print #of lines after search string in a big file

I have a command which prints #lines after and before the search string in the huge file nawk 'c-->0;$0~s{if(b)for(c=b+1;c>1;c--)print r;print;c=a}b{r=$0}' b=0 a=10 s="STRING1" FILE The file is 5 gig big. It works great and prints 10 lines after the lines which contains search string in... (8 Replies)
Discussion started by: prash184u
8 Replies

7. Shell Programming and Scripting

How to read each 2 records from a file

Hi, I have 10000 records in my test.dat file All records are under the following format a,12,45,bn,c a,16,46,bn1,c a,18,47,bn2,c a,12,47,bn3,c a,11,49,bn4,c I have to read each 2 records and assign it into a temp file .Can anybody help me in this? Thanks (3 Replies)
Discussion started by: kavithakuttyk
3 Replies

8. Shell Programming and Scripting

need shell script to read particular records from a file

i am reading an i/p file input.txt as below and want to read all filenames as in highlighted in bold below and put them in a different file output.txt. can someone help me with a shell script to do this? thanks in advance regards brad input.txt --------- START TYPE:OPT INIT_SEQ:01... (8 Replies)
Discussion started by: bradc
8 Replies

9. Shell Programming and Scripting

read records from a file

Hi all, I have a requirement where I need to read records one by one from a file. I have tried this below code: while read mLine do echo 'Line = '${mLine} done < input_file --- But the problem here is im getting the records with removed spaces. --Supposer if the record is like... (3 Replies)
Discussion started by: srilaxmi
3 Replies

10. Shell Programming and Scripting

Check valid records in really big file with one commend..

Hi, I have a 5 gig file, no record terminators, field terminators are newline. The record length is 768 and I would like to check that every 768th byte is a newline and print out the byte position if it isn't. I would like to do this going either forward or backwards with one command if... (3 Replies)
Discussion started by: vtischuk@yahoo.
3 Replies
Login or Register to Ask a Question