Sponsored Content
Top Forums Shell Programming and Scripting Read a file with n records as one big string using linux Post 302595593 by jim mcnamara on Friday 3rd of February 2012 01:45:19 PM
Old 02-03-2012
We can go around on this until you get your requirements clear. Basically there is no reasonable way to do this with shell commands -- you need some kind of language: perl, python, C.

Everybody with Linux has gcc by default, compile this to do what you want.
Code:
/* r3000.c read whole file of any character type
  step thru read buffer , write 2999 characters skip 3000th (remove every 3000th character)
  usage: ./r3000 inputfile outputfile
  to compile use gcc:  
      gcc r3000.c -o r3000
      
*/
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/stat.h>

size_t filesize(char *fname)
{
    struct stat st;
    if(stat(fname, &st)==-1)
    {
      perror("File open error on input");
      exit(1);
    }
    return st.st_size;
}
int main(int argc, char **argv)
{
    size_t  len=filesize(argv[1]);
    unsigned char *buf=malloc(len);
    unsigned char *p=buf;
    ssize_t n=0;
    FILE *in=fopen(argv[1], "rb");
    FILE *out=fopen(argv[2], "wb");

    if(out!=NULL && in !=NULL && buf!=NULL)
    {
       n=read(fileno(in), p, len);
       if(n==len)
       {
          int i=0;
          while(i<=len)
          {          
             write(fileno(out), p, 2999);
             i+=3000;
             p+=3000;
          }
       }
       else
       {
           perror("file read error");
           exit(1);
       }
    }
    else
    {
        perror("Fatal error");
        exit(1);
    }
    
    return 0;
}

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Check valid records in really big file with one commend..

Hi, I have a 5 gig file, no record terminators, field terminators are newline. The record length is 768 and I would like to check that every 768th byte is a newline and print out the byte position if it isn't. I would like to do this going either forward or backwards with one command if... (3 Replies)
Discussion started by: vtischuk@yahoo.
3 Replies

2. Shell Programming and Scripting

read records from a file

Hi all, I have a requirement where I need to read records one by one from a file. I have tried this below code: while read mLine do echo 'Line = '${mLine} done < input_file --- But the problem here is im getting the records with removed spaces. --Supposer if the record is like... (3 Replies)
Discussion started by: srilaxmi
3 Replies

3. Shell Programming and Scripting

need shell script to read particular records from a file

i am reading an i/p file input.txt as below and want to read all filenames as in highlighted in bold below and put them in a different file output.txt. can someone help me with a shell script to do this? thanks in advance regards brad input.txt --------- START TYPE:OPT INIT_SEQ:01... (8 Replies)
Discussion started by: bradc
8 Replies

4. Shell Programming and Scripting

How to read each 2 records from a file

Hi, I have 10000 records in my test.dat file All records are under the following format a,12,45,bn,c a,16,46,bn1,c a,18,47,bn2,c a,12,47,bn3,c a,11,49,bn4,c I have to read each 2 records and assign it into a temp file .Can anybody help me in this? Thanks (3 Replies)
Discussion started by: kavithakuttyk
3 Replies

5. Shell Programming and Scripting

Print #of lines after search string in a big file

I have a command which prints #lines after and before the search string in the huge file nawk 'c-->0;$0~s{if(b)for(c=b+1;c>1;c--)print r;print;c=a}b{r=$0}' b=0 a=10 s="STRING1" FILE The file is 5 gig big. It works great and prints 10 lines after the lines which contains search string in... (8 Replies)
Discussion started by: prash184u
8 Replies

6. UNIX for Dummies Questions & Answers

Delete records from a big file based on some condition

Hi, To load a big file in a table,I have a make sure that all rows in the file has same number of the columns . So in my file if I am getting any rows which have columns not equal to 6 , I need to delete it . Delimiter is space and columns are optionally enclosed by "". This can be ... (1 Reply)
Discussion started by: hemantraijain
1 Replies

7. Shell Programming and Scripting

How to read records in a file and sort it?

I have a file which has number of pipe delimited records. I am able to read the records....but I want to sort it after reading. i=0 while IFS="|" read -r usrId dataOwn expire email group secProf startDt endDt smhRole RoleCat DataProf SysRole MesgRole SearchProf do print $usrId $dataOwn... (4 Replies)
Discussion started by: harish468
4 Replies

8. Shell Programming and Scripting

Read File and check records for length

I need a script that will run in unix to: 1) Read and input file with 1 column that contains for ex: 0123456789 1234567890 ...etc 2) Checks the first column if it is: a. Numeric from 0 - 9 b. if it is not less... (4 Replies)
Discussion started by: mrn6430
4 Replies

9. Shell Programming and Scripting

How to read a particular records from a file?

Hi All Can anybody let me know the code to read a particular record from a file For e.g. File Name: File.txt File content: Script_path=/abc/def/script/ File_path=/xyz/data/ Business Date=19990905 SERVER_NAME=Server DATABASE_NAME=Database Login=NewUser Password=NewPassword ... (3 Replies)
Discussion started by: Siddhartha9833
3 Replies

10. UNIX for Advanced & Expert Users

Need help for faster file read and grep in big files

I have a very big input file <inputFile1.txt> which has list of mobile no inputFile1.txt 3434343 3434323 0970978 85233 ... around 1 million records i have another file as inputFile2.txt which has some log detail big file inputFile2.txt afjhjdhfkjdhfkd df h8983 3434343 | 3483 | myout1 |... (3 Replies)
Discussion started by: reldb
3 Replies
SSP(3)							   BSD Library Functions Manual 						    SSP(3)

NAME
ssp -- bounds checked libc functions LIBRARY
Buffer Overflow Protection Library (libssp, -lssp) SYNOPSIS
#include <ssp/stdio.h> int sprintf(char *str, const char *fmt, ...); int vsprintf(char *str, const char *fmt, va_list ap); int snprintf(char *str, size_t len, const char *fmt, ...); int vsnprintf(char *str, size_t len, const char *fmt, va_list ap); char * gets(char *str); char * fgets(char *str, int len, FILE *fp); #include <ssp/string.h> void * memcpy(void *str, const void *ptr, size_t len); void * memmove(void *str, const void *ptr, size_t len); void * memset(void *str, int val, size_t len); char * strcpy(char *str, const char *ptr, size_t len); char * strcat(char *str, const char *ptr, size_t len); char * strncpy(char *str, const char *ptr, size_t len); char * strncat(char *str, const char *ptr, size_t len); #include <ssp/strings.h> void * bcopy(const void *ptr, void *str, size_t len); void * bzero(void *str, size_t len); #include <ssp/unistd.h> ssize_t read(int fd, void *str, size_t len); int readlink(const char * restrict path, char * restrict str, size_t len); int getcwd(char *str, size_t len); DESCRIPTION
When _FORTIFY_SOURCE bounds checking is enabled as described below, the above functions get overwritten to use the __builtin_object_size(3) function to compute the size of str, if known at compile time, and perform bounds check on it in order to avoid data buffer or stack buffer overflows. If an overflow is detected, the routines will call abort(3). To enable these function overrides the following should be added to the gcc(1) command line: ``-I/usr/include/ssp'' to override the standard include files and ``-D_FORTIFY_SOURCE=1'' or ``-D_FORTIFY_SOURCE=2''. If _FORTIFY_SOURCE is set to 1 the code will compute the maximum possible buffer size for str, and if set to 2 it will compute the minimum buffer size. SEE ALSO
gcc(1), __builtin_object_size(3), stdio(3), string(3), security(7) HISTORY
The ssp library appeared NetBSD 4.0. BSD
March 21, 2011 BSD
All times are GMT -4. The time now is 04:46 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy