Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Check valid records in really big file with one commend..

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Check valid records in really big file with one commend..
# 1  
Old 05-18-2008
Check valid records in really big file with one commend..
Hi,
I have a 5 gig file, no record terminators, field terminators are newline. The record length is 768 and I would like to check that every 768th byte is a newline and print out the byte position if it isn't. I would like to do this going either forward or backwards with one command if possible. I read that awk has a 3000 byte limit for a record so maybe a complex grep?

Thanks,

Victor
# 2  
Old 05-18-2008
Many egreps will probably choke on this, but it's worth a try.

Code:
egrep -v '^.{768}$' file

# 3  
Old 05-18-2008
Maybe awk will help.
Code:
awk '{len=length($0); if(len < 768) {print NR, len}}' data.file

# 4  
Old 05-18-2008
thanks for the suggestions, I'll try them tomorrow..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Script to check field value from a file records

I need a script to check the records in a file , if any value match transfer the record in error.txt file. 1- If any of the any field value is NULL(nothing in this field) Record1|Record2|Record3|Record4|Record5|DATE1|DATE2 Example: 11111111|22222222|NULL|12|444|27042018|27042018... (8 Replies)
Discussion started by: vivekn
8 Replies

2. Shell Programming and Scripting

Check for valid hostnames

Hello, I am trying to develop a script to check for valid hostnames. Below are the prerequisites for a valid hostname which I got from wiki : Hostnames are composed of series of labels concatenated with dots, as are all domain names. For example, "en.wikipedia.org" is a hostname. Each label... (8 Replies)
Discussion started by: rahul2662
8 Replies

3. UNIX for Dummies Questions & Answers

How to check if file contains valid strings?

Hi All, I am a newbie...I would like to have a function which ll check if a file contains valid strings before "=" operator. Just to give you my requirement: assume my file has content: hello= gsdgsd sfdsg sgdsg sgdgdg world= gggg hhhh iiiii xxxx= pppp ppppp pppp my... (5 Replies)
Discussion started by: rtagarra
5 Replies

4. Shell Programming and Scripting

Read File and check records for length

I need a script that will run in unix to: 1) Read and input file with 1 column that contains for ex: 0123456789 1234567890 ...etc 2) Checks the first column if it is: a. Numeric from 0 - 9 b. if it is not less... (4 Replies)
Discussion started by: mrn6430
4 Replies

5. UNIX for Dummies Questions & Answers

Delete records from a big file based on some condition

Hi, To load a big file in a table,I have a make sure that all rows in the file has same number of the columns . So in my file if I am getting any rows which have columns not equal to 6 , I need to delete it . Delimiter is space and columns are optionally enclosed by "". This can be ... (1 Reply)
Discussion started by: hemantraijain
1 Replies

6. Shell Programming and Scripting

Extract certain entries from big file:Request to check

Hi all I have a big file which I have attached here. And, I have to fetch certain entries and arrange in 5 columns Name Drug DAP ID disease approved or notIn the attached file data is arranged with tab separated columns in this way: and other data is... (2 Replies)
Discussion started by: manigrover
2 Replies

7. Shell Programming and Scripting

Read a file with n records as one big string using linux

Hello! Is there a way i can read a file with n records as one big string using linux shell script? I have a file in the below format - REC1 REC2 REC3 . . . REC4 Record length is 3000 bytes per record and with a newline char at the end. What i need to do is - read this file as one... (5 Replies)
Discussion started by: mailme0205
5 Replies

8. Shell Programming and Scripting

how to check for valid password

I need to check if an account has a valid password. Would something like this work? read ACCNAME if grep -q "^$ACCNAME:\$6:" /etc/shadow; thenI noticed every entry in my shadow file that has a password starts with $6 ... it works for my current setup, but would it always work? I can't test... (4 Replies)
Discussion started by: ADay2Long
4 Replies

9. Shell Programming and Scripting

how to check if a file has zero records

I want to do conditional logic on whether a file has 0 bytes or more than 0 bytes. what is the easiest way to do this? (2 Replies)
Discussion started by: guessingo
2 Replies

10. Shell Programming and Scripting

Based on num of records in file1 need to check records in file2 to set some condns

Hi All, I have two files say file1 and file2. I want to check the number of records in file1 and if its atleast 2 (i.e., 2 or greater than 2 ) then I have to check records in file2 .If records in file2 is atleast 1 (i.e. if its not empty ) i have to set some conditions . Could you pls... (3 Replies)
Discussion started by: mavesum
3 Replies
Login or Register to Ask a Question