Visit The New, Modern Unix Linux Community

Top Forums Shell Programming and Scripting sed and awk not working on a large record file Post 302762945 by Gurkamal83 on Tuesday 29th of January 2013 09:11:54 AM
sed and awk not working on a large record file

Hi All,

I have a very large single record file.

abc;date||bcd;efg|......... pqr;stu||record_count;date

when i do wc -l on this file it gives me "0" records, coz of missing line feed.

my problem is there is an extra pipe that is coming at the end of this record
abc;date||bcd;efg|......... pqr;stu||record_count;date|

I want to take that extra pipe away. But, awk and sed not working for me.

awk - giving me error that record too long.
sed - tried a few flavours of sed and still not working.

I need this issue to be fixed ASAP. Can someone help me?

Last edited by Scrutinizer; 01-29-2013 at 10:33 AM.. Reason: code tags
Test Your Knowledge in Computers #693
Difficulty: Medium
The rights to Unixware, as well as OpenServer, were acquired by UnXis in 2011.
True or False?

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Sed working on lines of small length and not large length

Hi , I have a peculiar case, where my sed command is working on a file which contains lines of small length. sed "s/XYZ:1/XYZ:3/g" abc.txt > xyz.txt when abc.txt contains lines of small length(currently around 80 chars) , this sed command is working fine. when abc.txt contains lines of... (3 Replies)
Discussion started by: thanuman
3 Replies

2. UNIX for Advanced & Expert Users

fatal: record too large

while running a script to process a file , it ended in termination . I able to find from the logs that it ended due to the error : fatal: record too large 20480 . Then I tried lots to fix the issue and left in vain . Can any one help me in finding reason for it ? . If you provide me a solution , I... (1 Reply)
Discussion started by: sakthifire
1 Replies

3. Shell Programming and Scripting

Sed or awk script to remove text / or perform calculations from large CSV files

I have a large CSV files (e.g. 2 million records) and am hoping to do one of two things. I have been trying to use awk and sed but am a newbie and can't figure out how to get it to work. Any help you could offer would be greatly appreciated - I'm stuck trying to remove the colon and wildcards in... (6 Replies)
Discussion started by: metronomadic
6 Replies

4. Shell Programming and Scripting

Updating a line in a large csv file, with sed/awk?

I have an extremely large csv file that I need to search the second field, and upon matches update the last field... I can pull the line with awk.. but apparently you cant use awk to directly update the file? So im curious if I can use sed to do this... The good news is the field I want to... (5 Replies)
Discussion started by: trey85stang
5 Replies

5. Shell Programming and Scripting

how to delete \n in a large file with sed

Hello i have a big file with a specific format and delimiter is "§" : §field1§$field2§$field3§$field4§$field5§$field6§$field§ in this file we have a field which are very long (more than 20000 chars !!!!) so through vi i cant manipulate them. despite this i managed to suppress lines that... (11 Replies)
Discussion started by: ade05fr
11 Replies

6. Shell Programming and Scripting

How to delete 1 record in large file!

Hi All, I'm a newbie here, I'm just wondering on how to delete a single record in a large file in unix. ex. file1.txt is 1000 records nikki1 nikki2 nikki3 what i want to do is delete the nikki2 record in file1.txt. is it possible? Please advise, Thanks, (3 Replies)
Discussion started by: nikki1200
3 Replies

7. Shell Programming and Scripting

SED/AWK to edit/add field values in a record

Hi Experts, I am new to shell scripting. Need some help in doing one task given by the customer. The sample record in a file is as follows: 3538,,,,,,ID,ID1,,,,,,,,,,, It needs to be the following: 3538,,353800,353800,,,ID,ID1,,,,,COLX,,,,,COLY, And i want to modify this record in... (3 Replies)
Discussion started by: sugarcane
3 Replies

8. Shell Programming and Scripting

Split a large file in n records and skip a particular record

Hello All, I have a large file, more than 50,000 lines, and I want to split it in even 5000 records. Which I can do using sed '1d;$d;' <filename> | awk 'NR%5000==1{x="F"++i;}{print > x}'Now I need to add one more condition that is not to break the file at 5000th record if the 5000th record... (20 Replies)
Discussion started by: ibmtech
20 Replies

9. UNIX for Beginners Questions & Answers

sed awk: split a large file to unique file names

Dear Users, Appreciate your help if you could help me with splitting a large file > 1 million lines with sed or awk. below is the text in the file input file.txt scaffold1 928 929 C/T + scaffold1 942 943 G/C + scaffold1 959 960 C/T +... (6 Replies)
Discussion started by: kapr0001
6 Replies

10. UNIX for Advanced & Expert Users

How to split large file with different record delimiter?

Hi, I have received a file which is 20 GB. We would like to split the file into 4 equal parts and process it to avoid memory issues. If the record delimiter is unix new line, I could use split command either with option l or b. The problem is that the line terminator is |##| How to use... (5 Replies)
Discussion started by: Ravi.K
5 Replies

dbase_replace_record - Replaces a record in a database

bool dbase_replace_record (int $dbase_identifier, array $record, int $record_number) DESCRIPTION
Replaces the given record in the database with the given data. PARAMETERS
o $dbase_identifier - The database link identifier, returned by dbase_open(3) or dbase_create(3). o $record - An indexed array of data. The number of items must be equal to the number of fields in the database, otherwise dbase_replace_record(3) will fail. Note If you're using dbase_get_record(3) return value for this parameter, remember to reset the key named deleted. o $record_number - An integer which spans from 1 to the number of records in the database (as returned by dbase_numrecords(3)). RETURN VALUES
Returns TRUE on success or FALSE on failure. EXAMPLES
Example #1 Updating a record in the database <?php // open in read-write mode $db = dbase_open('/tmp/test.dbf', 2); if ($db) { // gets the old row $row = dbase_get_record_with_names($db, 1); // remove the 'deleted' entry unset($row['deleted']); // Update the date field with the current timestamp $row['date'] = date('Ymd'); // Replace the record dbase_replace_record($db, $row, 1); dbase_close($db); } ?> SEE ALSO
dbase_add_record(3), dbase_delete_record(3). PHP Documentation Group DBASE_REPLACE_RECORD(3)

Featured Tech Videos

All times are GMT -4. The time now is 05:16 PM.
Unix & Linux Forums Content Copyright 1993-2020. All Rights Reserved.
Privacy Policy