Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: sed and awk not working on a large record file
Top Forums Shell Programming and Scripting sed and awk not working on a large record file Post 302762945 by Gurkamal83 on Tuesday 29th of January 2013 09:11:54 AM
Old 01-29-2013
sed and awk not working on a large record file
Hi All,

I have a very large single record file.

Code:
abc;date||bcd;efg|......... pqr;stu||record_count;date

when i do wc -l on this file it gives me "0" records, coz of missing line feed.

my problem is there is an extra pipe that is coming at the end of this record
like
Code:
abc;date||bcd;efg|......... pqr;stu||record_count;date|

I want to take that extra pipe away. But, awk and sed not working for me.

awk - giving me error that record too long.
sed - tried a few flavours of sed and still not working.

I need this issue to be fixed ASAP. Can someone help me?
Last edited by Scrutinizer; 01-29-2013 at 10:33 AM.. Reason: code tags
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Sed working on lines of small length and not large length

Hi , I have a peculiar case, where my sed command is working on a file which contains lines of small length. sed "s/XYZ:1/XYZ:3/g" abc.txt > xyz.txt when abc.txt contains lines of small length(currently around 80 chars) , this sed command is working fine. when abc.txt contains lines of... (3 Replies)
Discussion started by: thanuman
3 Replies

2. UNIX for Advanced & Expert Users

fatal: record too large

while running a script to process a file , it ended in termination . I able to find from the logs that it ended due to the error : fatal: record too large 20480 . Then I tried lots to fix the issue and left in vain . Can any one help me in finding reason for it ? . If you provide me a solution , I... (1 Reply)
Discussion started by: sakthifire
1 Replies

3. Shell Programming and Scripting

Sed or awk script to remove text / or perform calculations from large CSV files

I have a large CSV files (e.g. 2 million records) and am hoping to do one of two things. I have been trying to use awk and sed but am a newbie and can't figure out how to get it to work. Any help you could offer would be greatly appreciated - I'm stuck trying to remove the colon and wildcards in... (6 Replies)
Discussion started by: metronomadic
6 Replies

4. Shell Programming and Scripting

Updating a line in a large csv file, with sed/awk?

I have an extremely large csv file that I need to search the second field, and upon matches update the last field... I can pull the line with awk.. but apparently you cant use awk to directly update the file? So im curious if I can use sed to do this... The good news is the field I want to... (5 Replies)
Discussion started by: trey85stang
5 Replies

5. Shell Programming and Scripting

how to delete \n in a large file with sed

Hello i have a big file with a specific format and delimiter is "§" : §field1§$field2§$field3§$field4§$field5§$field6§$field§ in this file we have a field which are very long (more than 20000 chars !!!!) so through vi i cant manipulate them. despite this i managed to suppress lines that... (11 Replies)
Discussion started by: ade05fr
11 Replies

6. Shell Programming and Scripting

How to delete 1 record in large file!

Hi All, I'm a newbie here, I'm just wondering on how to delete a single record in a large file in unix. ex. file1.txt is 1000 records nikki1 nikki2 nikki3 what i want to do is delete the nikki2 record in file1.txt. is it possible? Please advise, Thanks, (3 Replies)
Discussion started by: nikki1200
3 Replies

7. Shell Programming and Scripting

SED/AWK to edit/add field values in a record

Hi Experts, I am new to shell scripting. Need some help in doing one task given by the customer. The sample record in a file is as follows: 3538,,,,,,ID,ID1,,,,,,,,,,, It needs to be the following: 3538,,353800,353800,,,ID,ID1,,,,,COLX,,,,,COLY, And i want to modify this record in... (3 Replies)
Discussion started by: sugarcane
3 Replies

8. Shell Programming and Scripting

Split a large file in n records and skip a particular record

Hello All, I have a large file, more than 50,000 lines, and I want to split it in even 5000 records. Which I can do using sed '1d;$d;' <filename> | awk 'NR%5000==1{x="F"++i;}{print > x}'Now I need to add one more condition that is not to break the file at 5000th record if the 5000th record... (20 Replies)
Discussion started by: ibmtech
20 Replies

9. UNIX for Beginners Questions & Answers

sed awk: split a large file to unique file names

Dear Users, Appreciate your help if you could help me with splitting a large file > 1 million lines with sed or awk. below is the text in the file input file.txt scaffold1 928 929 C/T + scaffold1 942 943 G/C + scaffold1 959 960 C/T +... (6 Replies)
Discussion started by: kapr0001
6 Replies

10. UNIX for Advanced & Expert Users

How to split large file with different record delimiter?

Hi, I have received a file which is 20 GB. We would like to split the file into 4 equal parts and process it to avoid memory issues. If the record delimiter is unix new line, I could use split command either with option l or b. The problem is that the line terminator is |##| How to use... (5 Replies)
Discussion started by: Ravi.K
5 Replies

LEARN ABOUT OPENSOLARIS

fwtmp

fwtmp(1M)						  System Administration Commands						 fwtmp(1M)

NAME

       fwtmp, wtmpfix - manipulate connect accounting records

SYNOPSIS

       /usr/lib/acct/fwtmp [-ic]

       /usr/lib/acct/wtmpfix [file]...

DESCRIPTION

       fwtmp  reads  from  the	standard input and writes to the standard output, converting binary records of the type found in /var/adm/wtmpx to
       formatted ASCII records. The ASCII version is useful when it is necessary to edit  bad records.

       wtmpfix examines the standard input or named files in utmpx format, corrects the time/date stamps  to  make  the  entries  consistent,  and
       writes  to  the standard output. A hyphen (-) can be used in place of file to indicate the standard input. If time/date corrections are not
       performed, acctcon(1M) will fault when it encounters certain date-change records.

       Each time the date is set, a pair of date change records are written to /var/adm/wtmpx. The first record is the old  date  denoted  by  the
       string  "old time" placed in the line field and the flag OLD_TIME placed in the type field of the utmpx structure. The second record speci-
       fies the new date and is denoted by the string new time placed in the line field and the flag NEW_TIME placed in the  type  field.  wtmpfix
       uses these records to synchronize all time stamps in the file.

       In addition to correcting time/date stamps, wtmpfix will check the validity of the name field to ensure that it consists solely of alphanu-
       meric characters or spaces. If it encounters a name that is considered invalid, it will change the login name to INVALID and write a  diag-
       nostic to the standard error. In this way, wtmpfix reduces the chance that acctcon will fail when processing connect accounting records.

OPTIONS

       -ic    Denotes that input is in ASCII form, and output is to be written in binary form.

FILES

       /var/adm/wtmpx	 history of user access and administration information

ATTRIBUTES

       See attributes(5) for descriptions of the following attributes:

       +-----------------------------+-----------------------------+
       |      ATTRIBUTE TYPE	     |	    ATTRIBUTE VALUE	   |
       +-----------------------------+-----------------------------+
       |Availability		     |SUNWaccu			   |
       +-----------------------------+-----------------------------+

SEE ALSO

       acctcom(1),  ed(1),  acct(1M),  acctcms(1M),  acctcon(1M),  acctmerg(1M),  acctprc(1M),	acctsh(1M),  runacct(1M),  acct(2), acct.h(3HEAD),
       utmpx(4), attributes(5)

SunOS 5.11							    22 Feb 1999 							 fwtmp(1M)
All times are GMT -4. The time now is 09:27 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy