Sponsored Content
Full Discussion: Edit a Huge one line file
Top Forums Shell Programming and Scripting Edit a Huge one line file Post 302655339 by bakunin on Wednesday 13th of June 2012 05:00:39 AM
Old 06-13-2012
Well, at least the problem with "vi" i can explain:

A file cannot be edited in the same place where it is stored. This is the reason why i.e.

Code:
sed '<something>' infile > outfile

works, while

Code:
sed '<something>' infile > infile

will not (the same is true for awk, etc.).

Some (GNU-)versions of sed circumvent this principal limitation by introducing inline-editing ("-i"), which works the same way interactive editors like vi do: they create a copy at program start and only upon saving/finishing the work they copy this over the original file.

vi typically uses /var/tmp per default, but can be configured to use other places too (at least to my knowledge all versions of vi offer such an option via the .exrc file). If this filesystem has not enough free space to hold the copy an attempt to edit the file will fail even if there would be enough space in memory to hold it.

Another limitation is the maximum line length: this is a system limitation and how long lines can grow is laid down in the kernel header file limits.h in the constant "LINE_MAX".

I hope this helps.
This User Gave Thanks to bakunin For This Post:
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Insert a line as the first line into a very huge file

Hello, I need to insert a line (like a header) as the first line of a very huge file (about 3 ml rows). I am able to do it with sed, but redirecting the output and creating a new file takes quite some time. I was wondering if there was a more efficient way of doing it? Any help would be... (3 Replies)
Discussion started by: shriek
3 Replies

2. UNIX for Dummies Questions & Answers

edit each line in the file

I am trying to edit each line in a file. The file has several columns delimitted by '|'. I need to take out the last two columns. Each line starts with a unique word through which I am storing the lines in a variable and cutting the last two colums. But, when I am echoing the line, it is... (2 Replies)
Discussion started by: chiru_h
2 Replies

3. UNIX for Dummies Questions & Answers

How to remove FIRST Line of huge text file on Solaris

i need help..!!!! i have one big text file estimate data file size 50 - 100GB with 70 Mega Rows. on OS SUN Solaris version 8 How i can remove first line of the text file. Please suggest me for solutions. Thank you very much in advance:) (5 Replies)
Discussion started by: madoatz
5 Replies

4. Shell Programming and Scripting

Edit a line in a file with perl

Hi, How can I edit a line in a file? For example, a.txt contains: start: 1 2 3 4 stop: a b c d and I want to change "3" to "9" and to add "5" after "4" the result should be (a.txt): start: 1 9 3 4 5 stop: a b c d Thanks, zed (5 Replies)
Discussion started by: zed
5 Replies

5. Shell Programming and Scripting

Implement in one line sed or awk having no delimiter and file size is huge

I have file which contains around 5000 lines. The lines are fixed legth but having no delimiter.Each line line contains nearly 3000 characters. I want to delete the lines a> if it starts with 1 and if 576th postion is a digit i,e 0-9 or b> if it starts with 0 or 9(i,e header and footer) ... (4 Replies)
Discussion started by: millan
4 Replies

6. Shell Programming and Scripting

How to edit file to have one line entry?

Hello All, My file content is: DROP TABLE "FACT_WORLD"; CREATE TABLE "FACT_WORLD" ( "AR_ID" INTEGER NOT NULL, "ORG_ID" INTEGER NOT NULL ) DATA CAPTURE NONE COMPRESS YES; I want to change this file to have entries in one... (6 Replies)
Discussion started by: akash2508
6 Replies

7. Shell Programming and Scripting

Optimised way for search & replace a value on one line in a very huge file (File Size is 24 GB).

Hi Experts, I had to edit (a particular value) in header line of a very huge file so for that i wanted to search & replace a particular value on a file which was of 24 GB in Size. I managed to do it but it took long time to complete. Can anyone please tell me how can we do it in a optimised... (7 Replies)
Discussion started by: manishkomar007
7 Replies

8. Shell Programming and Scripting

Edit first line of a text file

Hi friends, Issue1: I have a text file with the first line like this #chrom start end Readcount_A Normalized_Readcount_A ReadcountB Normalized_Readcount_B Fc_A_vs_B pvalue_A_vs_B FDR_A_vs_B Fc_B_vs_A pvalue_B_vs_A FDR_B_vs_A <a href="http://unix.com/">Link</a> How can I change it to the... (11 Replies)
Discussion started by: jacobs.smith
11 Replies

9. UNIX for Dummies Questions & Answers

Need to replace new line characters in a huge file

Hi , I would like to replace new line characters(\n) in a huge file of about 2 million records . I tried this one (:%s/\n//g) but it's hanging there and no result. Does this command do not work if the file is big. Please let me know if you have any other options Regards Raj (1 Reply)
Discussion started by: rajeevm
1 Replies

10. Shell Programming and Scripting

Reading ALL BUT the first and last line of a huge file

Hi. Pardon me if I'm posting a duplicate thread but.. I have a text file with over 150 Million records, file size is in the range if MB(close to GB). The requirement is to read ALL the lines excepting the FIRST LINE which is the file header and the LAST LINE which is it's trailer record. ... (8 Replies)
Discussion started by: kumarjt
8 Replies
ICON(1) 						      General Commands Manual							   ICON(1)

NAME
icon - interpret or compile Icon programs SYNOPSIS
icont [ option ... ] file ... [ -x arg ... ] iconc [ option ... ] file ... [ -x arg ... ] DESCRIPTION
icont and iconc each convert an Icon source program into executable form. icont translates quickly and provides interpretive execution. iconc takes longer to compile but produces programs that execute faster. icont and iconc for the most part can be used interchangeably. This manual page describes both icont and iconc. Where there there are differences in usage between icont and iconc, these are noted. File Names: Files whose names end in .icn are assumed to be Icon source files. The .icn suffix may be omitted; if it is not present, it is supplied. The character - can be used to indicate an Icon source file given in standard input. Several source files can be given on the same command line; if so, they are combined to produce a single program. The name of the executable file is the base name of the first input file, formed by deleting the suffix, if present. stdin is used for source programs given in standard input. Processing: As noted in the synopsis above, icont and iconc accept options followed by file names, optionally followed by -x and arguments. If -x is given, the program is executed automatically and any following arguments are passed to it. icont: The processing performed by icont consists of two phases: translation and linking. During translation, each Icon source file is translated into an intermediate language called ucode. Two ucode files are produced for each source file, with base names from the source file and suffixes .u1 and .u2. During linking, the one or more pairs of ucode files are combined to produce a single icode file. The ucode files are deleted after the icode file is created. Processing by icont can be terminated after translation by the -c option. In this case, the ucode files are not deleted. The names of .u1 files from previous translations can be given on the icont command line. These files and the corresponding .u2 files are included in the linking phase after the translation of any source files. The suffix .u can be used in place of .u1; in this case the 1 is supplied auto- matically. Ucode files that are explicitly named are not deleted. iconc: The processing performed by iconc consists of two phases: code generation and compilation and linking. The code generation phase produces C code, consisting of a .c and a .h file, with the base name of the first source file. These files are then compiled and linked to produce an executable binary file. The C files normally are deleted after compilation and linking. Processing by iconc can be terminated after code generation by the -c option. In this case, the C files are not deleted. OPTIONS
The following options are recognized by icont and iconc: -c Stop after producing intermediate files and do not delete them. -e file Redirect standard error output to file. -f s Enable full string invocation. -o name Name the output file name. -s Suppress informative messages. Normally, both informative messages and error messages are sent to standard error output. -t Arrange for &trace to have an initial value of -1 when the program is executed and for iconc enable debugging features. -u Issue warning messages for undeclared identifiers in the program. -v i Set verbosity level of informative messages to i -E Direct the results of preprocessing to standard output and inhibit further processing. The following additional options are recognized by iconc: -f string Enable features as indicated by the letters in string: a all, equivalent to delns d enable debugging features: display(), name(), variable(), error trace back, and the effect of -f n (see below) e enable error conversion l enable large-integer arithmetic n produce code that keeps track of line numbers and file names in the source code s enable full string invocation -n string Disable specific optimizations. These are indicated by the letters in string: a all, equivalent to cest c control flow optimizations other than switch statement optimizations e expand operations in-line when reasonable (keywords are always put in-line) s optimize switch statements associated with operation invocations t type inference -p arg Pass arg on to the C compiler used by iconc -r path Use the run-time system at path, which must end with a slash. -C prg Have iconc use the C compiler given by prg ENVIRONMENT VARIABLES
When an Icon program is executed, several environment variables are examined to determine certain execution parameters. Values in paren- theses are the default values. BLKSIZE (500000) The initial size of the allocated block region, in bytes. COEXPSIZE (2000) The size, in words, of each co-expression block. DBLIST The location of data bases for iconc to search before the standard one. The value of DBLIST should be a blank-separated string of the form p1 p2 ... pn where the pi name directories. ICONCORE If set, a core dump is produced for error termination. ICONX The location of iconx, the executor for icode files, is built into an icode file when it is produced. This location can be overridden by setting the environment variable ICONX. If ICONX is set, its value is used in place of the location built into the icode file. IPATH The location of ucode files specified in link declarations for icont. IPATH is a blank-separated list of directories. The current directory is always searched first, regardless of the value of IPATH. LPATH The location of source files specified in preprocessor $include directives and in link declarations for iconc. LPATH is otherwise sim- ilar to IPATH. MSTKSIZE (10000) The size, in words, of the main interpreter stack for icont. NOERRBUF By default, &errout is buffered. If this variable is set, &errout is not buffered. QLSIZE (5000) The size, in bytes, of the region used for pointers to strings during garbage collection. STRSIZE (500000) The initial size of the string space, in bytes. TRACE The initial value of &trace. If this variable has a value, it overrides the translation-time -t option. FILES
icont Icon translator iconc Icon compiler iconx Icon executor SEE ALSO
The Icon Programming Language, Ralph E. Griswold and Madge T. Griswold, Prentice-Hall Inc., Englewood Cliffs, New Jersey, Second Edition, 1990. Version 9.1 of Icon, Ralph E. Griswold, Clinton L. Jeffery, and Gregg M. Townsend, IPD267, Department of Computer Science, The University of Arizona, 1995. Version 9 of the Icon Compiler, Ralph E. Griswold, IPD237, Department of Computer Science, The University of Arizona, 1995. icon_vt(1) LIMITATIONS AND BUGS
The icode files for the interpreter do not stand alone; the Icon run-time system (iconx) must be present. Stack overflow is checked using a heuristic that is not always effective. 1 November 1995 IPD244b ICON(1)
All times are GMT -4. The time now is 09:52 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy