Sponsored Content
Top Forums Shell Programming and Scripting Updating a line in a large csv file, with sed/awk? Post 302354229 by trey85stang on Thursday 17th of September 2009 02:02:33 PM
Old 09-17-2009
Updating a line in a large csv file, with sed/awk?

I have an extremely large csv file that I need to search the second field, and upon matches update the last field...

I can pull the line with awk.. but apparently you cant use awk to directly update the file? So im curious if I can use sed to do this... The good news is the field I want to update is the last field, so I can just do a right to left search for "," and update to the right.

anyways, the details.
Code:
...
field1,X.X.X.X,feild3,...,field13 
field1,X.X.X.X,feild3,...,field13 
field1,X.X.X.X,feild3,...,field14 <This is the field I need to change
field1,X.X.X.X,feild3,...,field13 
field1,X.X.X.X,feild3,...,field13 
field1,X.X.X.X,feild3,...,field13 
field1,X.X.X.X,feild3,...,field13 
field1,X.X.X.X,feild3,...,field13 
field1,X.X.X.X,feild3,...,field13
...

awk line to pull the info I want...
Code:
awk -v ip=$ip 'NF > 13 {if ($2 == ip && $14 =="*") print $0}' file.csv

now, besides turning the above into 3-4 long bash variables... Is there any easy way to change field $14 in the above line? Smilie
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk updating one file with another, comparing, updating

Hello, I read and search through this wonderful forum and tried different approaches but it seems I lack some knowledge and neurones ^^ Here is what I'm trying to achieve : file1: test filea 3495; test fileb 4578; test filec 7689; test filey 9978; test filez 12300; file2: test filea... (11 Replies)
Discussion started by: mecano
11 Replies

2. Shell Programming and Scripting

Sed or awk script to remove text / or perform calculations from large CSV files

I have a large CSV files (e.g. 2 million records) and am hoping to do one of two things. I have been trying to use awk and sed but am a newbie and can't figure out how to get it to work. Any help you could offer would be greatly appreciated - I'm stuck trying to remove the colon and wildcards in... (6 Replies)
Discussion started by: metronomadic
6 Replies

3. Shell Programming and Scripting

Need help with a script to process a CSV file using SED and AWK

I get a CSV file every day with 2 columns and multiple rows ex: date1,date2 ( both the fields are varchar fields) This data has to be updated in a table which is being done manually and i want to automate that. 1. I have to select all the data from the prod table( 2 columns { date1,date2}) into... (4 Replies)
Discussion started by: kkb
4 Replies

4. Shell Programming and Scripting

Using awk/sed in handling csv file.

Please study the below script and the output Script: echo "Minimum ${host} ${process} response time=${min} ms" >> ${OUTDIR}/${OUTFILE}; echo "Maximum ${host} ${process} response time=${max} ms" >> ${OUTDIR}/${OUTFILE}; echo "Average ${host} ${process} response time=${avg} ms" >>... (0 Replies)
Discussion started by: ajincoep
0 Replies

5. Shell Programming and Scripting

Script for updating a .csv file

Hello guys, I have one .csv file with all the file names(.c,.h,.sm,.txt etc)..... Now i have compared two baseline in UCM and got a report in .txt file,which looks like this...... DIFFBL INT_1029_Rel6.0.0.90@\DIR_PVOB STD_6.0.0.50_UCM@\DIR_PVOB (COMPONENT TEXT) Comparing the following: ... (14 Replies)
Discussion started by: suvenduperl
14 Replies

6. Shell Programming and Scripting

awk/sed/something else for csv file

Hi, I have a filename.csv in which there are 3 colums, ie: Name ; prefixnumber ; number root ; 020 ; 1234567 user1,2,3 ; 070 ; 7654321 What I want is to merge colum 2 and 3 that it becomes 0201234567 or even better +31201234567 so the country number is used and drop the leading 0.... (9 Replies)
Discussion started by: necron
9 Replies

7. Shell Programming and Scripting

Converting txt file into CSV using awk or sed

Hello folks I have a txt file of information about journal articles from different fields. I need to convert this information into a format that is easier for computers to manipulate for some research that I'm doing on how articles are cited. The file has some header information and then details... (8 Replies)
Discussion started by: ksk
8 Replies

8. Shell Programming and Scripting

sed and awk not working on a large record file

Hi All, I have a very large single record file. abc;date||bcd;efg|......... pqr;stu||record_count;date when i do wc -l on this file it gives me "0" records, coz of missing line feed. my problem is there is an extra pipe that is coming at the end of this record like... (6 Replies)
Discussion started by: Gurkamal83
6 Replies

9. Shell Programming and Scripting

Honey, I broke awk! (duplicate line removal in 30M line 3.7GB csv file)

I have a script that builds a database ~30 million lines, ~3.7 GB .cvs file. After multiple optimzations It takes about 62 min to bring in and parse all the files and used to take 10 min to remove duplicates until I was requested to add another column. I am using the highly optimized awk code: awk... (34 Replies)
Discussion started by: Michael Stora
34 Replies

10. UNIX for Beginners Questions & Answers

sed awk: split a large file to unique file names

Dear Users, Appreciate your help if you could help me with splitting a large file > 1 million lines with sed or awk. below is the text in the file input file.txt scaffold1 928 929 C/T + scaffold1 942 943 G/C + scaffold1 959 960 C/T +... (6 Replies)
Discussion started by: kapr0001
6 Replies
SORT(1) 						    BSD General Commands Manual 						   SORT(1)

NAME
sort -- sort or merge text files SYNOPSIS
sort [-bcdfHilmnrSsu] [-k field1[,field2]] [-o output] [-R char] [-T dir] [-t char] [file ...] DESCRIPTION
The sort utility sorts text files by lines. Comparisons are based on one or more sort keys extracted from each line of input, and are per- formed lexicographically. By default, if keys are not given, sort regards each input line as a single field. The following options are available: -c Check that the single input file is sorted. If the file is not sorted, sort produces the appropriate error messages and exits with code 1; otherwise, sort returns 0. sort -c produces no output. -H Ignored for compatibility with earlier versions of sort. -m Merge only; the input files are assumed to be pre-sorted. -o output The argument given is the name of an output file to be used instead of the standard output. This file can be the same as one of the input files. -S Don't use stable sort. Default is to use stable sort. -s Use stable sort, keeps records with equal keys in their original order. This is the default. Provided for compatibility with other sort implementations only. -T dir Use dir as the directory for temporary files. The default is the value specified in the environment variable TMPDIR or /tmp if TMPDIR is not defined. -u Unique: suppress all but one in each set of lines having equal keys. If used with the -c option, check that there are no lines with duplicate keys. The following options override the default ordering rules. When ordering options appear independent of key field specifications, the requested field ordering rules are applied globally to all sort keys. When attached to a specific key (see -k), the ordering options over- ride all global ordering options for that key. -d Only blank space and alphanumeric characters are used in making comparisons. -f Considers all lowercase characters that have uppercase equivalents to be the same for purposes of comparison. -i Ignore all non-printable characters. -l Sort by the string length of the field, not by the field itself. -n An initial numeric string, consisting of optional blank space, optional minus sign, and zero or more digits (including decimal point) is sorted by arithmetic value. (The -n option no longer implies the -b option.) -r Reverse the sense of comparisons. The treatment of field separators can be altered using these options: -b Ignores leading blank space when determining the start and end of a restricted sort key. A -b option specified before the first -k option applies globally to all -k options. Otherwise, the -b option can be attached independently to each field argument of the -k option (see below). Note that the -b option has no effect unless key fields are specified. -t char char is used as the field separator character. The initial char is not considered to be part of a field when determining key offsets (see below). Each occurrence of char is significant (for example, ``charchar'' delimits an empty field). If -t is not specified, the default field separator is a sequence of blank-space characters, and consecutive blank spaces do not delimit an empty field; further, the initial blank space is considered part of a field when determining key offsets. -R char char is used as the record separator character. This should be used with discretion; -R <alphanumeric> usually produces undesir- able results. The default record separator is newline. -k field1[,field2] Designates the starting position, field1, and optional ending position, field2, of a key field. The -k option replaces the obso- lescent options +pos1 and -pos2. The following operands are available: file The pathname of a file to be sorted, merged, or checked. If no file operands are specified, or if a file operand is -, the standard input is used. A field is defined as a minimal sequence of characters followed by a field separator or a newline character. By default, the first blank space of a sequence of blank spaces acts as the field separator. All blank spaces in a sequence of blank spaces are considered as part of the next field; for example, all blank spaces at the beginning of a line are considered to be part of the first field. Fields are specified by the -k field1[,field2] argument. A missing field2 argument defaults to the end of a line. The arguments field1 and field2 have the form m.n and can be followed by one or more of the letters b, d, f, i, l, n, and r, which correspond to the options discussed above. A field1 position specified by m.n (m, n > 0) is interpreted as the nth character in the mth field. A miss- ing .n in field1 means '.1', indicating the first character of the mth field; if the -b option is in effect, n is counted from the first non- blank character in the mth field; m.1b refers to the first non-blank character in the mth field. A field2 position specified by m.n is interpreted as the nth character (including separators) of the mth field. A missing .n indicates the last character of the mth field; m = 0 designates the end of a line. Thus the option -k v.x,w.y is synonymous with the obsolescent option +v-1.x-1-w-1.y; when y is omitted, -k v.x,w is synonymous with +v-1.x-1-w+1.0. The obsolescent +pos1 -pos2 option is still supported, except for -w.0b, which has no -k equivalent. ENVIRONMENT
If the following environment variable exists, it is used by sort. TMPDIR sort uses the contents of the TMPDIR environment variable as the path in which to store temporary files. FILES
/tmp/sort.* Default temporary files. outputNUMBER Temporary file which is used for output if output already exists. Once sorting is finished, this file replaces output (via link(2) and unlink(2)). EXIT STATUS
Sort exits with one of the following values: 0 Normal behavior. 1 On disorder (or non-uniqueness) with the -c option 2 An error occurred. SEE ALSO
comm(1), join(1), uniq(1), qsort(3), radixsort(3) HISTORY
A sort command appeared in Version 5 AT&T UNIX. This sort implementation appeared in 4.4BSD and is used since NetBSD 1.6. BUGS
Posix requires the locale's thousands separator be ignored in numbers. It may be faster to sort very large files in pieces and then explic- itly merge them. NOTES
This sort has no limits on input line length (other than imposed by available memory) or any restrictions on bytes allowed within lines. To protect data sort -o calls link(2) and unlink(2), and thus fails on protected directories. Input files should be text files. If file doesn't end with record separator (which is typically newline), the sort utility silently supplies one. The current sort uses lexicographic radix sorting, which requires that sort keys be kept in memory (as opposed to previous versions which used quick and merge sorts and did not.) Thus performance depends highly on efficient choice of sort keys, and the -b option and the field2 argument of the -k option should be used whenever possible. Similarly, sort -k1f is equivalent to sort -f and may take twice as long. BSD
December 18, 2010 BSD
All times are GMT -4. The time now is 02:40 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy