Sed or awk script to remove text / or perform calculations from large CSV files
I have a large CSV files (e.g. 2 million records) and am hoping to do one of two things. I have been trying to use awk and sed but am a newbie and can't figure out how to get it to work. Any help you could offer would be greatly appreciated - I'm stuck trying to remove the colon and wildcards in sed, and the average sample I've found using awk is giving me values of around 4e08.
The CSV file looks like this:
I am hoping to either use sed or another script to remove the seconds portion of the data lines (i.e. remove ":10 AM" and all similar occurrences, or preferably to use awk to average the flow rates for each minute or each 15 minutes (i.e. the column right after the time).
Could someone tell me how to perform calculations using numbers greater than 2150000000 in Korn Shell? When I tried to do it it gave me the wrong answer.
e.g. I have a ksh file with the contents below:
---------------------------------
#!/bin/ksh
SUM=`expr 2150000000 + 2`
PRODUCT=`expr... (3 Replies)
Hi friends,
I am in a little confusion.
I have a comma separated file contains hugh records.
say,
88562848,21-JAN-08,2741079, -1188,-7433,TESTING
88558314,21-JAN-08,2741189, -1273,-7976,TESTING
I want 4th and 5th field to be multiplied by 100.
like,
88562848,21-JAN-08,2741079,... (3 Replies)
I am trying to remove the ita from this file:
"1234ita","john","smith"
"56789ita","jim","thomas"
the command i am using is:
sed '/^ita/d' infile.csv > outfile.csv
it is running but doing nothing, very rarely use sed so trying to learn it any help would be appreciated (2 Replies)
I have an extremely large csv file that I need to search the second field, and upon matches update the last field...
I can pull the line with awk.. but apparently you cant use awk to directly update the file? So im curious if I can use sed to do this... The good news is the field I want to... (5 Replies)
I get a CSV file every day with 2 columns and multiple rows
ex:
date1,date2 ( both the fields are varchar fields)
This data has to be updated in a table which is being done manually and i want to automate that.
1. I have to select all the data from the prod table( 2 columns { date1,date2}) into... (4 Replies)
hi...... thanks for allowing me to start a discussion
i am collecting usb usage details of all users and convert it into csv files so that i can export it into some database..
the input text file is as follows:-
USB History Dump
by nabiy (c)2008
(1) --- Kingston DataTraveler 130 USB... (2 Replies)
HI all,
I want to script where all the server names will be in a text file like
server1
server2
server3 . and the script should take servernames from a text file and perform copy of files if the files are not present on those servers.after which it should take next servername till the end of... (0 Replies)
Hello,
Error
awk: Internal software error in the tostring function on TS1101?05044400?.0085498227?0?.0011041461?.0034752266?.00397045?0?0?0?0?0?0?11/02/10?09/23/10???10?no??0??no?sct_det3_10_20110516_143936.txt
What it is
It is a unix shell script that contains an awk program as well as... (4 Replies)
Hi,
I am new to awk/unix and am trying to put together an awk script to perform an action similar to vlookup between the two csv files.
Here are the contents of the two files:
File 1:
Date,ParentID,Number,Area,Volume,Dimensions
2014-01-01,ABC,247,83430.33,857.84,8110.76... (9 Replies)
Hi,
I have a file (stats.txt) with columns like in the example below. Destination IP address, timestamp, TCP packet sequence number and packet length.
destIP time seqNo packetLength
1.2.3.4 0.01 123 500
1.2.3.5 0.03 44 1500
1.3.2.5 0.08 44 1500
1.2.3.4 0.44... (12 Replies)
Discussion started by: Zooma
12 Replies
LEARN ABOUT DEBIAN
igawk
IGAWK(1) Utility Commands IGAWK(1)NAME
igawk - gawk with include files
SYNOPSIS
igawk [ all gawk options ] -f program-file [ -- ] file ...
igawk [ all gawk options ] [ -- ] program-text file ...
DESCRIPTION
Igawk is a simple shell script that adds the ability to have ``include files'' to gawk(1).
AWK programs for igawk are the same as for gawk, except that, in addition, you may have lines like
@include getopt.awk
in your program to include the file getopt.awk from either the current directory or one of the other directories in the search path.
OPTIONS
See gawk(1) for a full description of the AWK language and the options that gawk supports.
EXAMPLES
cat << EOF > test.awk
@include getopt.awk
BEGIN {
while (getopt(ARGC, ARGV, "am:q") != -1)
...
}
EOF
igawk -f test.awk
SEE ALSO gawk(1)
Effective AWK Programming, Edition 1.0, published by the Free Software Foundation, 1995.
AUTHOR
Arnold Robbins (arnold@skeeve.com).
Free Software Foundation Nov 3 1999 IGAWK(1)