07-05-2018
2 million lines is quite something, and you have to read all into memory. Checking against 40 * 0.4 E6 (16 million) will take its time. Not sure if the system will be already into swapping with data amounts like those. Try cutting the 2 million in half or quarter.
Not sure if I understand your second query. Your usage of "csv" is not quite consistent and obvious to me.
awk needs files to operate upon, not directories. If you open, read, and close 40 (or so) files for every single line read from the (lookup / test / csv / GUID) file, you'll thrash your file system. Not clever.
Last edited by RudiC; 07-18-2018 at 05:51 AM..
Reason: Corrected an unclear formulation.
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hi All,
I've two .csv files as below
file1.csv
abc, tdf, 223, tpx
jgsd, tex, 342, rpy
a, jdjdsd, 423, djfkld
Where as file2.csv is the new version of file1.csv with some added fields in the end of each line and some additional lines.
lfj, eru, 98, jkldj, 39, jdkj9
abc, tdf, 223, tpx,... (3 Replies)
Discussion started by: ganapati
3 Replies
2. Shell Programming and Scripting
Hello,
I have the following problem:
There are two csv files
csv-file #1:
aaa1, aaa2, ... aaan
aaa1, bbb2, ... bbbn
aaa1, ccc2, ... cccn
bbb1, bbb2, ... bbbn
...
zzz1, zzz2, ... zzzn
csv-file #2:
aaa1, matchvalue1
ccc1, matchvalue2 (7 Replies)
Discussion started by: ghl10000
7 Replies
3. Shell Programming and Scripting
EDIT: My problems have been solved thanks to the help of bartus11 and pravin27
This code is just to help me learn. It serves no purpose other than that.
Here's a sample csv that I'm working with -
#listofpeeps.csv
Jackie Chan,1954,M
Chuck Norris,1930,M
Bruce Lee,1940,M
This code is... (13 Replies)
Discussion started by: chickeneaterguy
13 Replies
4. Shell Programming and Scripting
Hi!
I'm just new to shell scripting n simple tasks looks so tough in initial stage. i need to write a script which will read a property file, property file will be containing count of the csv files, and in a folder(same folder) there will be respective csv files.
like Property file
data1=100... (3 Replies)
Discussion started by: sukhdip
3 Replies
5. Shell Programming and Scripting
Hi All,
I want to remove the rows from File1.csv by comparing a column/field in the File2.csv. If both columns matches then I want that row to be deleted from File1 using shell script(awk). Here is an example on what I need.
File1.csv:
RAJAK,ACTIVE,1
VIJAY,ACTIVE,2
TAHA,ACTIVE,3... (6 Replies)
Discussion started by: rajak.net
6 Replies
6. Shell Programming and Scripting
Hi All,
I want to remove the rows from File1.csv by comparing the columns/fields in the File2.csv. I only need the records whose first column is same and the second column is different for the same record in both files.Here is an example on what I need.
File1.csv:
RAJAK|ACTIVE|1... (2 Replies)
Discussion started by: rajak.net
2 Replies
7. Shell Programming and Scripting
Hello,
I have about 10 csv files which range from csv1 - csv10.
Each csv file has same type/set of tabs and we have around 5-6 tabs for each of the csv file which have slightly different content(data).
A sample of CSV1 is shown below:
Joins: Data related to Joins, it can be any number of... (2 Replies)
Discussion started by: bobby1015
2 Replies
8. Shell Programming and Scripting
(say) I have 2 csv files - file1.csv & file2.csv as mentioned below:
file1.csv
ID,version,cost
1000,1,30
2000,2,40
3000,3,50
4000,4,60
file2.csv
ID,version,cost
1000,1,30
2000,2,45
3000,4,55
6000,5,70
... (1 Reply)
Discussion started by: Naresh101
1 Replies
9. Shell Programming and Scripting
Hi All,
My requirement is to write a shell script to compare two large csv files.
I've created sample files for explaining my problem
i.e., a.csv and b.csv
contents of files:
-----------------
a.csv
------
Type,Memory (Kb),Location HD,Size (Mb),Serial #
XT,640,D402,0,MG0010... (2 Replies)
Discussion started by: vasavi
2 Replies
10. Shell Programming and Scripting
I have two csv files and im trying to compare them. e.g.
SAMPLE DATA:
file one:
ZipCode Name
20878 Washington
10023 Missouri
20304 Maryland
file two:
ID Name City ZipCode
11654 ... (11 Replies)
Discussion started by: dan139
11 Replies
LEARN ABOUT DEBIAN
bb-csvinfo.cgi
BB-CSVINFO.CGI(1) General Commands Manual BB-CSVINFO.CGI(1)
NAME
bb-csvinfo.cgi - CGI program to show host information from a CSV file
SYNOPSIS
bb-csvinfo.cgi
DESCRIPTION
bb-csvinfo.cgi is invoked as a CGI script via the bb-csvinfo.sh CGI wrapper. Based on the parameters it receives, it searches a comma-
separated file for the matching host, and presents the information found as a table.
bb-csvinfo.cgi is passed a QUERY_STRING environment variable with the following parameters:
key (string to search for, typically hostname)
column (columnnumber to search - default 0)
db (name of the CSV database file in $BBHOME/etc/, default hostinfo.csv)
delimiter (delimiter character for columns, default semi-colon)
CSV files are easily created from e.g. spreadsheets, by exporting them in CSV format. You should have one host per line, with the first
line containing the column headings. Despite their name, the default delimiter for CSV files is the semi-colon - if you need a different
delimiter, invoke bb-csvinfo.cgi with the "delimiter=<character>" in the query string.
Example usage
This example shows how you can use the bb-csvinfo CGI. It assumes you have a CSV-formatted file with information about the hosts stored as
$BBHOME/etc/hostinfo.csv, and the hostname is in the first column of the file.
Use with the bbgen --docurl
The --docurl option to bbgen(1) sets up all of the hostnames on your Xymon webpages to act as links to a CGI script. To invoke the
bb-csvinfo CGI script, run bbgen with the option
--docurl=/cgi-bin/bb-csvinfo.sh?db=hostinfo.csv&key=%s
SEE ALSO
bb-hosts(5), hobbitserver.cfg(5), bbgen(1)
Xymon Version 4.2.3: 4 Feb 2009 BB-CSVINFO.CGI(1)