Login or Register to Ask a Question and Join Our Community


Programming

Pandas vs Python CSV module

 
Thread Tools Search this Thread
Top Forums Programming Pandas vs Python CSV module
Prev   Next
# 3  
Old 03-16-2017
If you are on windows open the resource monitor (hit windows +r then type "resmon").
If you are running MacOS or Linux there are similar tools.

Run the program and check the number of hard faults and the amount of physical memory used. You can checkmark the python process to get the numbers for only that process. If there are a lot of hard faults and the amount of physical memory is high you don't have enough RAM.

You could also do some testing with smaller .csv files. If a file with only x% the entries takes x% of the time then RAM is not the problem. If for small x (where less memory is needed and the whole thing might fit in your RAM) it takes less than x% of the time then RAM is the problem.


I don't know about the library you are using but if possible log the time every 10000 (or s0) lines parsed with the time. This will allow you to monitor if the rat at which the file is parsed stays constant or slows down.
Last edited by RudiC; 03-16-2017 at 06:43 AM.. Reason: Removed personal URL.
 
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to insert a CSV within xml element tag using Python?

Hi Team, I have a CSV file which I have to read through and needs to insert the content within an XML file using Python ONLY ( as most of the code base we have in python only). I managed to find the first part, missing how to insert to XML under "specific" tags. cat input.csv... (0 Replies)
Discussion started by: panyam
0 Replies

2. Shell Programming and Scripting

Convert excel to csv in python date not display exactly

Hi, Anyone can help I am just converting the excel file to csv using python, I can get the conversion output but the date not display exactly. test.xlsx date format 167 1588 risks/SCS JP CAMPANA & CIE.pdf SCS JP CAMPANA & CIE 2 1 1 0 2015-03-16 16:56:25 167 1146 risks/AirBNB... (1 Reply)
Discussion started by: fspalero
1 Replies

3. Shell Programming and Scripting

Import 2 columns from 8 .csv files into pandas df (side by side) and write a new csv

I have 8 .csv files with 16 columns and "n" rows with no Header. I want to parse each of these .csv and get column and put the data into a new.csv. Once this is done, the new.csv should have 16 columns (2 from each input.csv) and "n" rows. Now, I want to just take the average of Column from... (3 Replies)
Discussion started by: Zam_1234
3 Replies

4. Programming

Web browser module in python

Hi, I am trying to multiple urls in multiple tabs within a browser window. I am using webbrowser module in python: My code: import webbrowser li = for url in li: webbrowser.open_new_tab(url) This code works fine, if the browser is already opened. If the browser is not started... (5 Replies)
Discussion started by: pandeesh
5 Replies

5. Shell Programming and Scripting

Perl: module to convert xlsx to csv

Is there any perl module to convert .xlsx file(excel sheet 2007) to a csv file. (1 Reply)
Discussion started by: giridhar276
1 Replies

6. Shell Programming and Scripting

suggest a python module for automation..

Im planning to automate my testcases. Please suggest me a python module that will ease my automation scripting. General steps involved in the testcase execution: (all steps below are done in linux machine) 1) start the application under test (AUT) 2) connect to oracle/ms sql db and update... (1 Reply)
Discussion started by: Arun_Linux
1 Replies

7. Shell Programming and Scripting

Converting specific Excel file tabs to CSV in Python

Hi list, This is probably something really simple, but I am not particularly familiar with Python so I thought I would ask as I know that python has an excel module. I have an excel document with multiple tabs of data and graphs. One of the tabs is just data which I require to have dumped to... (8 Replies)
Discussion started by: landossa
8 Replies

8. Shell Programming and Scripting

Python subprocess module

I need to run this command using python subprocess module (notice I'm using only variables): cmd = TESTPATH + ' -s ' + serviceName + ' -r ' + rdir + \ ' -m ' + masterAcct + ' -p ' + persona + ' -P ' + passwd (3 Replies)
Discussion started by: erick_tuk
3 Replies

9. Shell Programming and Scripting

Looking for a Python CLI module

When I say "Command Line Interface" I mean the wrapper that allows you to type "commands" and have then correspond to pre-programmed actions... all the while making sure that it looks like you are INSIDE a shell of sorts. I had good hope for "common.cli" but that didnt last long, when I found... (1 Reply)
Discussion started by: jjinno
1 Replies
Login or Register to Ask a Question

LEARN ABOUT DEBIAN

bb-csvinfo.cgi

BB-CSVINFO.CGI(1)					      General Commands Manual						 BB-CSVINFO.CGI(1)

NAME

       bb-csvinfo.cgi - CGI program to show host information from a CSV file

SYNOPSIS

       bb-csvinfo.cgi

DESCRIPTION

       bb-csvinfo.cgi  is  invoked  as	a CGI script via the bb-csvinfo.sh CGI wrapper.  Based on the parameters it receives, it searches a comma-
       separated file for the matching host, and presents the information found as a table.

       bb-csvinfo.cgi is passed a QUERY_STRING environment variable with the following parameters:

	  key (string to search for, typically hostname)
	  column (columnnumber to search - default 0)
	  db  (name of the CSV database file in $BBHOME/etc/, default hostinfo.csv)
	  delimiter (delimiter character for columns, default semi-colon)

       CSV files are easily created from e.g. spreadsheets, by exporting them in CSV format. You should have one host per  line,  with	the  first
       line  containing  the  column headings. Despite their name, the default delimiter for CSV files is the semi-colon - if you need a different
       delimiter, invoke bb-csvinfo.cgi with the "delimiter=<character>" in the query string.

Example usage
       This example shows how you can use the bb-csvinfo CGI. It assumes you have a CSV-formatted file with information about the hosts stored	as
       $BBHOME/etc/hostinfo.csv, and the hostname is in the first column of the file.

       Use with the bbgen --docurl
	      The  --docurl  option to bbgen(1) sets up all of the hostnames on your Xymon webpages to act as links to a CGI script. To invoke the
	      bb-csvinfo CGI script, run bbgen with the option

		 --docurl=/cgi-bin/bb-csvinfo.sh?db=hostinfo.csv&key=%s

SEE ALSO

       bb-hosts(5), hobbitserver.cfg(5), bbgen(1)

Xymon							    Version 4.2.3:  4 Feb 2009						 BB-CSVINFO.CGI(1)