Login or Register to Ask a Question and Join Our Community


Programming

Pandas vs Python CSV module

 
Thread Tools Search this Thread
Top Forums Programming Pandas vs Python CSV module
Prev   Next
# 3  
Old 03-16-2017
If you are on windows open the resource monitor (hit windows +r then type "resmon").
If you are running MacOS or Linux there are similar tools.

Run the program and check the number of hard faults and the amount of physical memory used. You can checkmark the python process to get the numbers for only that process. If there are a lot of hard faults and the amount of physical memory is high you don't have enough RAM.

You could also do some testing with smaller .csv files. If a file with only x% the entries takes x% of the time then RAM is not the problem. If for small x (where less memory is needed and the whole thing might fit in your RAM) it takes less than x% of the time then RAM is the problem.


I don't know about the library you are using but if possible log the time every 10000 (or s0) lines parsed with the time. This will allow you to monitor if the rat at which the file is parsed stays constant or slows down.
Last edited by RudiC; 03-16-2017 at 06:43 AM.. Reason: Removed personal URL.
 
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to insert a CSV within xml element tag using Python?

Hi Team, I have a CSV file which I have to read through and needs to insert the content within an XML file using Python ONLY ( as most of the code base we have in python only). I managed to find the first part, missing how to insert to XML under "specific" tags. cat input.csv... (0 Replies)
Discussion started by: panyam
0 Replies

2. Shell Programming and Scripting

Convert excel to csv in python date not display exactly

Hi, Anyone can help I am just converting the excel file to csv using python, I can get the conversion output but the date not display exactly. test.xlsx date format 167 1588 risks/SCS JP CAMPANA & CIE.pdf SCS JP CAMPANA & CIE 2 1 1 0 2015-03-16 16:56:25 167 1146 risks/AirBNB... (1 Reply)
Discussion started by: fspalero
1 Replies

3. Shell Programming and Scripting

Import 2 columns from 8 .csv files into pandas df (side by side) and write a new csv

I have 8 .csv files with 16 columns and "n" rows with no Header. I want to parse each of these .csv and get column and put the data into a new.csv. Once this is done, the new.csv should have 16 columns (2 from each input.csv) and "n" rows. Now, I want to just take the average of Column from... (3 Replies)
Discussion started by: Zam_1234
3 Replies

4. Programming

Web browser module in python

Hi, I am trying to multiple urls in multiple tabs within a browser window. I am using webbrowser module in python: My code: import webbrowser li = for url in li: webbrowser.open_new_tab(url) This code works fine, if the browser is already opened. If the browser is not started... (5 Replies)
Discussion started by: pandeesh
5 Replies

5. Shell Programming and Scripting

Perl: module to convert xlsx to csv

Is there any perl module to convert .xlsx file(excel sheet 2007) to a csv file. (1 Reply)
Discussion started by: giridhar276
1 Replies

6. Shell Programming and Scripting

suggest a python module for automation..

Im planning to automate my testcases. Please suggest me a python module that will ease my automation scripting. General steps involved in the testcase execution: (all steps below are done in linux machine) 1) start the application under test (AUT) 2) connect to oracle/ms sql db and update... (1 Reply)
Discussion started by: Arun_Linux
1 Replies

7. Shell Programming and Scripting

Converting specific Excel file tabs to CSV in Python

Hi list, This is probably something really simple, but I am not particularly familiar with Python so I thought I would ask as I know that python has an excel module. I have an excel document with multiple tabs of data and graphs. One of the tabs is just data which I require to have dumped to... (8 Replies)
Discussion started by: landossa
8 Replies

8. Shell Programming and Scripting

Python subprocess module

I need to run this command using python subprocess module (notice I'm using only variables): cmd = TESTPATH + ' -s ' + serviceName + ' -r ' + rdir + \ ' -m ' + masterAcct + ' -p ' + persona + ' -P ' + passwd (3 Replies)
Discussion started by: erick_tuk
3 Replies

9. Shell Programming and Scripting

Looking for a Python CLI module

When I say "Command Line Interface" I mean the wrapper that allows you to type "commands" and have then correspond to pre-programmed actions... all the while making sure that it looks like you are INSIDE a shell of sorts. I had good hope for "common.cli" but that didnt last long, when I found... (1 Reply)
Discussion started by: jjinno
1 Replies
Login or Register to Ask a Question

LEARN ABOUT DEBIAN

ifpps

ifpps(8)							netsniff-ng-toolkit							  ifpps(8)

NAME

       ifpps - fetch and format kernel network statistics

SYNOPSIS

       ifpps	 -d|--dev <netdev> [-t|--interval <sec>][-p|--promisc][-c|--term]
	    [-C|--csv][-H|--csv-tablehead][-l|--loop][-v|--version][-h|--help]

DESCRIPTION

       A tiny tool to provide top-like reliable networking statistics.	ifpps reads out the 'real' kernel statistics, so it does not give
       erroneous statistics on high I/O load.

OPTIONS

       ifpps --dev eth0
	   Fetch eth0 interface statistics.

       ifpps --dev eth0 --interval 60 --csv
	   Output eth0 interface statistics every minute in CSV format.

OPTIONS

       -h|--help
	   Print help text and lists all options.

       -v|--version
	   Print version.

       -d|--dev <netdev>
	   Device to fetch statistics for i.e., eth0.

       -p|--promisc
	   Put the device in promiscuous mode

       -t|--interval <time>
	   Refresh time in sec (default 1 sec)

       -c|--term
	   Output to terminal

       -C|--csv
	   Output in CSV format.  E.g. post-processing with Gnuplot et al.

       -H|--csv-tablehead
	   Print CSV table head.

       -l|--loop
	   Loop terminal output.

AUTHOR

       Written by Daniel Borkmann <daniel@netsniff-ng.org>

DOCUMENTATION

       Documentation by Emmanuel Roullit <emmanuel@netsniff-ng.org>

BUGS

       Please report bugs to <bugs@netsniff-ng.org>

								    2012-06-29								  ifpps(8)