Sponsored Content
Full Discussion: Pandas vs Python CSV module
Top Forums Programming Pandas vs Python CSV module Post 302993902 by grabbitmedia on Thursday 16th of March 2017 05:25:20 AM
Old 03-16-2017
If you are on windows open the resource monitor (hit windows +r then type "resmon").
If you are running MacOS or Linux there are similar tools.

Run the program and check the number of hard faults and the amount of physical memory used. You can checkmark the python process to get the numbers for only that process. If there are a lot of hard faults and the amount of physical memory is high you don't have enough RAM.

You could also do some testing with smaller .csv files. If a file with only x% the entries takes x% of the time then RAM is not the problem. If for small x (where less memory is needed and the whole thing might fit in your RAM) it takes less than x% of the time then RAM is the problem.


I don't know about the library you are using but if possible log the time every 10000 (or s0) lines parsed with the time. This will allow you to monitor if the rat at which the file is parsed stays constant or slows down.

Last edited by RudiC; 03-16-2017 at 06:43 AM.. Reason: Removed personal URL.
 

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Looking for a Python CLI module

When I say "Command Line Interface" I mean the wrapper that allows you to type "commands" and have then correspond to pre-programmed actions... all the while making sure that it looks like you are INSIDE a shell of sorts. I had good hope for "common.cli" but that didnt last long, when I found... (1 Reply)
Discussion started by: jjinno
1 Replies

2. Shell Programming and Scripting

Python subprocess module

I need to run this command using python subprocess module (notice I'm using only variables): cmd = TESTPATH + ' -s ' + serviceName + ' -r ' + rdir + \ ' -m ' + masterAcct + ' -p ' + persona + ' -P ' + passwd (3 Replies)
Discussion started by: erick_tuk
3 Replies

3. Shell Programming and Scripting

Converting specific Excel file tabs to CSV in Python

Hi list, This is probably something really simple, but I am not particularly familiar with Python so I thought I would ask as I know that python has an excel module. I have an excel document with multiple tabs of data and graphs. One of the tabs is just data which I require to have dumped to... (8 Replies)
Discussion started by: landossa
8 Replies

4. Shell Programming and Scripting

suggest a python module for automation..

Im planning to automate my testcases. Please suggest me a python module that will ease my automation scripting. General steps involved in the testcase execution: (all steps below are done in linux machine) 1) start the application under test (AUT) 2) connect to oracle/ms sql db and update... (1 Reply)
Discussion started by: Arun_Linux
1 Replies

5. Shell Programming and Scripting

Perl: module to convert xlsx to csv

Is there any perl module to convert .xlsx file(excel sheet 2007) to a csv file. (1 Reply)
Discussion started by: giridhar276
1 Replies

6. Programming

Web browser module in python

Hi, I am trying to multiple urls in multiple tabs within a browser window. I am using webbrowser module in python: My code: import webbrowser li = for url in li: webbrowser.open_new_tab(url) This code works fine, if the browser is already opened. If the browser is not started... (5 Replies)
Discussion started by: pandeesh
5 Replies

7. Shell Programming and Scripting

Import 2 columns from 8 .csv files into pandas df (side by side) and write a new csv

I have 8 .csv files with 16 columns and "n" rows with no Header. I want to parse each of these .csv and get column and put the data into a new.csv. Once this is done, the new.csv should have 16 columns (2 from each input.csv) and "n" rows. Now, I want to just take the average of Column from... (3 Replies)
Discussion started by: Zam_1234
3 Replies

8. Shell Programming and Scripting

Convert excel to csv in python date not display exactly

Hi, Anyone can help I am just converting the excel file to csv using python, I can get the conversion output but the date not display exactly. test.xlsx date format 167 1588 risks/SCS JP CAMPANA & CIE.pdf SCS JP CAMPANA & CIE 2 1 1 0 2015-03-16 16:56:25 167 1146 risks/AirBNB... (1 Reply)
Discussion started by: fspalero
1 Replies

9. Shell Programming and Scripting

How to insert a CSV within xml element tag using Python?

Hi Team, I have a CSV file which I have to read through and needs to insert the content within an XML file using Python ONLY ( as most of the code base we have in python only). I managed to find the first part, missing how to insert to XML under "specific" tags. cat input.csv... (0 Replies)
Discussion started by: panyam
0 Replies
bcopy(9F)						   Kernel Functions for Drivers 						 bcopy(9F)

NAME
bcopy - copy data between address locations in the kernel SYNOPSIS
#include <sys/types.h> #include <sys/sunddi.h> void bcopy(const void *from, void *to, size_t bcount); INTERFACE LEVEL
Architecture independent level 1 (DDI/DKI). PARAMETERS
from Source address from which the copy is made. to Destination address to which copy is made. bcount The number of bytes moved. DESCRIPTION
The bcopy() function copies bcount bytes from one kernel address to another. If the input and output addresses overlap, the command exe- cutes, but the results may not be as expected. Note that bcopy() should never be used to move data in or out of a user buffer, because it has no provision for handling page faults. The user address space can be swapped out at any time, and bcopy() always assumes that there will be no paging faults. If bcopy() attempts to access the user buffer when it is swapped out, the system will panic. It is safe to use bcopy() to move data within kernel space, since kernel space is never swapped out. CONTEXT
The bcopy() function can be called from user, interrupt, or kernel context. EXAMPLES
Example 1 Copying data between address locations in the kernel: An I/O request is made for data stored in a RAM disk. If the I/O operation is a read request, the data is copied from the RAM disk to a buffer (line 8). If it is a write request, the data is copied from a buffer to the RAM disk (line 15). bcopy() is used since both the RAM disk and the buffer are part of the kernel address space. 1 #define RAMDNBLK 1000 /* blocks in the RAM disk */ 2 #define RAMDBSIZ 512 /* bytes per block */ 3 char ramdblks[RAMDNBLK][RAMDBSIZ]; /* blocks forming RAM /* disk ... 4 5 if (bp->b_flags & B_READ) /* if read request, copy data */ 6 /* from RAM disk data block */ 7 /* to system buffer */ 8 bcopy(&ramdblks[bp->b_blkno][0], bp->b_un.b_addr, 9 bp->b_bcount); 10 11 else /* else write request, */ 12 /* copy data from a */ 13 /* system buffer to RAM disk */ 14 /* data block */ 15 bcopy(bp->b_un.b_addr, &ramdblks[bp->b_blkno][0], 16 bp->b_bcount); SEE ALSO
copyin(9F), copyout(9F) Writing Device Drivers WARNINGS
The from and to addresses must be within the kernel space. No range checking is done. If an address outside of the kernel space is selected, the driver may corrupt the system in an unpredictable way. SunOS 5.11 16 Jan 2006 bcopy(9F)
All times are GMT -4. The time now is 05:20 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy