Sponsored Content
Full Discussion: C++ help in large data set
Top Forums Programming C++ help in large data set Post 303020687 by arunkumar_mca on Wednesday 25th of July 2018 02:20:47 PM
Old 07-25-2018
C++ help in large data set

Hi All,

We are trying to replace a 3rdparty where we don't know how they handled the reader part here. The query below is getting 197 * 2038017 row in the table.

In the below code we are trying to run the query and execute in the DB part and fetch and read the record. That is where it is failing.while loop not entered until executeSql returns (function returns an integer we don't check), resulting in entire result in memory.

Any help


Code:
 CosBuf << "SELECT " << m_DefnVec.getColNameList() << " "
          << "FROM " << OBTTable.c_str() << " "
          << "ORDER BY " << DATE_COL;
    strSQLStmt = CosBuf.str();
    m_ptrDBConnect->executeSql(strSQLStmt.c_str(),reader);
while (reader.read())
   {
     if (reader.isValid())
     {
      
      ...
      ..

DB part :

void DSDbReader::AddRecord(int nSize, COLUMN_DATA* data)
{
std::shared_ptr<DSRecord> pRec = std::make_shared<DSRecord>();
for(int k = 0; k < nSize; k++)
{
pRec->AddColumnData();
COLUMN_DATA *pColumnData = pRec->GetColumnData(k);
pColumnData->AllocateValue(data[k].m_nValueLen);
   memcpy(pColumnData->m_psValue, data[k].m_psValue, data[k].m_nValueLen);
   pColumnData->m_nValueLen = data[k].m_nValueLen;
}
m_oRecords.push_back(pRec);

}

If I limit the table to 1 or 2 rows or with lesser column say 50 record it works fine. Possible the malloc is bringing more data to fit in memory

Last edited by arunkumar_mca; 07-25-2018 at 03:34 PM..
 

9 More Discussions You Might Find Interesting

1. HP-UX

large file options is set

Can someone tell me the right or exact syntax to check if the large file options is set on a filesystem Thanks! (2 Replies)
Discussion started by: catwomen
2 Replies

2. AIX

Value too large to be stored in data type???

Hello, I get this message : "Value too large to be stored in data type" when I try to open a 3Gb file. Can someone helps me to resolve the problem. Thank you very much (5 Replies)
Discussion started by: limame
5 Replies

3. Shell Programming and Scripting

Extracting data from large logs.

Hi all, I have a large log file that gets created daily. I need to be able to pull text out of the log when I hit a pattern including the 7 lines above it and the 3 lines below and output it to a new text file. Line 1 Line 2 Line 3 Line 4 Line 5 Line 6 Line 7 Pattern Line 9 Line 10... (11 Replies)
Discussion started by: bas
11 Replies

4. Shell Programming and Scripting

Drop common lines at head/tail of a large set of files

Hi! I have a large set of pairs of text files (each pair in their own subdirectory) and each pair shares head/tail (a couple of first and last lines) but differs in the middle part. I need to delete the heads/tails and keep only the middle portions in which they differ. The lengths of heads/tails... (1 Reply)
Discussion started by: dobryden
1 Replies

5. Shell Programming and Scripting

Finding data in large no. of files

I need to find some data in a large no. of files. The data is in the following format : VALUE A VALUE B VALUE C VALUE D 10 4 65 1 12 4.5 65.5 2 10.75 5.1 ... (2 Replies)
Discussion started by: cooker97
2 Replies

6. Shell Programming and Scripting

Need to delete large set of files (i.e) close to 100K from a directory based on the input file

Hi all, I need a script to delete a large set of files from a directory under / based on an input file and want to redirect errors into separate file. I have already prepared a list of files in the input file. Kndly help me. Thanks, Prash (36 Replies)
Discussion started by: prash358
36 Replies

7. AIX

AIX and Value too large to be stored in data type.

root@test8:/config1>oslevel -s 5300-01-00-0000 root@test8:/config1>oslevel -r 5300-01 while moving a file sized 9GB i got this message root@test8:/configapp>mv bkp_14JUN16_oraapp_2.tgz /oraapp bkp_14JUN16_oraapp_2.tgz: Value too large to be stored in data type. possibly need APAR... (2 Replies)
Discussion started by: filosophizer
2 Replies

8. UNIX for Beginners Questions & Answers

Transpose large data in UNIX

Hi I have the following sample of data: my full data dimention is 900,000* 1119 rs987435 C G 1 1 1 0 2 rs345783 C G 0 0 1 0 0 rs955894 G T 1 1 2 2 1 rs6088791 ... (7 Replies)
Discussion started by: marwah
7 Replies

9. Shell Programming and Scripting

Reversing large data set with awk?

Hello! I have quite a bit of map data that I have to edit. I originally had a DOS script that would reverse x1, y1 coordinates in order to change the direction of a particular segment in a map file. It worked wonderfully and all was well, but my bossman told me that there is a boatload of nodes... (9 Replies)
Discussion started by: Mothra
9 Replies
DBLINK_GET_RESULT(3)					  PostgreSQL 9.2.7 Documentation				      DBLINK_GET_RESULT(3)

NAME
dblink_get_result - gets an async query result SYNOPSIS
dblink_get_result(text connname [, bool fail_on_error]) returns setof record DESCRIPTION
dblink_get_result collects the results of an asynchronous query previously sent with dblink_send_query. If the query is not already completed, dblink_get_result will wait until it is. ARGUMENTS
conname Name of the connection to use. fail_on_error If true (the default when omitted) then an error thrown on the remote side of the connection causes an error to also be thrown locally. If false, the remote error is locally reported as a NOTICE, and the function returns no rows. RETURN VALUE
For an async query (that is, a SQL statement returning rows), the function returns the row(s) produced by the query. To use this function, you will need to specify the expected set of columns, as previously discussed for dblink. For an async command (that is, a SQL statement not returning rows), the function returns a single row with a single text column containing the command's status string. It is still necessary to specify that the result will have a single text column in the calling FROM clause. NOTES
This function must be called if dblink_send_query returned 1. It must be called once for each query sent, and one additional time to obtain an empty set result, before the connection can be used again. When using dblink_send_query and dblink_get_result, dblink fetches the entire remote query result before returning any of it to the local query processor. If the query returns a large number of rows, this can result in transient memory bloat in the local session. It may be better to open such a query as a cursor with dblink_open and then fetch a manageable number of rows at a time. Alternatively, use plain dblink(), which avoids memory bloat by spooling large result sets to disk. EXAMPLES
contrib_regression=# SELECT dblink_connect('dtest1', 'dbname=contrib_regression'); dblink_connect ---------------- OK (1 row) contrib_regression=# SELECT * FROM contrib_regression-# dblink_send_query('dtest1', 'select * from foo where f1 < 3') AS t1; t1 ---- 1 (1 row) contrib_regression=# SELECT * FROM dblink_get_result('dtest1') AS t1(f1 int, f2 text, f3 text[]); f1 | f2 | f3 ----+----+------------ 0 | a | {a0,b0,c0} 1 | b | {a1,b1,c1} 2 | c | {a2,b2,c2} (3 rows) contrib_regression=# SELECT * FROM dblink_get_result('dtest1') AS t1(f1 int, f2 text, f3 text[]); f1 | f2 | f3 ----+----+---- (0 rows) contrib_regression=# SELECT * FROM contrib_regression-# dblink_send_query('dtest1', 'select * from foo where f1 < 3; select * from foo where f1 > 6') AS t1; t1 ---- 1 (1 row) contrib_regression=# SELECT * FROM dblink_get_result('dtest1') AS t1(f1 int, f2 text, f3 text[]); f1 | f2 | f3 ----+----+------------ 0 | a | {a0,b0,c0} 1 | b | {a1,b1,c1} 2 | c | {a2,b2,c2} (3 rows) contrib_regression=# SELECT * FROM dblink_get_result('dtest1') AS t1(f1 int, f2 text, f3 text[]); f1 | f2 | f3 ----+----+--------------- 7 | h | {a7,b7,c7} 8 | i | {a8,b8,c8} 9 | j | {a9,b9,c9} 10 | k | {a10,b10,c10} (4 rows) contrib_regression=# SELECT * FROM dblink_get_result('dtest1') AS t1(f1 int, f2 text, f3 text[]); f1 | f2 | f3 ----+----+---- (0 rows) PostgreSQL 9.2.7 2014-02-17 DBLINK_GET_RESULT(3)
All times are GMT -4. The time now is 09:30 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy