The_Great_Archive/libretroshare
1
0
Fork 0
mirror of https://github.com/RetroShare/libretroshare.git synced 2026-09-12 19:50:03 +05:00
Code Issues Packages Projects Releases Wiki Activity

GXS: batch message retrieval into IN(...) queries instead of one per message

Browse Source 
RsDataService::retrieveNxsMsgs() issued one prepared statement per requested
message id. Each one rebuilds the SQL text, runs a full sqlite3_prepare_v2
(SQL parse plus query planner), allocates a cursor and finalizes it -- an
overhead that dominates the actual row lookup, and is paid thousands of times
whenever a request covers a large id set.

Pack the ids into "msgId IN (...)" batches of 500 instead. The message ids are
plain hex strings so they need no escaping, and the batch size keeps both the
generated SQL and sqlite's expression tree small.

The previous commit removed this path for unfiltered whole-group requests;
this one covers everything else: a channel post with its comments, forum
threads, and any filtered request.

retrieveGxsMsgMetaData() still has the same one-query-per-id shape in its
non-empty branch, but there each id is first looked up in the meta cache, so
the remaining queries are only the cache misses. Left alone for now so it can
be measured on its own.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
jolavillette 2026-07-27 10:52:45 +02:00
parent f9b90cb26b
commit 4a5e8f79ac
1 changed files with 26 additions and 7 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

33
src/gxs/rsdataservice.cc
View File

@ -47,6 +47,11 @@
#define MSG_INDEX_GRPID std::string("INDEX_MESSAGES_GRPID")
// Maximum number of message ids packed into a single "IN (...)" clause. Keeps
// the generated SQL and sqlite's expression tree to a sane size while making
// the per-statement preparation cost negligible.
static const uint32_t MAX_MSG_IDS_PER_QUERY = 500;
// generic
#define KEY_NXS_DATA std::string("nxsData")
#define KEY_NXS_DATA_LEN std::string("nxsDataLen")
@ -1205,14 +1210,28 @@ int RsDataService::retrieveNxsMsgs(const GxsMsgReq &reqIds, GxsMsgResult &msg,
{
RS_STACK_MUTEX(mDbMutex);
// request each grp
for( std::set<RsGxsMessageId>::const_iterator sit = msgIdV.begin();
sit!=msgIdV.end();++sit )
{
const RsGxsMessageId& msgId = *sit;
// Retrieve the requested messages in batches, using a single
// "msgId IN (...)" clause per batch. Preparing one statement per
// message dominates the cost as soon as a request covers more than
// a handful of them (a channel post and its comments, a forum
// thread, a filtered group request).
RetroCursor* c = mDb->sqlQuery(MSG_TABLE_NAME, withMeta ? mMsgColumnsWithMeta : mMsgColumns, KEY_GRP_ID+ "='" + grpId.toStdString()
+ "' AND " + KEY_MSG_ID + "='" + msgId.toStdString() + "'", "");
const std::string selection_prefix = KEY_GRP_ID + "='" + grpId.toStdString()
+ "' AND " + KEY_MSG_ID + " IN (";
for(auto sit = msgIdV.begin(); sit != msgIdV.end(); )
{
std::string selection = selection_prefix;
for(uint32_t i=0; i<MAX_MSG_IDS_PER_QUERY && sit!=msgIdV.end(); ++i,++sit)
{
if(i > 0) selection += ",";
selection += "'" + sit->toStdString() + "'";
}
selection += ")";
RetroCursor* c = mDb->sqlQuery(MSG_TABLE_NAME, withMeta ? mMsgColumnsWithMeta : mMsgColumns, selection, "");
++prof_queries;
if(c)