August 4, 2011

victoryfox victoryfox
Lab Rat
8 posts

[solved] tableWidget building time 45sec (85000 rows, 1 for loop). How to optimize build time?

 

Greatings everyone,

I have a little qt project, a LogViewer : reads a .log file (wich can have cca 100-150mb: about million+ lines), parses the file and then builds the gui.

So you see, the app is divided in two parts (parsing data & filling the table). I have done that to see which part needs to be optimized better. I have no backend database, only one colleciton of objects.

My two main methods are:

void ParseLog(QString strInputFile); -> parser
void RefreshLog(); -> tableWidget builder

Parser has only one loop (while) – reads the entire file line by line and parses needed data
Builder has only one loop (for) – for every object i have in my collection stored, create a row

My parsing method for a file of 130mb (cca 1.222.212 lines) lasts about 20sec, which is ok, eventually i will optimize the parser further.

The problem is the GUI building method “RefreshLog()”

  1. void Logviewer::RefreshLog()
  2. {
  3.  startGui = QDateTime::currentDateTime();
  4.  int nlastRow = 0;
  5.  QTableWidgetItem *itemId;
  6.  QTableWidgetItem *itemCode;
  7.  QTableWidgetItem *itemCallResult;
  8.  QTableWidgetItem *itemClientIP;
  9.  QTableWidgetItem *itemServerIP;
  10.  QTableWidgetItem *itemCallLength;
  11.  QTableWidgetItem *itemDate;
  12.  
  13.  for(int i=0; i<m_lstLogData.size(); i++)
  14.  {
  15.  
  16.   itemId = new QTableWidgetItem();
  17.   itemCode = new QTableWidgetItem();
  18.   itemCallResult = new QTableWidgetItem();
  19.   itemClientIP = new QTableWidgetItem();
  20.   itemServerIP = new QTableWidgetItem();
  21.   itemCallLength = new QTableWidgetItem();
  22.   itemDate = new QTableWidgetItem();
  23.  
  24.   itemId->setData(Qt::DisplayRole, QVariant(m_lstLogData[i].strID));
  25.   itemCode->setData(Qt::DisplayRole, QVariant(m_lstLogData[i].strCode));
  26.   itemCallResult->setData(Qt::DisplayRole, QVariant(m_lstLogData[i].strResult));
  27.   itemClientIP->setData(Qt::DisplayRole, QVariant(m_lstLogData[i].strClientIP));
  28.   itemServerIP->setData(Qt::DisplayRole, QVariant(m_lstLogData[i].strServerIP));
  29.   itemCallLength->setData(Qt::DisplayRole, QVariant(m_lstLogData[i].strCallLength));
  30.   itemDate->setData(Qt::DisplayRole, QVariant(m_lstLogData[i].strDate));
  31.  
  32.   ui.tableWidget->insertRow(nlastRow);
  33.  
  34.   ui.tableWidget->setItem(nlastRow,0, itemId);  
  35.   ui.tableWidget->setItem(nlastRow,1, itemCode);    
  36.   ui.tableWidget->setItem(nlastRow,2, itemCallResult);
  37.   ui.tableWidget->setItem(nlastRow,3, itemClientIP);
  38.   ui.tableWidget->setItem(nlastRow,4, itemServerIP);
  39.   ui.tableWidget->setItem(nlastRow,5, itemCallLength);
  40.   ui.tableWidget->setItem(nlastRow,6, itemDate);
  41.  
  42.   nlastRow++;
  43.  }
  44.  endGui = QDateTime::currentDateTime();
  45. }

This function right above takes about 45 seconds for 85000 rows, 85000 loops and for every loop doing:

  • new QTableWidgetItem()
  • setData
  • insertRow
  • setItem

Which i believe some cost too much.

Can you please give me advice on how to optimize this, or should i move to QTableView?

I haven’t done model-view programming, so if it is gonna be much faster and better, i will need to switch then.

Any advice for mobel-view for my project will be helpfull, i will start with some model-view tutorials for now (should this be correct materials for my need? [qt.nokia.com])

Or can i stay with the simplier widget and optimize it a little?

THANKS
victoryfox

5 replies

August 4, 2011

loladiro loladiro
Lab Rat
596 posts

For data sets that large Model/View is the only way. There is a nice tutorial on the docs [doc.qt.nokia.com]

August 4, 2011

Gerolf Gerolf
Robot Herder
3287 posts

You should definitely switch to model/view.
We also have some log tools here and we build them completely by model/view (but it’s closed source…). Just take care that you insert all rows and then update the view and do not update the view after each row. otherwise you will have the same problems as now. Adding data to your internal storage in the model will be fast, UI updates are cost intensive.

Your internal model might directly use your existing structure (m_lstLogData) and just give access to it, that’s what we do with huge models.

 Signature 

Nokia Certified Qt Specialist.
Programming Is Like Sex: One mistake and you have to support it for the rest of your life. (Michael Sinz)

August 4, 2011

victoryfox victoryfox
Lab Rat
8 posts
loladiro wrote:
For data sets that large Model/View is the only way. There is a nice tutorial on the docs [doc.qt.nokia.com]

thanks

August 4, 2011

victoryfox victoryfox
Lab Rat
8 posts

Gerolf wrote:
You should definitely switch to model/view.
We also have some log tools here and we build them completely by model/view (but it’s closed source…). Just take care that you insert all rows and then update the view and do not update the view after each row. otherwise you will have the same problems as now. Adding data to your internal storage in the model will be fast, UI updates are cost intensive.

Your internal model might directly use your existing structure (m_lstLogData) and just give access to it, that’s what we do with huge models.

Then i will go with model-view implementation. I’m not sure if the automatic sorting capabilities of the column headers are any good for a large number of rows, but in the tableWidget you wish you have not pressed the header cell for sorting :) – maybe i will implement a custom “Filter & Search” option lateron in model-view.

Thanks for the good advices. Do i need to tag this topic as closed or something?

Greetings,
victoryfox

August 4, 2011

Eddy Eddy
Area 51 Engineer
1612 posts

Do i need to tag this topic as closed or something?

Yes, please could you edit your title and add [solved] in front of it. There are no other means at this moment, but the Trolls are implementing a way now.

 Signature 

Moderator
Qt Certified Specialist
Nokia Qt Ambassador

 
  ‹‹ QVTKWidget Error!! Please Help      How to make QListview and QFilesystemModel item with multiple line text? ››

You must log in to post a reply. Not a member yet? Register here!