January 26, 2011

Kxyu Kxyu
Lab Rat
145 posts

QNetworkAccessManager doesn’t download whole web page

 

Hi. I’m trying to donwload page via http using QNetworkAccessManager
I store data in QByteArray and append readAll() from respective QNetworkReply object on each call of readyRead signal

but when I check QByteArray content on finished signal of QNetworkAccessManager, it contains not whole page, it is cut in a certain place.
I see there is a kind of limitation, but I can’t find the way to avoid it

8 replies

January 26, 2011

Volker Volker
Robot Herder
5428 posts

Did you really read the last chunk?

finished() is emitted as soon as the request is done from a protocol perspective. You may have pending data in the reply that you have not read yet. It’s good practice to call readAll() in your slot connected to finished() a last time to get the last chunk of data.

If your data is not big or you do not need the data chunkwise as soon as possible, you can have QNetworkReply collect the data and read the whole page after finished() with readAll(). See QNetworkReply::finished() [doc.qt.nokia.com] for some details.

January 26, 2011

Kxyu Kxyu
Lab Rat
145 posts

I tried doing only readAll() on QNetworkAccessManager finished signal, and it downloads the same exact piece of page!
is there any significant difference between QNetworkAccessManager finished and QNetworkReply finished?

January 26, 2011

Volker Volker
Robot Herder
5428 posts

The finished signal of QNetworkReply is connected to a private slot of QNetworkAccessManager which in turn immediately emits finished(QNetworkReply*). So, no, there is no significant difference.

As things work for most of the other people, it would be handy, if you could provide us a small, complete, compilable and running example that demonstrates the error and that we can run on our boxes.

January 26, 2011

Kxyu Kxyu
Lab Rat
145 posts

Well thr code is as following

  1. void bot::recieveId() // this function initiates request
  2. {
  3.     connect(manager,SIGNAL(finished(QNetworkReply*)),this,SLOT(idRecieved(QNetworkReply*)));
  4.     manager->get(QNetworkRequest(QUrl("http://qt.nokia.com")));//it's not actual URL, but problem is the same
  5. }
  6.  
  7. void bot::idRecieved(QNetworkReply *_reply) // this slot dumps the whole data
  8. {
  9.     rawData=_reply->readAll();
  10.     qDebug()<<rawData;
  11.     disconnect(manager,SIGNAL(finished(QNetworkReply*)),this,SLOT(idRecieved(QNetworkReply*)));
  12. }

manager and rawData are QNetworkAccessManager (instantiated in counstructor) and QByteArray (static)
the last thing I have in console is
  1. <li><a href="http://qt.nokia.com/developer/eclipse-integration">Eclipse plug-in</a></li>

while there is a hundred or more strings after

January 26, 2011

Volker Volker
Robot Herder
5428 posts

I just tried and it works for me. I get the complete page.

I’m on Mac OS X 10.6, Qt 4.7.0.

Do you have any proxy settings in your program? Or some corporate hidden proxy inbetween? Can you get the page correctly with another tool on the command line, eg.

  1. curl http://qt.nokia.com

January 26, 2011

Kxyu Kxyu
Lab Rat
145 posts

well, I think I found the answer. I tried some operations with rawData (like finding </html> tag) and everything was ok. I guess it’s just a limitation of qDebug()

October 25, 2011

mady mady
Lab Rat
61 posts

Hi volker,

I am using QNAM to download a XML file from server but getting “connection timed out” error message in reply. In your previous post you were asking whether Kxyu is using proxy in his program. My program is behind a corporate proxy, can it cause this issue? If yes, could you please let me know how to resolve the problem?

Note: Through browser I can access the page successuflly

October 25, 2011

peppe peppe
Ant Farmer
1025 posts

What’s the point of necroposting? Please, start a brand new thread with all the relevant information.

 Signature 

Software Engineer
KDAB (UK) Ltd., a KDAB Group company

 
  ‹‹ Hidden information in a QTextDocument      Not able to view the multiple instances of my application in Task Manager & Taskbar ››

You must log in to post a reply. Not a member yet? Register here!