December 12, 2010

ÖlaufmKeks ÖlaufmKeks
Lab Rat
1 posts

Content-Encoding: Gzip with Qt Webkit?

 

Hi everyone :),
I try to open a website which only supports “Content-Encoding: gzip”. A custom made header with “Accept-Encoding: Identity” is completly ignored by the webserver…
If I try to open this site in a QTWebKit widget… Guess what… Nothing happens :). I already recompiled QtWebKit in its newest version but the site still won’t be rendered by WebKit :(…
Is gzipped encoded content not supported in QTWebKit? Or is there another way to show this specific page with WebKit?
Thanks in advance for every answer :)…

8 replies

December 12, 2010

fcrochik fcrochik
Lab Rat
447 posts

Have you tried using firefox with firebug to get all the headers sent and received?
The problem may be with the server filtering the user agent or something else (not the gzip content per se).

 Signature 

Certified Specialist & Qt Ambassador Maemo, Meego, Symbian, Playbook, RaspberryPi, Desktop… Qt everywhere!

December 12, 2010

Bradley Bradley
Lab Rat
300 posts

Chrome also has some great features for web developers, under the Tools menu and off the Inspect element in the context menu.

 Signature 

Nokia Certified Qt Specialist.

January 8, 2011

ÖlaufmKeks ÖlaufmKeks
Lab Rat
1 posts

First of all thanks four your answers so far :)…
Secondly, sorry for my late answer, another project took too much time.

So, I’m still stuck with this WebKit problem. I already recompiled WebKit and tried the QtTestBrowser on this page. QtTestBrowser throws a “Frame load interrupted by policy change”, even if I change the User-Agent to Opera or any other (every other page works fine)…

Then I tried curl. All I get is scrambled output which looks like gzip encoded data.
If I run “curl —compressed http://specificsite” everything looks good with nice html output.

This page works fine with midori, chrome, opera, firefox. But aurora won’t render this page too…

So I really guess this is a content-encoding: gzip problem… I’m thankful for every tip!!!

January 8, 2011

Benjamin Poulain Benjamin Poulain
Lab Rat
144 posts

Any chance to get and example to reproduce that?

January 9, 2011

shayz shayz
Lab Rat
1 posts

Hi,

I have something of the same issue but I think Qt WebKit supports gzip but takes for ever to end the http load operation. For example, www.weather.com in Arora, takes 10-20 minutes to load.
After some investigations, it seems that only when the underlying TCP connection closes, Qt Network (qhttpnetworkreply.cpp and qhttpnetworkconnectionchannel.cpp) ‘understands’ that the page has been fully received. Qt Network disregards the content length header if the page is gzipped –
QHttpNetworkReplyPrivate::removeAutoDecompressHeader is called when the content encoding is gzip.

—shay

June 29, 2011

Thijs Thijs
Lab Rat
10 posts

I’ve run into the same problem and come to the same conclusion:

  • QtWebKit doesn’t support (all?) gzip content
  • This is because it strips the content length is removed by QtNetwork (as shayz mentioned) or because of missing end-of-stream marker (as mentioned in the bug reports below).
  • For most websites you can work around this issue by using a custom NetworkAccessManager and using : req.setRawHeader(“Accept-Encoding”, “none”) inside createRequest() function

Unfortunately there are some websites that always return gzipped content. Those are FUBAR, untill QtWebKit gets fixed :(

June 29, 2011

Thijs Thijs
Lab Rat
10 posts

For future visitors, there are some relevant bug reports:

https://bugs.webkit.org/show_bug.cgi?id=58727
http://bugreports.qt.nokia.com/browse/QTBUG-16022

June 30, 2011

Thijs Thijs
Lab Rat
10 posts

I made a small test-case which allowed me to find the cause. From the source code of qhttpnetworkconnection.cpp :

  1.     value = request.headerField("accept-encoding");
  2.     if (value.isEmpty()) {
  3. #ifndef QT_NO_COMPRESS
  4.         request.setHeaderField("Accept-Encoding", "gzip");
  5.         request.d->autoDecompress = true;
  6. #else
  7.         // if zlib is not available set this to false always
  8.         request.d->autoDecompress = false;
  9. #endif
  10.     }

I believe this is a bug: If you manually set the “Accept-Encoding” header, autoDecompress becomes false (even if you set it to “gzip”). So QtWebKit will never decompress the data and it will not be able to render the page obviously.

[EDIT: code formatting, please use @-tags, not code, Volker]

 
  ‹‹ QGraphicswebview and plugin      [Solved] Debugging scripts that run in a QWebView widget ››

You must log in to post a reply. Not a member yet? Register here!