Download with HTTP crashes files - how to fix it?
Hi,
I try to download some files, mostly archives (*.7z, *.rar), using HTTP.
My code for it is:
[CODE]
void downloader::Do_Download()
{
http = new QHttp(this);
connect(http, SIGNAL(stateChanged(int)), this, SLOT(stateChanged(int)));
connect(http, SIGNAL(responseHeaderReceived(QHttpResponseHeader)), this, SLOT(responseHeaderReceived(QHttpResponseHeader)));
connect(http, SIGNAL(requestFinished(int,bool)), this, SLOT(requestFinished(int,bool)));
QString host = "***"; //
qDebug() << "Host: " + host;
QString dl = "***";
qDebug() << "Datei: " + dl;
http->setHost(host);
http->get(dl);
}
void downloader::stateChanged ( int state )
{
switch (state)
{
case 0:
qDebug() << "Unconnected";
break;
case 1:
qDebug() << "Hhost Lookup";
break;
case 2:
qDebug() << "Connecting";
break;
case 3:
qDebug() << "Sending";
break;
case 4:
qDebug() << "Reading";
break;
case 5:
qDebug() << "Connect";
break;
case 6:
qDebug() << "Close";
break;
}
}
void downloader::responseHeaderReceived ( const QHttpResponseHeader & resp )
{
qDebug() << "Size" << resp.contentLength();
qDebug() << "Type" << resp.contentType();
qDebug() << "Status" << resp.statusCode();
qDebug() << "Content" << resp.toString();
QFile file("C:/html-test/resp.txt");
if (file.open(QIODevice::WriteOnly))
{
QTextStream stream(&file);
stream << resp.toString();
file.close();
}
}
void downloader::requestFinished ( int id, bool error )
{
if (error)
{
qDebug() << "ERROR";
}
else
{
qDebug() << "OK";
QFile file("C:/html-test/test.rar");
if (file.open(QIODevice::WriteOnly))
{
QTextStream stream(&file);
stream << http->readAll();
file.close();
}
}
}
[/CODE]
It does download the archives, but it crashes the files. WinRAR reports that the file headers are corrupt and 7Zip reports that the hole archive is corrupt.
So I downloaded some archives using download managers contained in Webbrowsers like Firefox and Opera. I compared my "corrupted" archives with them from the browsers and found out:
The archives are really corrupt, there are differences at some bytes...
The questions are:
[LIST=1][*]Is it at server problem (do not think so)?[*]Is it impossible to use this code to download archives (seems to work with "text" files like .txt or .php)?[*]If it is possible: What do I have to change to get it working?[/LIST]
Maybe these are dumb questions but downloads are a absolutely new topic for me.
Re: Download with HTTP crashes files - how to fix it?
I had created download manager with QNetworkAccessManager. Here is link
[url]http://kunalmaemo.blogspot.com/2011/07/simple-download-manager-with-pause.html[/url]
you can try this code and check if it works. And one more thing to write binary data try to use QDataStream instead of QTextStrem class.
Re: Download with HTTP crashes files - how to fix it?
Thank you, the hint with the QDataStream class helped - it works now.
Re: Download with HTTP crashes files - how to fix it?
I was suspecting QTextStream for cause, Glad that you solved your problem.
Re: Download with HTTP crashes files - how to fix it?
Again I have a problem with my downloads.:mad:
I moved the downloads in a separate thread and the result is:
[QUOTE=debug log]
"the url I entered"
"host extracted from the url"
"file extracted from the url"
Fertig -> fired by requestFinished
Connecting
Sending
Reading
Size 0
Type "text/html"
Status 302
Inhalt "HTTP/1.1 302 Moved Temporarily
Date: Sat, 07 Jan 2012 19:42:21 GMT
Server: Apache
Set-Cookie: 00016ec1e4295b42ce874cf12ffd17b1=-; path=/
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Last-Modified: Sat, 07 Jan 2012 19:42:21 GMT
Cache-Control: no-store, no-cache, must-revalidate
Cache-Control: post-check=0, pre-check=0
Pragma: no-cache
Location: [B]here I get a new url[/B]
Vary: Accept-Encoding
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/html
"
"neue url: url extracted from response
"
Connected
Fertig -> ok we're at the end, so this may be correct
Close
Unconnected
[/QUOTE]
downloadthread.h
[CODE]
#ifndef DOWNLOADTHREAD_H
#define DOWNLOADTHREAD_H
#include <QThread>
#include <QHttp>
class DownloadThread : public QThread
{
Q_OBJECT
public:
explicit DownloadThread(QObject *parent = 0);
void run();
QString url;
int step;
signals:
public slots:
void stateChanged(int state);
void responseHeaderReceived(const QHttpResponseHeader &resp);
void dataReadProgress(int read, int total);
void requestFinished(int id, bool error);
private:
QHttp *http;
private slots:
void Do_Download();
};
#endif // DOWNLOADTHREAD_H
[/CODE]
downloadthread.cpp
[CODE]
#include "downloadthread.h"
#include <QDebug>
#include <QHttp>
#include <QFile>
#include <QApplication>
DownloadThread::DownloadThread(QObject *parent) :
QThread(parent)
{
}
void DownloadThread::run()
{
Do_Download();
}
void DownloadThread::Do_Download()
{
qDebug() << url;
if (url.startsWith("http://"))
{
url.remove("http://");
}
http = new QHttp();
http->moveToThread(QApplication::instance()->thread());
connect(http, SIGNAL(stateChanged(int)), this, SLOT(stateChanged(int)));
connect(http, SIGNAL(responseHeaderReceived(QHttpResponseHeader)), this, SLOT(responseHeaderReceived(QHttpResponseHeader)));
connect(http, SIGNAL(dataReadProgress(int,int)), this, SLOT(dataReadProgress(int, int)));
connect(http, SIGNAL(requestFinished(int,bool)), this, SLOT(requestFinished(int,bool)));
QString host = url.mid(0, url.indexOf("/"));
QString dl = url.mid(url.indexOf("/"));
qDebug() << host;
qDebug() << dl;
http->setHost(host);
http->get(dl);
}
void DownloadThread::stateChanged(int state)
{
switch (state)
{
case 0:
qDebug() << "Unconnected";
break;
case 1:
qDebug() << "Host Lookup";
break;
case 2:
qDebug() << "Connecting";
break;
case 3:
qDebug() << "Sending";
break;
case 4:
qDebug() << "Reading";
break;
case 5:
qDebug() << "Connected";
break;
case 6:
qDebug() << "Close";
this->terminate();
break;
}
}
void DownloadThread::responseHeaderReceived(const QHttpResponseHeader &resp)
{
qDebug() << "Size" << resp.contentLength();
qDebug() << "Type" << resp.contentType();
qDebug() << "Status" << resp.statusCode();
qDebug() << "Inhalt" << resp.toString();
if (resp.toString().contains("Location: "))
{
QStringList splitted1 = resp.toString().split("Location: ");
QStringList splitted2 = splitted1.at(1).split("Vary");
url = splitted2.at(0);
qDebug() << "neue url: " + url;
}
}
void DownloadThread::dataReadProgress(int read, int total)
{
//qDebug() << read + "/" + total;
}
void DownloadThread::requestFinished(int id, bool error)
{
if (error)
{
qDebug() << "Fehler";
}
else
{
qDebug() << "Fertig";
QFile file(QApplication::applicationDirPath() + "/test.7z");
if (file.open(QIODevice::WriteOnly))
{
QTextStream stream(&file);
stream << http->readAll();
file.close();
}
}
}
[/CODE]
My are questions are now: Why is requestFinished fired before it has connected to the host? How can I fix this?
And my problem:
I have to send 3 requests. The first contains a url you can find at a website (request 1).
The response header gives me another url which I have to send (request 2). Again I get another url in the response header. This is the "real" url.
Now I have to request the "real" url, which allows me to download the file.
This should not be a problem. Normally I would call do it like this:
[LIST=1][*]set url-var with thread->url = "bla"[*]start thread[*]call Do_Download from "case 6" of [I]stateChanged[/I] because here I should have a new url[/LIST]
But this is the debug log:
[QUOTE]
"url I set myself"
"host extracted from url"
"file extracted from url"
Fertig -> Again that false one
Connecting
Sending
Reading
Size 0
Type "text/html"
Status 302
Inhalt "HTTP/1.1 302 Moved Temporarily
Date: Sat, 07 Jan 2012 19:51:12 GMT
Server: Apache
Set-Cookie: 00016ec1e4295b42ce874cf12ffd17b1=-; path=/
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Last-Modified: Sat, 07 Jan 2012 19:51:12 GMT
Cache-Control: no-store, no-cache, must-revalidate
Cache-Control: post-check=0, pre-check=0
Pragma: no-cache
Location: [B]here I get the new url[/B]
Vary: Accept-Encoding
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/html
"
"neue url: my new url from response header (and it seems to be correct
"
Connected
Fertig
Close
(request 2 starts here)
"the url I got from response header"
"host extraction works again"
"file extraction works again too"
Fertig -> *ugh*
Connecting
Unconnected
Sending
Reading
Close -> What happend here? I did not get a response...
[/QUOTE]
I hope someone can help me with my two little problems because I tried to find the solution for a hole week now.
Of course I could start the thread 3 times to get the urls and the files, but I think this is not a real solution.
Re: Download with HTTP crashes files - how to fix it?
have you checked my download manager implementation. That might give you some hints.
Re: Download with HTTP crashes files - how to fix it?
It seems to be, more or less, the same. The only difference is you use Networkmanager and you have a pause/resume function. I will try to rewrite my download function with Networkmanager instead of Http.
But I wonder why my function does not work anymore. It worked while I called it directly from the the gui thread. Now, after moving it in a separate thread, I get these errors. So I think it has something to do with the threads and I want to know what exactly the problems because I do not like it if something does not work and I do not know why.