Download with HTTP crashes files - how to fix it?

[_DICE_]

Hi,
I try to download some files, mostly archives (*.7z, *.rar), using HTTP.
My code for it is:

void downloader::Do_Download()
{
    http = new QHttp(this);

    connect(http, SIGNAL(stateChanged(int)), this, SLOT(stateChanged(int)));
    connect(http, SIGNAL(responseHeaderReceived(QHttpResponseHeader)), this, SLOT(responseHeaderReceived(QHttpResponseHeader)));
    connect(http, SIGNAL(requestFinished(int,bool)), this, SLOT(requestFinished(int,bool)));

    QString host = "***"; //
    qDebug() << "Host: " + host;

    QString dl = "***";
    qDebug() << "Datei: " + dl;


    http->setHost(host);
    http->get(dl);
}

void downloader::stateChanged ( int state )
{
    switch (state)
    {
        case 0:
            qDebug() << "Unconnected";
            break;
        case 1:
            qDebug() << "Hhost Lookup";
            break;
        case 2:
            qDebug() << "Connecting";
            break;
        case 3:
            qDebug() << "Sending";
            break;
        case 4:
            qDebug() << "Reading";
            break;
        case 5:
            qDebug() << "Connect";
            break;
        case 6:
            qDebug() << "Close";
            break;
    }
}

void downloader::responseHeaderReceived ( const QHttpResponseHeader & resp )
{
    qDebug() << "Size" << resp.contentLength();
    qDebug() << "Type" << resp.contentType();
    qDebug() << "Status" << resp.statusCode();
    qDebug() << "Content" << resp.toString();

    QFile file("C:/html-test/resp.txt");
    if (file.open(QIODevice::WriteOnly))
    {
        QTextStream stream(&file);
        stream << resp.toString();
        file.close();
    }
}

void downloader::requestFinished ( int id, bool error )
{
    if (error)
    {
        qDebug() << "ERROR";
    }
    else
    {
        qDebug() << "OK";

        QFile file("C:/html-test/test.rar");
        if (file.open(QIODevice::WriteOnly))
        {
            QTextStream stream(&file);
            stream << http->readAll();
            file.close();
        }
    }
}

It does download the archives, but it crashes the files. WinRAR reports that the file headers are corrupt and 7Zip reports that the hole archive is corrupt.

So I downloaded some archives using download managers contained in Webbrowsers like Firefox and Opera. I compared my "corrupted" archives with them from the browsers and found out:
The archives are really corrupt, there are differences at some bytes...

The questions are:

1. Is it at server problem (do not think so)?
2. Is it impossible to use this code to download archives (seems to work with "text" files like .txt or .php)?
3. If it is possible: What do I have to change to get it working?

Maybe these are dumb questions but downloads are a absolutely new topic for me.

[kunal_the_one]

I had created download manager with QNetworkAccessManager. Here is link

http://kunalmaemo.blogspot.com/2011/...ith-pause.html

you can try this code and check if it works. And one more thing to write binary data try to use QDataStream instead of QTextStrem class.

[_DICE_]

Thank you, the hint with the QDataStream class helped - it works now.

[kunal_the_one]

I was suspecting QTextStream for cause, Glad that you solved your problem.

[_DICE_]

Again I have a problem with my downloads.
I moved the downloads in a separate thread and the result is:

"the url I entered"
"host extracted from the url"
"file extracted from the url"
Fertig -> fired by requestFinished
Connecting
Sending
Reading
Size 0
Type "text/html"
Status 302
Inhalt "HTTP/1.1 302 Moved Temporarily
Date: Sat, 07 Jan 2012 19:42:21 GMT
Server: Apache
Set-Cookie: 00016ec1e4295b42ce874cf12ffd17b1=-; path=/
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Last-Modified: Sat, 07 Jan 2012 19:42:21 GMT
Cache-Control: no-store, no-cache, must-revalidate
Cache-Control: post-check=0, pre-check=0
Pragma: no-cache
Location: here I get a new url
Vary: Accept-Encoding
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/html

"
"neue url: url extracted from response
"
Connected
Fertig -> ok we're at the end, so this may be correct
Close
Unconnected

downloadthread.h

#ifndef DOWNLOADTHREAD_H
#define DOWNLOADTHREAD_H

#include <QThread>
#include <QHttp>

class DownloadThread : public QThread
{
    Q_OBJECT
public:
    explicit DownloadThread(QObject *parent = 0);

    void run();

    QString url;

    int step;

signals:

public slots:
    void stateChanged(int state);
    void responseHeaderReceived(const QHttpResponseHeader &resp);
    void dataReadProgress(int read, int total);
    void requestFinished(int id, bool error);

private:
    QHttp *http;

private slots:
    void Do_Download();

};

#endif // DOWNLOADTHREAD_H

downloadthread.cpp

#include "downloadthread.h"

#include <QDebug>
#include <QHttp>
#include <QFile>
#include <QApplication>

DownloadThread::DownloadThread(QObject *parent) :
    QThread(parent)
{
}

void DownloadThread::run()
{
    Do_Download();
}

void DownloadThread::Do_Download()
{
    qDebug() << url;

    if (url.startsWith("http://"))
    {
        url.remove("http://");
    }

    http = new QHttp();
    http->moveToThread(QApplication::instance()->thread());
    connect(http, SIGNAL(stateChanged(int)), this, SLOT(stateChanged(int)));
    connect(http, SIGNAL(responseHeaderReceived(QHttpResponseHeader)), this, SLOT(responseHeaderReceived(QHttpResponseHeader)));
    connect(http, SIGNAL(dataReadProgress(int,int)), this, SLOT(dataReadProgress(int, int)));
    connect(http, SIGNAL(requestFinished(int,bool)), this, SLOT(requestFinished(int,bool)));

    QString host = url.mid(0, url.indexOf("/"));
    QString dl = url.mid(url.indexOf("/"));

    qDebug() << host;
    qDebug() << dl;

    http->setHost(host);
    http->get(dl);
}

void DownloadThread::stateChanged(int state)
{
    switch (state)
    {
        case 0:
            qDebug() << "Unconnected";
            break;
        case 1:
            qDebug() << "Host Lookup";
            break;
        case 2:
            qDebug() << "Connecting";
            break;
        case 3:
            qDebug() << "Sending";
            break;
        case 4:
            qDebug() << "Reading";
            break;
        case 5:
            qDebug() << "Connected";
            break;
        case 6:
            qDebug() << "Close";
            this->terminate();
            break;
    }
}

void DownloadThread::responseHeaderReceived(const QHttpResponseHeader &resp)
{
    qDebug() << "Size" << resp.contentLength();
    qDebug() << "Type" << resp.contentType();
    qDebug() << "Status" << resp.statusCode();
    qDebug() << "Inhalt" << resp.toString();

    if (resp.toString().contains("Location: "))
    {
        QStringList splitted1 = resp.toString().split("Location: ");
        QStringList splitted2 = splitted1.at(1).split("Vary");

        url = splitted2.at(0);
        qDebug() << "neue url: " + url;
    }
}

void DownloadThread::dataReadProgress(int read, int total)
{
    //qDebug() << read + "/" + total;
}

void DownloadThread::requestFinished(int id, bool error)
{
    if (error)
    {
        qDebug() << "Fehler";
    }
    else
    {
        qDebug() << "Fertig";
        QFile file(QApplication::applicationDirPath() + "/test.7z");
        if (file.open(QIODevice::WriteOnly))
        {
            QTextStream stream(&file);
            stream << http->readAll();
            file.close();
        }

    }
}

My are questions are now: Why is requestFinished fired before it has connected to the host? How can I fix this?

And my problem:
I have to send 3 requests. The first contains a url you can find at a website (request 1).
The response header gives me another url which I have to send (request 2). Again I get another url in the response header. This is the "real" url.
Now I have to request the "real" url, which allows me to download the file.

This should not be a problem. Normally I would call do it like this:

1. set url-var with thread->url = "bla"
2. start thread
3. call Do_Download from "case 6" of stateChanged because here I should have a new url

But this is the debug log:

"url I set myself"
"host extracted from url"
"file extracted from url"
Fertig -> Again that false one
Connecting
Sending
Reading
Size 0
Type "text/html"
Status 302
Inhalt "HTTP/1.1 302 Moved Temporarily
Date: Sat, 07 Jan 2012 19:51:12 GMT
Server: Apache
Set-Cookie: 00016ec1e4295b42ce874cf12ffd17b1=-; path=/
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Last-Modified: Sat, 07 Jan 2012 19:51:12 GMT
Cache-Control: no-store, no-cache, must-revalidate
Cache-Control: post-check=0, pre-check=0
Pragma: no-cache
Location: here I get the new url
Vary: Accept-Encoding
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/html

"
"neue url: my new url from response header (and it seems to be correct
"
Connected
Fertig
Close

(request 2 starts here)
"the url I got from response header"
"host extraction works again"
"file extraction works again too"
Fertig -> *ugh*
Connecting
Unconnected
Sending
Reading
Close -> What happend here? I did not get a response...

I hope someone can help me with my two little problems because I tried to find the solution for a hole week now.
Of course I could start the thread 3 times to get the urls and the files, but I think this is not a real solution.

[kunal_the_one]

have you checked my download manager implementation. That might give you some hints.

[_DICE_]

It seems to be, more or less, the same. The only difference is you use Networkmanager and you have a pause/resume function. I will try to rewrite my download function with Networkmanager instead of Http.

But I wonder why my function does not work anymore. It worked while I called it directly from the the gui thread. Now, after moving it in a separate thread, I get these errors. So I think it has something to do with the threads and I want to know what exactly the problems because I do not like it if something does not work and I do not know why.