javascript - How to handle downloads with phantomjscasperjs? - Stack Overflow

Is it possible to download a file to a folder and give it a specific name with panthomjs/casperjs ?

For instance how can I download the .csv at the bottom of this page : and name it aapl.txt ?

The download link is :

<a href="javascript:getQuotes(true);" id="lnkDownLoad">
                Download this file in Excel Format
            </a>

Its goal is to call a javascript function whose goal is to obfuscate the direct download link (I think) but when you click on it it summons a classic download prompt. I would like phantomjs to handle that download normally (to change the name of the file and chose where to save it on the drive disk)

Edit : This code is supposed to click on the download link and listen for ining ressources :

var casper = require('casper').create();

var x = require('casper').selectXPath;

casper.userAgent("Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2049.0 Safari/537.36")

casper.start('', function () {
    //this.echo(this.getTitle());
    console.log('TITLE : ' + this.getTitle());

});

casper.wait(5000, function() {
    casper.on('resource.received', function (resource) {
        casper.echo("LISTENING");
        casper.echo(resource.url);
    });
});

casper.thenClick(x('//*[@id="lnkDownLoad"]'), function() {
    console.log('CLICKED');
});



casper.run();

But for some reason I dont receive any file unlike on a normal browser. The console log is : b'TITLE : (AAPL) Historical Prices & Data - NASDAQ\r\nCLICKED\r\nLISTENING\r\n\r\n'

Any idea ?

Is it possible to download a file to a folder and give it a specific name with panthomjs/casperjs ?

For instance how can I download the .csv at the bottom of this page : http://www.nasdaq./symbol/aapl/historical and name it aapl.txt ?

The download link is :

<a href="javascript:getQuotes(true);" id="lnkDownLoad">
                Download this file in Excel Format
            </a>

Its goal is to call a javascript function whose goal is to obfuscate the direct download link (I think) but when you click on it it summons a classic download prompt. I would like phantomjs to handle that download normally (to change the name of the file and chose where to save it on the drive disk)

Edit : This code is supposed to click on the download link and listen for ining ressources :

var casper = require('casper').create();

var x = require('casper').selectXPath;

casper.userAgent("Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2049.0 Safari/537.36")

casper.start('http://www.nasdaq./symbol/aapl/historical', function () {
    //this.echo(this.getTitle());
    console.log('TITLE : ' + this.getTitle());

});

casper.wait(5000, function() {
    casper.on('resource.received', function (resource) {
        casper.echo("LISTENING");
        casper.echo(resource.url);
    });
});

casper.thenClick(x('//*[@id="lnkDownLoad"]'), function() {
    console.log('CLICKED');
});



casper.run();

But for some reason I dont receive any file unlike on a normal browser. The console log is : b'TITLE : (AAPL) Historical Prices & Data - NASDAQ.\r\nCLICKED\r\nLISTENING\r\nhttp://www.nasdaq./symbol/aapl/historical\r\n'

Any idea ?

• javascript
• web-scraping
• phantomjs
• casperjs

Share Improve this question Follow edited Sep 8, 2014 at 2:40 Wicelo asked Sep 5, 2014 at 23:33 WiceloWicelo 2,42622 gold badges2929 silver badges5050 bronze badges 5

• did you try my link ? the direct download link is obfuscated with javascript. I need a way to handle the download prompt. – Wicelo Commented Sep 7, 2014 at 6:09
• the download link calls a javascript function named getQuoteswhose goal is to ofuscate the direct download link. I don't have the knoweldge to analyse what it does precisely. If you click on the link with firefox or chrome a download prompt appears and you can download a .csv. I basically would like phantomjs to handle that download prompt to rename the file and save it where I chose to, unless there are other ways. – Wicelo Commented Sep 7, 2014 at 8:33
• What happens when you try the linked solution? Are the any problems? – Artjom B. Commented Sep 7, 2014 at 8:43
• well yes when I use the method given in your link I don't get any resource I think, it looks like it just reload the page since after the "LISTENING" string in the log I just get the url of the page and not the expected .csv. I use casper.wait just to get rid of the page resources such as javascript files and so on in the logs. Also I dont really understand why there is the string CLICKED before LISTENING since I call it afterwards. – Wicelo Commented Sep 8, 2014 at 2:41
• Related: casperjs download csv file – Artjom B. Commented Sep 8, 2014 at 7:43

Add a ment  | 

1 Answer 1

Sorted by: Reset to default 4

When you look into the code, you see that it is not really obfuscated. By clicking on the download link, the file is actually downloaded through casper, but it cannot be easily accessed. The culprit is PhantomJS, because PhantomJS does not expose the request and response contents (see page.onResourceReceived), but only meta data.

It means that you need to repeat the request through the download function. When you look into the page source in your browser's developer tools, you see that getQuotes(true) is called on click. By searching for getQuotes (Ctrl+Shift+F in Chrome), you find the function in question.

By analysing the function, you could arrive at the conclusion that $("#getFile").submit(); is the actual download which is just a POST request from a form with a lot of hidden values. If you look closely into getQuotes, you see that the function also adds one of the hidden values to the form. This means that you need to call getQuotes before faking the submission of the form.

The actual faking is relatively easy. The first thing is to generate the request object from the form to be used in the POST request and second thing is finding out the URL of the request. Here is the plete code:

var casper = require('casper').create();
var x = require('casper').selectXPath;

casper.userAgent("Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2049.0 Safari/537.36")

casper.start('http://www.nasdaq./symbol/aapl/historical');
casper.wait(5000); // probably not necessary
casper.thenClick('#lnkDownLoad');
casper.then(function(){
    var parameters = this.evaluate(function(){
        // from http://stackoverflow./a/2403206
        var paramObj = {};
        $.each($('#getFile').serializeArray(), function(_, kv) {
            paramObj[kv.name] = kv.value;
        });
        return paramObj;
    });
    var url = this.getElementAttribute('#getFile', 'action');
    this.download(url, 'aapl.csv', 'POST', parameters);
});
casper.run();