Home >> Blog >> Downloading big files with Titanium Desktop

Friday, 08 April 2011 15:36

Downloading big files with Titanium Desktop

Written by  Nicholas K. Dionysopoulos
Rate this item
(1 Vote)

I have already talked about Appcelerator's Titanium in a previous article. One of the things that I found extremely hard to accomplish is downloading very big files with it. Using Titanium.Network.HTTPClient blocks the user interface and makes the application seem "hang" while the download is in progress. So, how can you download big files in Titanium without causing the whole application to freeze? The solution is easy; just use Web Workers!

If you are asking yourself what is a Web Worker, the short answer is that it's the Javascript implementation of threaded applications. The (overly simplified and not 100% technically correct) idea is that a Web Worker is like an application -technically, it's called a "thread"- which can run in parallel with your main Javascript application and still be able to communicate with your main application. This means that it runs asynchronously to your main Javascript application and whatever happens inside it does not interfere with your main app. In other words, you can tell it to download a huge file and it won't block your application! How cool is that? It's very cool, and it has already made it to the HTML 5 spec. If you want more information about Web Workers, a shiny new feature of HTML 5, please take a look at the W3C draft specification.

There is also one pitfall to this... Just like in any other programming language that supports threads, Web Workers in Javascript are isolated from your main application thread. This means that they can not exchange data directly with your main thread. They can, however, post messages to your application's main thread. What happens is that the message is dispatched from the worker thread (which momentarily pauses), caught by an event handler in the main application thread and, while it is being processed, both the Worker thread and the main application appear to be frozen, so be really quick with your processing! Another pitfall is that since the thread runs inside its own sandbox, you can not reuse code from your main Javascript application. Nope. You have to provide all of the thread's code in a separate file instead.

So, using that knowledge, we'll build a download application which notifies us of the download progress without blocking the main application's interface. So, let's start with our index.html file:

 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
<html>
<head>
<script type="text/javascript">
	var BINARY_UNITS= [1024, 'Ki', 'Mi', 'Gi', 'Ti', 'Pi', 'Ei', 'Zi', 'Yo'];
	var SI_UNITS= [1000, 'k', 'M', 'G', 'T', 'P', 'E', 'Z', 'Y'];
 
	function unitify(n, units) {
	    for (var i= units.length; i-->1;) {
	        var unit= Math.pow(units[0], i);
	        if (n >= unit) {
		var result = n / unit; 
		return result.toFixed(2) + units[i];
		}
	    }
    	return n;
	}
 
	// How many byte have been downloaded yet
	var dlbytes = 0;
 
	// Uses a Worker thread to download everything in the background
	function downloadFiles()
	{
		var $worker = Titanium.Worker.createWorker('download.js');
 
		$worker.onmessage = function($event)
		{
			var newdl = parseInt($event.message);
			if(newdl == 0) {
				dlbytes = 0;
			} else if(newdl == -1) {
				alert('Download failed');
				dlbytes = 0;
				$worker.terminate();
			} else if(newdl == -2) {					
				alert('Download complete');
				$worker.terminate();
			} else {
				dlbytes += newdl;
			}
			document.getElementById('dlsize').innerText = unitify(dlbytes,BINARY_UNITS);
		};
 
		$worker.start();
	}
</script>
</head>
<body style="background-color:#1c1c1c;margin:0" onload="downloadFiles();">
    <div style="border-top:1px solid #404040">
        <div style="color:#fff;;padding:10px">
            Welcome to Titanium
        </div>
 
		<div style="color:#ff0;padding:20px">
			Downloaded <span id="dlsize">0 </span>B
		</div>
    </div>
</body>
</html>
 


That's pretty standard stuff... Take a look at how we attach an even handler with the .onmessage() method that does minimal processing (updates a downloaded bytes counter). Also note that the Web Worker thread will run eternally unless we .terminate() it Wink

Now, that code above references a file named dowload.js. Here's the code to it and where all the fun happens:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
var url = 'http://www.example.com/bigfile.zip';
var httpClient = Titanium.Network.createHTTPClient();
httpClient.xxxWorker = this;
 
//Check for d/l finished event
httpClient.onreadystatechange = function(e) {
	if (e.readyState == 4) {
		this.xxxWorker.postMessage(-2);
	}
};
 
httpClient.onerror = function(e) {
	this.xxxWorker.postMessage(-1);
};
 
if (httpClient.open('GET', url)) {
	this.postMessage(0);
	var file = Titanium.Filesystem.createTempFile();
	var filePath = 	Titanium.Filesystem.getDesktopDirectory().toString()+
				Titanium.Filesystem.getSeparator()+
				'bigfile.zip';
	file.copy(filePath);
 
	httpClient.xxxFile = filePath;
 
	// Handle the received data (Titanium.Filesystem.File can also be used as a handler)
	httpClient.receive(function(data) {
		var file = Titanium.Filesystem.getFile(this.xxxFile);
		var fileStream = file.open(Titanium.Filesystem.MODE_APPEND);
		fileStream.write(data);
		fileStream.close();
		this.xxxWorker.postMessage(data.length);
	});
} else {
	this.postMessage(-1);
}


Awesome! We just told it to download http://www.example.com/bigfile.zip to a same-named file on our user's desktop. Look at all the .postMessage() method calls. We basically send a 0 when the d/l starts, -1 if it fails, -2 if it is over or the number of downloaded bytes when we receive more data. If you do some close inspection, you'll realize that the receive() handler is called for every 16KiB of data received (that's the default data window) so you can have a very granular download progress display in your application.

Well, that's all folks! Until the next time, stay safe and stick to the code!

 

The author would like to thank Johan Janssens who gave the hint to use Web Worker threads to overcome the large files download problem.

Last modified on Friday, 08 April 2011 16:07

5 comments

  • Comment Link Alexandre Tuesday, 20 December 2011 14:20 posted by Alexandre

    How can I have this php code in the url :



    ?
    please help me ! :)

  • Comment Link Nicholas K. Dionysopoulos Saturday, 03 September 2011 17:04 posted by Nicholas K. Dionysopoulos

    As far as I know there is no way to change the chunk size. And, yes, the speed is pathetic. Moreover, the memory leaks are profound and if you're downloading over a few hundred megabytes or so, you're in for a big system-wide crash as soon as the paging file fills up the entire hard drive :(

  • Comment Link Mus Wednesday, 31 August 2011 08:31 posted by Mus

    Thanks for the code, very helpful. I'm downloading some big files with this from an apache running on my machine. The download speed is pathetic. I thought maybe if I increase the chunk size it will help but found no reference to it in the docs. Is it possible to do that?

  • Comment Link Nicholas K. Dionysopoulos Wednesday, 13 April 2011 08:46 posted by Nicholas K. Dionysopoulos

    Yes, there is a way. You can put the worker code inside a function onmessage(event){ (code goes here) } function. Then, from your main application, just run a $worker.postMessage({url: 'http://www.example.com'}). You can then access from the context of the worker's onmessage function as event.message.url.

    Do not get confused by having two onmessage event handlers! Remember that Web Workers are real OS-level threads. The only way to exchange information between them is posting messages (called "synchronization" in many other programming languages). So you postMessage from the main application to your worker thread and the worker's onmessage handler reads the incoming data. Then you postMessage from the worker thread to the main application and the application-side onmessage event handler of your thread objects reads the worker's outgoing data.

    It sounds very confusing, at best, but once you get the hang of it you can totally create ass-kicking, data crunching applications in Javascript! Threaded programming in any programming language is not for the faint at heart, but it will truly reward you :)

  • Comment Link Dean Wednesday, 13 April 2011 01:16 posted by Dean

    Thanks for sharing this code it was a great help. Only issue is that I don't like having to hard-code the url to the file. Is there a way to pass this to the worker from the main thread? I can't seem to find any way to manage it. :(

Leave a comment

Make sure you enter the (*) required information where indicated.
Basic HTML code is allowed.
All comments are auto-published. Unless you are a spammer, violate any laws or ask me to unpublish your comment, I won't unpublish any comments. Feel free to say anything you like.