I’ve recently been collaborating with Christian Hansen on HTTP::Tiny, a minimalist, HTTP/1.1 client library for Perl. For basic web client tasks like grabbing a single page or mirroring a file, it does the job in a fraction of the code that would be needed to install LWP::UserAgent. Because it has no non-core dependencies, it is ideal for what I need to get CPAN.pm to bootstrap itself with pure Perl.
Here’s a quick look at how you would use it for the two common tasks I mentioned:
use HTTP::Tiny;
my $http = HTTP::Tiny->new;
my $response;
# get a single page
$response = $http->get('http://example.com/');
die "Failed!\n" unless $response->{success};
print $response->{content};
# mirror a file
$response = $http->mirror('http://example.com/file.tar.gz', 'file.tar.gz');
die "Failed!\n" unless $response->{success};
print "Unchanged!\n" if $response->{status} eq '304';
The example above is almost the same as you’d get using LWP::UserAgent with one big exception. HTTP::Tiny returns the response as a hash reference rather than as an object. Just like LWP::UserAgent, the mirror method will send an If-Modified-Since header for an existing file to skip downloading if the file is unchanged. HTTP::Tiny doesn’t (yet) handle query parameters – you have to prepare those yourself, but for simple downloads, you don’t generally need those anyway.
Where did HTTP::Tiny come from? When I was working on getting CPAN.pm to support a pure-Perl HTTP bootstrap, I started with HTTP::Lite. When I discussed it on #p5p, Christian Hansen pointed out a number of serious shortcomings and decided that it would be easier for him to write a new, lightweight HTTP/1.1 client from scratch rather than try to redo the plumbing in HTTP::Lite.
The result is HTTP::Tiny and I’ve been collaborating with Christian to get it ready for use by CPAN.pm and ready for the Perl core. Unlike HTTP::Lite, HTTP::Tiny is a conditionally conforming HTTP/1.1 client. It supports both redirection and mirroring, which HTTP::Lite does not, both of which are important features for a CPAN client.
As a “Tiny” module, HTTP::Tiny achieves its HTTP/1.1 conformance in a just a fraction the code required for LWP::UserAgent and its non-core dependents. Don’t get me wrong – LWP::UserAgent is a great piece of software. But sometimes – like for the Perl core or for a fatpacked application – something much smaller and simpler will do just as well.
Let’s see what a difference there is. CPANdeps shows us all of LWP::UserAgent’s dependencies and highlights the ones that are non-core (as of Perl 5.10.1):
I used David A. Wheeler’s SLOCCOUNT to count lines of code in each of the distributions containing these modules as well as for HTTP::Lite and HTTP::Tiny (based on the “soon-to-be” 0.007 release). In all cases, I excluded files in t/ and examples directories. (Lines of Perl includes any programs distributed with the distribution.)
Here are the results:
Distribution .pm files Lines (Perl) Lines (C) Total Lines
----------------- --------- ------------ ---------- -----------
libwww-perl-5.837 52 10258 0 10258
HTML-Parser-3.68 7 883 1972 2855
HTML-Tagset-3.20 1 139 0 139
URI-1.56 52 2715 0 2715
----- ----- ----- -----
TOTAL LWP & friends 112 13995 1972 15967
versus
Distribution .pm files Lines (Perl) Lines (C) Total Lines
----------------- --------- ------------ ---------- -----------
HTTP-Lite-2.3 1 634 0 634
HTTP-Tiny-0.007 1 603 0 603
Wow! Both HTTP::Lite and HTTP::Tiny accomplish simple HTTP tasks with less than 4% of the SLOC of LWP and its non-core dependencies.
What about memory usage? Here’s the “null” case that shows memory usage of just loading the three client modules:
VSZ RSS COMMAND 30440 5512 perl -MLWP::UserAgent -e 1 while 1 26700 3960 perl -MHTTP::Tiny -e 1 while 1 26040 3168 perl -MHTTP::Lite -e 1 while 1
HTTP::Lite is smallest, but it also doesn’t have the features or conformance I need.
That’s not much of a real-world test, so let’s try something else – downloading the 950K CPAN 02packages.details.txt.gz file. Here is the test code for HTTP::Tiny and LWP::UserAgent side by side:
# http-tiny-mirror.pl # lwp-useragent-mirror.pl
#!/usr/bin/env perl #!/usr/bin/env perl
use strict; use strict;
use warnings; use warnings;
use HTTP::Tiny; use LWP::UserAgent;
my ($url, $file) = @ARGV; my ($url, $file) = @ARGV;
die unless $url & $file; die unless $url & $file;
my $http = HTTP::Tiny->new; my $http = LWP::UserAgent->new;
my $res = $http->mirror($url, $file); my $res = $http->mirror($url, $file);
die "Failed!\n" die "Failed!\n"
unless $res->{success}; unless $res->is_success;
1 while 1; 1 while 1;
What about HTTP::Lite? It doesn’t have a mirror method and doesn’t do redirection, which I get for free with HTTP::Tiny and LWP::UserAgent. I’d have to write dozens of lines of code to emulate that for a “fair” test, so I skipped it.
Let’s add those two programs (after downloading the 02packages file to files with different names) to our memory benchmarks:
VSZ RSS COMMAND 50916 9824 perl ./lwp-useragent-mirror.pl [...] 35376 4348 perl ./http-tiny-mirror.pl [...] 30440 5512 perl -MLWP::UserAgent -e 1 while 1 26700 3960 perl -MHTTP::Tiny -e 1 while 1 26040 3168 perl -MHTTP::Lite -e 1 while 1
HTTP::Tiny wins – not by a huge amount in absolute terms, admittedly, but if you don’t need the extra features that LWP::UserAgent offers, HTTP::Tiny might just be all you need.
Thank you, Christian, for HTTP::Tiny!