Curl is great and all but I don't think this quote makes much sense. Are there examples of services that would, for example, load in a browser but not through curl? I'd think it's rather the opposite since you can through pretty much anything at curl and it will work.
On a serious note, “progressive enhancement – still the right way”. I don't think client-side rendering is going away but everyone who's relied on it exclusively has learned the hard way that it's just too unreliable and slow to have a failure mode which is an empty page unless a lot of complex code works perfectly.
Just think about how many engineer-hours Twitter flushed with that silly #! kludge – and then when they switched back, saw an 80% improvement in page load time.
> Just think about how many engineer-hours Twitter flushed with that silly #! kludge – and then when they switched back, saw an 80% improvement in page load time.
No hours were wasted, and they didn't really switch back. They're just using HTML5's History API on browsers that support it now. Essentially the same mechanism under the hood, just prettier URLs for it.
They did more than just switch to the history API. During that period, if anything went wrong, you saw a blank page and, of course, robots saw only the generic launcher HTML instead of any content.
Now, here's what a tweet looks like without JavaScript enabled:
(The main difference would be that things like the search & menus are either interactive controls or simple links to basic HTML forms depending whether JavaScript loads)
During the hashbang era you couldn't use a page without a full rendering ending. Now, however, all of the content is available with fairly rich markup:
I think he was making reference to the DOM rendering in the twitter website which used to be on client side until recently when they have switched back to server side rendering. But, it's very similar to pre-rendering React components on the server then mounting it so it might not be a total waste, that was a fad that didn't make a lot of sense for the time though.
>but everyone who's relied on it exclusively has learned the hard way that it's just too unreliable and slow to have a failure mode which is an empty page unless a lot of complex code works perfectly.
FUD. I develop HTML5 gambling for a living and this anti-javascript sentiment on HN is getting really tiresome. You honestly sound like a bunch of old people, complaining that a PC isn't a typewriter or a fountain pen.
Yeah your fountain pen doesn't require power and it writes your name really well, but that doesn't mean that the PC isn't better.
Client side rendering means you will have to test in all browsers, writing android apps means you have to test on a lot of units.
You rather spectacularly missed the point: I wasn't saying not to do the fancy stuff but rather to start with something which degrades well and then have your JavaScript enhance that basic experience.
If you want to know why this is a good idea, you should start using something like getsentry.com or errorception.com to record your JavaScript errors. That won't tell you who couldn't execute JavaScript at all but it'll show how many times something didn't load due to a flaky ISP, adware, buggy anti-virus, odd browser settings, etc. With progressive enhancement, those people still have a reasonable chance of at least seeing the content on the page. With a pure JS approach, they're only going to see a blank and will probably be heading over to a competitor whose site degrades well.
(Note that this is only the question of the site working at all. In most cases, the progressive site will also render considerably faster – Twitter found an 80% improvement! – since the pure-JS approach breaks the browser's prefetch optimizations and requires much more work to achieve comparable performance)
> I develop HTML5 gambling for a living and this anti-javascript sentiment on HN is getting really tiresome.
I've been using websites since the early 90s and this pro-single-page sentiment is getting really tiresome. You are breaking the web. You are destroying users' security.
Sure, there are plenty of reasons to use JavaScript, and plenty of places where it's appropriate. It probably is a good idea for games and so forth. But requiring users to load and execute constantly-changing code from across the web in order to read a page or submit a form is in-friggin-sane.
Some one else pointed out that it'd be nice if browsers offered more support for things that certain types of developers clearly want to do. I completely agree; it'd definitely be nice to take advantage of many of the technologies which currently exist to do more, in a more structured way. But requiring code execution in order to read data is madness.
I really wish building single-page apps was better supported by browsers. The current setup is to try to kludge over the fact that DOM is for documents, not apps, and use the awfulness that is CSS and current layout paradigms (note, no CSS is strictly worse than yes CSS, but that doesn't make CSS good).
Breaking away from the DOM but keeping the ability to remotely load code and static assets would be the best of both worlds. Something like Google Maps is an application, not a document. Why are we rendering it with a document renderer? Why are we styling it with CSS, which is again, document-oriented? The good part about single page apps is that they deliver a package of code, then are able to keep local state and communicate with the server over a stateless protocol. Oh, and the current breed runs on a platform (browsers) that is installed on every PC and mobile device. Browsers just need a better format for delivering packages of code and better support for these than "manipulate the DOM".
Your server should just be providing data that's displayed by the JS App. The JS app can handle a bunch of requests, but each request should provide something useful and be accessible from curl.
bonus, it makes it easier to write your app in other languages or for other platforms since the web server really is a server and the front-end is just a client.
The two are not necessarily mutually exclusive, especially with JavaScript frameworks you can run on the server to render the initial page, which then gets wired up to the client-side application. i.e. "Isomorphic JavaScript" [1][2]
Website != API. A cleverly-written API can exist in the same space as a Website (even the same URLs, if you differentiate by things like Accept: headers), but they're not the same thing.
It's perfectly acceptable for Websites to include, and even require the use of, JavaScript. It's also perfectly acceptable to offer a JavaScript client for your API. In fact, you pretty much have to do that if you want your Website to work with the API anyway. What's not OK is for an API to require downloading and using additional JavaScript while the API is being used.
As a contrived (and somewhat ludicrous) example, let's say that I queried "http://foo.com/bars" for some kind of collection. It's OK to return the collection. It's even OK to use an HTTP redirect if the collection actually resides elsewhere: HTTP 302, perhaps, with a Location of "http://foo.com.bazes". What's not OK is to return a line of JavaScript reading "window.location = 'http://foo.com/bazes';" which might work for the browser, but wouldn't work for most other clients.
curl has a --libcurl option which you can add to a curl command line, and then it'll generate a libcurl-using C code template for that same operation...
Those were the early days of HTTP, so no, people didn't really get a lot of HTTP "manually" like that. We used other existing tools to get gopher and ftp though.
When I started my work on httpget in 1996, I didn't know about wget and I didn't become aware of it until several years later....
I would figure that by 1997 most people stopped using gopher/archive/veronica. I think I stopped by 1995 or so.
now days we use curl/wget for web dev it seems? probably used it for the same thing back then but I also image that lots of people just used their own ad-hoc scripts or whatever; i wish i had firsthand knowledge of that.
i used wget to download copies of websites when i found one that was mostly documents/images rather than webapps.
I use wget for plain, old regular downloads a lot. If I need to save a file to a specific place (e.g. datasheets to a doc/ directory in my project) and I have a terminal already open there (which I usually do), doing wget and middle-click is far more convenient than dealing with the browsers' "Save to..." dialog. Especially ever since they started considering a sane download manager to be one of those confusing features that users totally don't want.
In the HTTP/1.0 days I used telnet quite a bit. Or libwww-perl, when Perl was the language of choice for web applications. LWP has copyright notice:
Copyright 1995-2009, Gisle Aas
Copyright 1995, Martijn Koster
Round about then I wrote something similar to WWW::Mechanize in order to automate access to the Orange Web-SMS gateway, which had an inconveniently deep login procedure.
I had to use that the other year whilst attempting to migrate an old old old Slackware box running on two Pentium 3s with an array of SCSI disks (so quite fancy for its time!). The machine had kernel sources that were different to the kernel running so no chance of compiling various bits of software, and I couldn't grab any GCC (EGCS?) binaries from anywhere.
I can't remember what I eventually did but I do remember discovering lynx -dump, so thanks for the reminder!
I believe they originally come from the libwww-perl project? According to wikipedia it was first released with perl 4.0036, which was in Februrary 1993, but I can't find out if it included GET and HEAD etc already then...
HTTP has been in use by the World-Wide Web global information initiative since 1990.
This specification reflects common usage of the protocol referred to as "HTTP/1.0".
I used netcat - and still do for interacting with protocols in ways that curl doesn't support. AFAIK it was released in 1995, so it's only three years older than curl.
It's slightly weird to me that I am older than Curl and Wget. They always seemed like Unix Monoliths to me; I had just assumed they had always existed.
Curl is great, but I also recently came across HTTPie (https://github.com/jakubroztocil/httpie) which has some nice features for playing around with HTTP APIs (JSON formatting, syntax highlighting, etc)
I was wgetting all my http requests until about 2 years ago. Curl's undeniable coolness won me over after 15 years. Now I practically live in curl when I'm setting up webservices, and libcurl for PHP does something on nearly every page request I have.
For PHP people, curl_multi_exec is the new event loop.
PHP applications may use libcurl when making outbound HTTP requests, right? That is only a small subset of these web sites. Most web sites just normally process HTTP requests.
This does not seem to explain the 1 billion users figure.
WordPress uses libcurl, according to a google search there are 74,652,825 sites using WordPress. How many unique visitors do you think these sites get? I would guess over a billion.
Just because Wordpress, which is a blog platform, uses libcurl for something you didn't even state (probably some outbound http stuff), doesn't mean it uses it to process most of the incoming requests for the millions of users.
We don't say Solitare is the most successful game ever just because it's installed with every copy of Windows.
If you count a user as a system which has curl installed, and has run it at some point, is curl on iOS and Android? If so, you will be in that ball park I imagine.
That wouldn't be that surprising depending on what they mean. Assuming that Google or Facebook use curl in various applications, 1 billion sounds about right.
Depends if you're defining that as someone who has manually run a cURL command or as someone who has used an app/hardware that uses cURL "behind the scenes". If you use the latter I would say 1 billion might be a significant underestimate.
curl is just awesome, thanks so much to the author and all maintainers over the years! It's still my go to application for testing and debugging HTTP requests.
Curl is pretty great to use, but be warned it has old smelly code and is probably full of security issues. I wouldn't use it for anything too critical.
Also wget can recursively mirror webpages and there are nice options to carefully select contents you want to download. It's quite dated though, I wish it could use an external downloader (like aria2) and only do the walking and converting links part itself.
Do note that just a plain wget command isn't enough to properly archive/"back up" a webpage, because it doesn't save headers and other important metadata. Make sure to always use WARCs, which preserve metadata and have wonderful support, including the possibility to get absorbed into the Wayback Machine. More info:
That mirroring option in wget has been very useful over the years, even more so if you remember to add --no-parent, else you'll end up with the ENTIRE site.
wget to make a static copy of a site is the one thing I use it for. It's awesome. I tend to reach for curl first in most other cases, for one thing it's included in the "minimal" RHEL installation so I can pretty much depend on it always being there, whereas wget is extra.
So, so true. Thanks, curl.