I’ve been following with interest as Derek Willis explores Caspio, a sort of hosted data-driven web app tool for journalists. The following started out as a comment on his blog, but soon ballooned, so I’m posting it over here where it’ll have more space to breathe.
Of course, for this to make any sense, you should read Derek’s articles first:
- Outsourcing Database Development, or the Caspio Issue (be sure to also read Caspio CEO David Milliron’s comments, too).
- On Trials, Software and Otherwise
Like Derek, after reading David Milliron’s reply, I went to go poke at Caspio, and had a very similar experience to Derek, who writes:
I went to caspio.com to see about a free 14-day trial in order to test things out. Then I read the Terms of Service, which contains this sentence: “In addition, you may not access the Service for purposes of monitoring its availability, performance or functionality, or for any other benchmarking or competitive purposes.”
Like Derek, that stopped me cold.
I missed the part about using the demo for “competitive purposes,” too. I’m pretty sure that my role as the lead developer of both commercial and non-commercial competition would get snagged under that bit, as well.
As far as I’m concerned, that right there is the mark of bad software. Good software makes its developers proud: they want to show it off to everyone and shout its praises from the mountaintops. Bad software makes its developers embarrassed and scared: they know any competent programmer could improve on their functionality in no time. Those TOSes that forbid competitors and benchmarking seem like coded messages from the developers saying “our software sucks!”
But Derek’s right on about the appeal of tools like Caspio — I’ve done my share of fighting with IT, and I can actually speak their language. I shudder to picture an editor trying to get a site from IT done on deadline. In fact, if I’ve got my Journal-World history correct, the online news division actually hired its own programmers to route around corporate IT.
Still, in my mind it all comes down to shortsighted management. You don’t need a lot of resources to roll your own awesome online coverage, and those minimal resources are a necessary investment in the future of the news industry.
In the past couple of years the web development landscape has changed dramatically. The Third Age — the age of the web framework — is upon us, and that means that developers can do a lot more with limited resources.
A single example I’m currently crushing on is PolitiFact: it was developed at the St. Petersburg Times — not a large organization at all — by a single programmer, Matt Waite.
I think even the smallest newspaper should be able to afford a single on-staff programmer. Let’s compare costs, using PolitiFact as a rough starting point.
I don’t know what Matt makes, but an average programmer’s yearly salary, according to the US Bureau of Labor Statistics is $61,000 (the BLS data is only availble through POSTed forms, which means I can’t link you directly to that data; sorry.) In practice I know that newspapers pay far below average, but let’s use $61k as a strawman anyway.
So what would PolitiFact cost if it was built with Caspio (assuming that was even possible)? Well, it’s hard to figure out given Caspio’s pricing — Caspio has a number of different price points depending on how long archives are maintained, level of service, etc. However, given that only the highest level plan offers 24/7 support and an SLA (both of which you get for “free” by calling your full-time developer), I think we’re talking about the top plan, which runs $12 per “DataPage” per month.
Now, I’m not sure what a “DataPage” is, but it looks like it’s basically any database-driven “page” presented as part of your app. Google tells me that there are 205 pages under politifact.com; thats $2.460 per month, or almost $30k per year — half the average programmer’s salary.
I don’t know how long it took Matt to build PolitiFact, but I doubt it was a full six months. Even if it was, though, a on-staff programmer costs a fixed amount reguardless of how many sites you launch. LJWorld.com has dozens of data-driven projects, most written in a couple of days at most, that are mostly self-maintaining — reporters and producers can feed them with updated data at negligable cost.
if you’re using something like Caspio, however you’d need to keep paying that $30,000 every year you want to keep the site up.nce you’re up to four sites of that size you’re looking at a $120,000 yearly ransom just to keep your data from 404ing.
Of course, none of this changes the sad fact that many news organizations are so shortsighted that they think hiring a programmer is an extravagant expense. At least tools like Caspio let these organizations start learning the techniques they’ll need to stay in business, but in an ideal online media landscape there’d be no need for silly tools like Caspio.
Mikkel
Sept. 13th, 2007
8:50 a.m.
Well, this was an interesting read. I think the quote from AberdeenGroup on Caspio's front page says it all: "It's so easy that it's two steps simpler than even Microsoft Access."
As any developer knows, simplicity comes with a price. Too bad that the price in the case of Caspio includes cruddy output that refuses to display any data at all when viewed without JavaScript enabled, thereby prohibiting your data to be found by Google and the like.
Matt Waite
Sept. 13th, 2007
10:11 a.m.
I'm honored, Jacob. I'm a huge fan of the stuff you guys are doing in Lawrence and with the Django framework.
Couple of details to help out your comparison (which is dead on):
I won't say how much I make, but I can tell you a lot when I say I'm actually a reporter, not a programmer.
And from the date I created the app in Django on my test box to code freeze on the production servers was about 90 work days. To newspaper people this seems really fast and to online people seems agonizingly slow. If I were to start over today, I could cut that development time down to next to nothing. Why? Because in those 90 days, *I was learning Django as I went*. PolitiFact is the first Django app I've deployed -- the first one the Times has deployed as a company. The most complex view I'd written before PolitiFact was variations on database.objects.all(). So, with a competent professional at the helm, PolitiFact would have happened much faster. Because I've learned so much making PolitiFact, our next apps will fly like greased lightning.
Jeff Croft
Sept. 13th, 2007
12:22 p.m.
I think Matt's point about how much faster you can do something after you've learned a good framework like Django is really significant. It seems as though every Django project I create takes half the time the last one did. There are really two reasons: first, I learn more wach time, and that knowledge makes the next tiem faster. And second, the nature of programming in Python and Django (and probably most other good frameworks) is to create reuable code. As such, with each project I complete, I build up more of a library of reusable, plug-and-play type of pieces, as well as code snippets that can be copied-and-pasted-and-modified-a-bit into the next project where appropriate.
To be frank, it's something that's actually posed a problem for me doing client work. It means that each sucessive project takes me less time, which means it's harder to bill the client for as many hours. :)
This, of course, would actually be a major plus for an in-house team. For a newspaper, they can easily hire a programmer or two and start with a big project that takes -- at Matt says -- 90 days or so. The next project of a similar size might take 30 days. The next one might take 15. Then 10. Pretty soon, you're The Lawrence Journal-World, writing cool apps in a matter of a few days, because you're reusing code and knowledge for incredible efficiency.
Ryan Pitts
Sept. 13th, 2007
1:39 p.m.
A-effing-men.
Frank Wiles
Sept. 14th, 2007
3:09 p.m.
I think *most* businesses could discover serious benefits to having at least one on staff programmer. Or at least a long term relationship with a small development company or freelancer. I've seen many businesses stop using Product X, that they paid big money for 1-3 years ago, to new Product Y because Product X could only export data as CSV and another system they now need to feed this data into only takes XML.
Companies without a programmer are used to operating this way, so scrapping the perfectly working Product X and spending big $$$ on Product Y is just a cost of doing business for them. But they don't realize they just spent say $20k on something a developer could build in a matter of a couple of hours.
One warning, having just *one* single lone programmer can spell disaster for a company as well. Solo coding makes it very easy and tempting to cut corners, document poorly, etc. If possible get at least two programmers and your situation improves dramatically.