Why the PhyloPic Relaunch Took So Long

Or, A Lesson in Development Strategy.

As I announced last week, my website, PhyloPic, has been relaunched with a massive update. One of the key updates is a public API for developers. A lot of people have been looking forward to this, and it was actually almost ready for release last summer. So why didn't I release it?

Failure to Branch

Basal tracheophyte.
Public Domain.

As I was writing up the documentation for the API, I learned of Bootstrap, a CSS/JavaScript framework. I realized that it could solve a lot of the design issues I was having — problems with the site on mobile devices, older browsers, etc.

What I should have done: Created a new development branch for adding Bootstrap while continuing to polish up the API branch. That way, I could have released the API shortly while still being able to work on the design issues in parallel.

What I actually did: Continued working in the same branch, ensuring that I couldn't release the API update until the Bootstrap update was complete.

Having Other Projects

By the end of summer I was mostly done with the revisions, but there was still some cleanup to do. By now some other projects I'm attached to, one with other collaborators, were suffering. So I spent most of my free time in the autumn working on those. (I have a full-time job and a toddler, so that isn't much.)

Homo habilis.
Public Domain.

Becoming Enamored of New Technology

In the autumn, Microsoft release a preview version of TypeScript, and I quickly saw that it was going to be extremely useful. So I rewrote PhyloPic's client-side code — it wasn't too hard and it made further development a lot easier. This caused some delay up-front, but I don't regret it.

Becoming Enamored of the Wrong New Technology

Around this time I also realized that I could finally do away with the last bit of Flash on the website: the Image Submission Tool. HTML5 had become mature enough to do all the image manipulation in the browser itself. I did a lot of research, learning about the Canvas, Typed Arrays, etc. And after a lot of work I actually created an image-processing workflow that work in HTML5-enabled browsers. As a bonus, I got a little standalone project out of it: Pictish.

But there were problems. One is that the best existing JavaScript library for creating PNG files doesn't use Typed Arrays — it uses strings, which means that it is slow for large files. I tried creating my own PNG encoder, or adapting that one, but soon realized it was far too much work. Another problem is that I was no longer supporting older browsers (although this was a trade-off against supporting mobile platforms, so I didn't feel too bad about it).

But there was a much more fundamental danger: doing the image-processing in the client side meant that the API had to trust the client to do it properly. What if some developer used the PhyloPic API to add images to the database but didn't do it right? That could be disastrous.

Octopus bimaculatus.
Public Domain.

I realized I would have to do things the old-fashioned way: on the server. After a bit of research, I identified Image Magick and Inkscape as the best tools. The new methodology was so completely different that I ended up making a lot of database changes, too. Until recently, all files were stored in the database — now they're just stored as flat files. The good news is that this makes load times faster.

Doing Things the "Right" Way

Throughout all this I had been making an effort to "dogfood" my own API, i.e., to use it on the site itself. This has the advantage of making load times faster, since the basic page can be cached and then the data can be loaded in secondarily in a much smaller format. Unfortunately this meant a lot of rewrites for how the pages are rendered.

After a while, the code to generate pages from the data had gotten really complex (mostly involving on-the-fly element generation using jQuery). Around the time I was redoing the Image Submission Page, I realized my whole approach was untenable. I needed a cleaner way to divorce presentation logic from control logic.

I ended up using Knockout for the entire site. It made things a lot more manageable.

In Summary

The biggest problem was my branching model, or, rather, my lack of one. Solitary developers often fall into this trap: we think that, since we're doing all the work, there's no need to have more than a single branch of development. At work, we've been using this model and found it very successful. Going forward, I plan to do this on PhyloPic as well. No more massive updates where everything is different. Just incremental features and fixes.

PhyloPic Launch: API, Responsive Design, etc.

On Good Friday I took PhyloPic down. On Holy Saturday, I wrestled with errors caused by incongruities between the server and dev environments. And, lo, now, on Easter Sunday I announce that PhyloPic is back! (Actually, I already announced in on Twitter, but whatever.)

How smartphone users should see PhyloPic, more or less.

Major New Features:

• A Developer API (using JSON). Now other people can build applications using PhyloPic data and images. (Yes, I am dogfooding it, so most of it should be pretty well-tested.)
• Responsive design (using the ever-more-ubiquitous Bootstrap) — the site is now much more useable on mobile devices.
• A Links Page, showing off work that uses PhyloPic or features it in some way.
• Speedier load times (in theory, anyway).
• Ranks for Contributors — if you submit one image, you're a "Specialist". Two, and you're a "General". Six, and you're a "Familiar". See where this is going?
• Fewer requirements — most notably, Flash is no longer required to submit images.
• Handy little icons on most taxon links — now you can tell if you're clicking on Gastonia the dicotyledonous plant or Gastonia the dinosaur. (Still rolling this out to all taxa.)

Preview Screenshots

A little glimpse of what I've been working on:

"Year of Macrauchenia": Third and Final "All Your Yesterdays" Entry

I made a last-minute entry for the All Your Yesterdays contest:

Year of Macrauchenia

Macrauchenia was the greatest and last of the litopterns, a clade of stem-euungulates. This bizarre Pleistocene South American herbivore is often described as having the body of a llama and the head of a tapir. However, the body is only superficially llama-like (but with gigantic elbows) and the head is not at all tapir-like. (If any part of it resembles tapirs, it's the feet.) In fact, the skull isn't like that of any living terrestrial mammal. It has extremely dorsal nares, like trunked animals and cetaceans, but it lacks any place for trunk muscles to attach.

But there is a possible analogue — another group of terrestrial herbivores with extremely dorsal nares — and they even had long necks, too! I refer, of course, to sauropod dinosaurs. Unfortunately, none are extant for comparison, but recent work has shown that, despite the dorsal placement of the nares in the skull, the external nostrils were still placed rostrally, close to the mouth, thanks to fleshy tubes. I've restored Macrauchenia similarly.

This mandala depicts the life of Macrauchenia across the seasons. At bottom, a lone Macrauchenia wanders the frozen highlands in relative comfort, having grown a shaggy winter coat. Its fleshy nostril tubes serve to warm the air before it enters the body. At right, spring is in effect — a bull courts a cow by inflating his nostril tubes, similar to a hooded seal. At top, a young calf frolics under his mother's watchful eye — his green color comes from the algae living in his fur (similar to the camouflage of those other South American indigenes, the sloths). This extra measure of color accuracy is necessary because, unlike today's ungulates, Macrauchenia must contend with predators that have excellent color vision: phorusrhacids. At left, a wary bull faces off against a Smilodon, an invading predator from the north. (It is restored after linsangs, the extant sister group to felids, instead of the felids themselves, since it is, properly, a stem-felid, not a true felid.) Macrauchenia will survive this great faunal interchange, but not for long — another invader from the north, a large primate, will be the end of it, and hence all litopterns.

Another "All Your Yesterdays" entry: Denisovan, or "Polar Neandertal"

They've extended the deadline for the All Your Yesterdays contest, so I've decided to do a couple more entries. I started this one years ago as a Neandertal restoration. Since that time, new genomic discoveries showed that the speculative pigmentation was incorrect. But, other discoveries identified a new candidate for the subject matter!

Known from a few scrappy pieces, the Siberian Denisovans (Homo sp. or Homo sapiens ssp., depending on how large you like your species) are a true challenge to reconstruct. We have their entire genome, but know almost nothing about their anatomy. The few fossil elements we have are not morphologically distinct from Neandertals (Homo neanderthalensis or Homo sapiens neanderthalensis) or humans (Homo sapiens sapiens).

But the genomic facts are highly intriguing:

1. Some Oceanian humans have inherited up to 6% of their nuclear DNA from Denisovans (with the highest ratios in Meganesia [Australia and New Guinea]). 
2. The nuclear DNA indicates a common ancestor with Neandertals, shortly after the split from proto-humans.
3. But the mitochondrial DNA indicates a motherline that branched off much earlier. (Possibly Homo erectus?)
4. Genes for pigments are consistent with dark skin.

Here I've imagined a Siberian Denisovan as a sort of "polar Neandertal". As with polar bears, his skin is dark, trapping heat, but his pelage is light, allowing for camouflage against the taiga and tundra. He is the last of his kind — his southern kin mixed with the strange, baby-faced people who keep invading from the west. But he does not welcome them. He will fight to his death.

"Texan Mama", my "All Your Yesterdays" contest entry

I would have worked on this a bit longer, but the "All Your Yesterdays" contest deadline is tomorrow.

Texan Mama

Dimetrodon and its kin have often been described as "mammal-like reptiles", but in fact they they are just as closely related to modern reptiles as we are (in terms of shared descent). Creatures like Dimetrodon, Moschops, Lystrosaurus, Cynognathus, Morganucodon, etc. are more properly termed "stem-mammals", meaning that they are not mammals, but are more closely related to mammals than to any other living organisms.

We can infer, in the absence of direct evidence, that all stem-mammals probably possessed any characteristics shared by us mammals and our closest living non-mammalian relatives, the sauropsids: turtles, tuataras, lizards (including snakes), crocodylians, and birds. But mammalian characteristics not shared by sauropsids are trickier. When did hair evolve? When did lactation evolve? We have a few clues but no definite answers.

In this piece, I have pushed fur back to an extremely early time — Dimetrodon is one of the furthest stem-mammals from Mammalia proper. While we know that a later stem-mammal, Estemmenosuchus, had glandular skin without any sign of fur, it is possible that fur evolved earlier and was simply lost or reduced in some lineages, as it has been in many lineages of placental mammal.

I have also posited parental feeding, but not, strictly speaking, lactation. Other lineages of tetrapod, including caecilians and pigeons, have evolved ways of feeding the young from foodstuffs produced by the mother. The mother Dimetrodon's sides are swollen with nutritious substances which seep out as her pups gobble it up. Is it milk? Sort of and sort of not.

Finally, I have scrupulously avoided any suggestion that these are in any way reptilian. They do retain some plesiomorphies evidenced in some reptiles, amphibians, and lungfishes, such as a sprawling gait, belly scales, and acute color vision, but they lack the dry skin and derived scales of true reptiles. Not a single sauropsid texture was used in the photocollage; instead they have the textures of Hippopotamus, Phacochoerus, Procyon, Homo, Zaglossus, Caecilia, Litoria, and even Neoceratodus. These Dimetrodon are moist, glandular creatures — not reptiles at all.

If you're wondering, here's a quick breakdown of the textures used:

Hippopotamus amphibius (hippo)

general, especially mother's head and torso

Phacochoerus (warthog)

general, especially mother's tail and torso

Procyon (raccoon)

soles

Homo sapiens (human)

general, especially mother's sail

Zaglossus (long-beaked echidna)

juveniles' faces

Caecilia (caecilian)

mother's sides and underside

Litoria caerulea (Australian green tree frog)

general, especially mother's limbs

Neoceratodus forsteri (Queensland lungfish)

juveniles' underbelly

I've also submitted a variant, done in a Pop Art style inspired by Roy Lichtenstein's work, to the related Love in the Time of Chasmosaurs: All Yesterdays contest:

Note: Yes, I do descend from a long line of Texan mothers.

JSEN: JavaScript Expression Notation

That idea I was talking about yesterday? Storing mathematical expressions as JSON? I went ahead and made it as a TypeScript project and released it on GitHub:

JavaScript Expression Notation (JSEN)

Still need to complete the unit test coverage and add a couple more features. I made a change from my original post to the syntax for namespace references. (The reason? I realized I needed to be able to use "*" as a local identifier for multiplication.) They work within Namespace declaration blocks, but I need to make them work at the higher level of Namespaces declaration blocks as well. (Done.) I also want to allow functions to be used as namespaces. (Done.)

This is possible right now:

jsen.decl('my-fake-namespace', {
   'js': 'http://ecma-international.org/ecma-262/5.1',

   'x': 10,
   'y': ['js:Array', 1, 2, 3],
   'z': ['js:[]', 'y', 1]
});

jsen.eval('my-fake-namespace', 'x'); // 10
jsen.eval('my-fake-namespace', 'y'); // [1, 2, 3]
jsen.eval('my-fake-namespace', 'z'); // 2

jsen.expr('my-fake-namespace', 'x'); // 10 // Deprecated
jsen.expr('my-fake-namespace', 'y'); // Deprecated
    // ["http://ecma-international.org/ecma-262/5.1:Array", 1, 2, 3]
jsen.expr('my-fake-namespace', 'z'); // Deprecated
    // ["http://ecma-international.org/ecma-262/5.1:[]", "y", 1]

Eventually something like this will be possible as well: