Of Fish, Shell-Fish and Fish Py

FluidDB is dead; long live Fluidinfo.

Whither fdb?

Obviously, fdb should become fi; it’s perfect. Thirty-three-and-a-third per cent shorter is 33⅓% better for a command-line command. And fi is just so beautiful. It could almost become a ligature: how perfect would fi be?

Except, of course, there’s one tiny problem. In Unix shells, fi is reserved as the closing counterpart to if. Even if I could make fi kinda, sorta work, I wouldn’t want to. The closing counterpart to if should be fi; it’s part of the cosmic order.

So what to do? The procrastinator’s dictum to the rescue:

Why put off till tomorrow that which doesn’t really need to be done until the day after that?

Why does fdb need to change at all? It could be a throwback, a reminder of glories past, a piece of Fluidinfo’s cultural legacy (along with the fluiddb superuser).

That’s what I thought.

Until I decided to put fdb into the sky. I’ve long thought it would be cool to have a browser-based version of fdb that anyone could simply use without installation. “No software”, as Salesforce.com likes to say.

For better or for worse, I tend to use Google’s App Engine to write web apps at the moment, so I went to register a new app there.

In Search of a Google App Engine App Name¶

Unfortunately, registering a new app is a bit like picking a domain; most of the desirable onare are gone already, not helped by the fact that all google usernames are considered taken. Add to that a minimum-of-six-characters requirement, and fdb looks to be in trouble.

As I was doing all this, I was chatting online with Terry (@terrycojones), who is the leading advocate of taking the DB out of FluidDB, and who, while entirely willing for me to plough my own furrow, had a very clear preference for expunging the db from fdb too.

Clearly, in reality, fdb is a shell for Fluidinfo. Unix has a long history of shells. In roughly chronological order I have used sh (the original “Bourne” shell), csh (the C shell), tcsh, ssh (Simon’s shell; not the Secure Shell; though I use that daily too), and now, always, nearly exclusively, bash, the truly wonderful Bourne-Again Shell. I’ve also dabbled with ksh, zsh and no doubt various others that fall into the large things-I-used-to-know category.

So give this, what would you call a shell for Fluidinfo? It just has to be fish. It’s screaming out to be fish. The only problem is that it’s thirty-three per cent worse (33% more typing).

Well, the fact that it’s 33% worse and a bit fishy.

Well, the fact that it’s 33% worse and a bit fishy and isn’t actually long enough to be a Google App Engine ID. (Too long and yet too short; not long enough and yet altogether too long. Such a paradox.)

FDB’s Fate Sealed by a Typo¶

As I was checking availability on App ID after App ID, I eventually got to shell-fish. Now shell-fish is just silly. I mean, it’s redundant (shell-Fluidinfo shell?). It’s hyphenated. It’s even fishier than fish. It sounds like a drunken version of selfish. Clearly, no person in his right mind was never going to choose shell-fish.

But then, instead of clicking the “Check Availability Button”, I typed return. And discovered that shell-fish was available. And that I had registered it.

Now, give me some credit. I do appreciate that this wasn’t really it. I could have changed it. But it could have been worse. I checked availability of fishnet (taken) and fish-net (available) and countless dozens I’ve have to go back to the IRC logs to recall so memorable were they. But in the end, I wasn’t convinced that I was gong to do better than shell-fish. And it does lock in fish, which is the perfect name for the Fluidinfo Shell—well, except for being 33% worse, and fishy, and not available as a Google App Engine ID, and . . .

Shell-Fish¶

So there it is. If you wish to be a guinea pig, head on over to http://shell-fish.appspot.com, where you can try fish online. It’s mostly the same as fdb was, and fish is, except that

• you don’t need to prefix commands with fdb (or fish), obviously.
• you are subject to the Google App Engine, 5–10 second maximum for an HTTP request. This can be an issue; timeouts are not uncommon, especially for complex queries, and when Fluidinfo is under load.

It’s almost certainly buggy and subject to change.

Right now, if you don’t log in, you will use the Fluidinfo test user. Before too long (when registrations are fixed), it’ll be a different user. But you can log in using your own Fluidinfo credentials if you like.

The way you do that is that you log into the appliction using a Google Account. (My app doesn’t get to see your Google password.)

Then, if you go into settings, you can add one or more Fluidinfo accounts by specifying your username and password. (You can also choose whether to use the default, Fluidinfo-style full paths for all tags and namespaces (njr/rating etc.) or whether your own tags and namespaces will be abbreviated to rating etc., at the cost of having to use a leading / for other people’s (/ntoll/rating etc.).

IMPORTANT: PASSWORD SECURITY

If you register a Fluidinfo username and password, shell-fish will store these in Google’s data store. I’m not particularly comfortable either with asking people for their passwords or with storing them, but I don’t think there’s much alternative at the moment. (There may be in an OAuth future.)

Obviously, before you hand over your password, you need to consider a few things:

• Do you trust me? I could steal your password.
• Do you trust fish? Even if you think me worthy of your trust, do you consider me competent? [Disclosure: sometimes, I make mistakes. See the discussion above on how shell-fish got its name.]
• Are you happy with your password living in Google’s data store?

On the last point, I have taken what might be called minimal precautions. I do not store your password in plain text, partly so that should anyone happen to gain access to Google’s data store, they won’t just be able to read your password, and even more so that if I browse the shell-fish Google data store, I won’t inadvertantly see your password. (I’d have to decide to be evil.)

But I should also admit that what I’ve done to the password, while presumably technically qualifying as encryption, would probably be more accurately termed obfuscation. Let’s put it this way: it’s better than ROT-13, but it’s not as strong as PGP.

The other thing to know is that when you remove a user from your shell-fish settings, I simply the datastore (well, shell-fish tells the data store) to delete the record. I certainly don’t have access to it after that; whether a DrEvil@google.com could recover it, I know not.

Of Fish, Shell-Fish and Fish Py¶

So there it is.

I am in the process of changing the name of fdb to fish. (In fact, I’ve done it locally, but I don’t plan to push it to Github for a few days.)

The web app shell-fish is available at http://shell-fish.appspot.com for brave early adopters. I’m fiddling with it all the time. It will change a lot (most importantly, I hope it will end up looking more like a scrolling terminal than a one-shot search engine—think Goosh rather than Google. But right now, it’s very Google.

As fdb becomes fish, so will fdb.py become fish.py (geddit?).

One more thing . . . Amazon Product Pages¶

I’ll blog about this separately, but even if you don’t want to use fish per se, you might be interested in one neat little experimental feature.

At the top of the page in shell-fish, there’s a link called az-fish (though it might become amazon-fish). This is bookmarklet. Don’t click on it on the page; rather drag it to your tool bar. (It works even if you don’t give fish details of your Fluidinfo account.)

Then, click it when you are on an Amazon product page for a book, an eBook, a CD or an MP3 track. (I’ve only tested it on Amazon UK and Amazon US; it will probably need to tweaked for others, especially for non-English others.) It will take the URL and give it to fish (at shell-fish), which will attempt to figure out the about tag for the corresponding book, album or track in Fludiinfo, using its new amazon command (which may eventually become a thing command.)

How cool is that?

DRM, Readability and the Ownership of Bits

Summary¶

It is not only pirates and thieves who should be deeply worried by DRM.

We need legal protection of our bits (our digital information).

As we rush (inevitably) towards digital everything, there are real risks of losing everything. As a society, we should be putting in place disaster recovery plans for our culture, using safer and longer-established technologies such as paper.

DRM¶

The debate on DRM too often seems to be a private argument between two kinds of parasites. In the red corner, we hear endlessly from the consumer parasites, people who think they have some God-given right to unpaid access to anything that anyone creates. In the blue corner, we hear equal and opposite special pleading from parasitic dinosaur executives whose only real wish is to keep suckering the public into paying them yet again, as the gatekeepers, for works they’ve already bought in several different formats without ever actually gaining any kind of right to content. But DRM concerns us all; and should be of concern to all of us.

Officially, DRM stands for Digital Rights Management. But this is a peculiarly Orwellian inversion. From the majority perspective, as Richard Stallman points out, the term is more easily understood if the word “rights” is replaced with “restrictions”. For DRM systems are not munificent helpers designed to make protect the ability of the purchaser to use that which she has purchased. They are not helpful gofers that will search out a suitable decoder to allow you to use your legally purchased content on a different computer. DRM systems are not dedicated to ensuring that your DVD will play in any DVD player in the world if you have legally purchased the right to watch the DVD. They are not like escrows, designed to ensure that even if the company that sold you the content goes out of business, your ability to utilize that content will not be compromised. If DRM were any of these things, the word “rights” might be appropriate. In reality, the only rights DRM systems have any concern for at all are the “rights” of the “rights” holders—the record companies, film studios and publishers, rather than the artists, in most cases—to restrict, disable, and destroy.

If you doubt the malice of DRM systems and the umbrella of related technologies they exemplify, Kindlegate lays bare the toxic nature of the systems that are increasingly invading our lives.

Kindlegate and the Unselling of Orwell¶

In the summer of 2009, Amazon Kindle customers who had purchased George Orwell’s Nineteen Eighty Four and Animal Farm found, one day, that these works had disappeared from their Kindles. This was not the result of some horrible software glitch. Amazon deleted them. Amazon deliberately deleted them. Amazon, having taken these people’s money, and sold them a “Kindle Edition” e-book, decided to unsell them. [1] It simply reached into these readers’ Kindles and erased the relevant bits. Clearly there is no “all sales are final” notice on amazon.com. Amazon did not ask its Kindle customers to delete them for a refund (though it did refund them). Amazon did not produce a warrant and come round with police officers. It just reached a digital arm into its customers’ property and took them. The irony could only have been more complete if Farenheit 451 had also been deleted.

In “justifying” this action, Amazon said that the works by Orwell had been pulled because the Kindle publisher did not own the rights.

“When we were notified of this by the rights holder, we removed the illegal copies from our systems and from customers’ devices, and refunded customers.”

— Drew Herdener, a spokesperson with Amazon.com, quoted by Bobbie Johnson in The Guardian, 17th July 2009

Amazon’s CEO, Jeff Bezos, is no fool, and sounds as if he was almost as horrified as I and others were by this action by Amazon. He wrote this, which is better than I would expect from almost anyone else in his position.

This is an apology for the way we previously handled illegally sold copies of ‘1984’ and other novels on Kindle. Our ‘solution’ to the problem was stupid, thoughtless and painfully out of line with our principles. It is wholly self-inflicted, and we deserve the criticism we’ve received. We will use the scar tissue from this painful mistake to help make better decisions going forward, ones that match our mission.

With deep apology to our customers,

— Jeff Bezos, CEO & Founder, Amazon.com, Kindle Community, 23rd July 2009.

It’s a pretty good statement and rings true to me. But it doesn’t say what Amazon should have done. It doesn’t say what Amazon will do next time a similar situation arises. More importantly, It doesn’t promise never again to reach into people’s devices and reset their bits without their permission. The statement is pretty good for a CEO in Bezos’s position; but it still falls woefully short of providing any real reassurance.

The Threats to our Bits, our Culture, our Knowledge¶

I do not believe there is any future in opposing or resisting the migration of almost all forms of creative, cultural and intellectual products to digital form. This is partly because I think the benefits of digital works on balance outweigh the downsides. But it is also because I think the transition to digital forms is a historical inevitability. Just as we cannot uninvent the bomb, we cannot uninvent digital music, text or images, and happily, we have much less reason to do so.

We can, however, think carefully about how we maximize the benefits from digital technologies and ameliorate the concomitant risks. We can use legal, financial and societal measures to try to ensure our digital future is bright.

As we move from storing books and music on paper and vinyl to storing them as configurations of bits on various kinds of computer memory, we face various significant risks, both as individuals and as societies—even as a species.

Broadly, the threats to our information come from disasters and accidents, from (illegal) sabotage, from legalized sabotage, from technical obsolescence and from insouciance and the inability or unwillingness of our law-makers to challenge vested interests.

Owning our Bits¶

As ever more of that which we value migrates from the domain of matter—whose properties, durability and character we have learned over millennia—to the digital domain of ones and zeroes, with which we have only short experience, we need to start treating bits rather more seriously. We have learned over centuries to increase the durability of non-digital information, both through physical measures (acid-free paper, stable inks, reprints and so on) and legal protections (freedom of speech, public libraries, central libraries, fair use doctrines and so forth).

Whatever the unread, unreasonable and unjust licence agreement that Kindle owners presumably assent to may say, it is outrageous that Amazon could have any kind of right willfully to destroy information on its customers’ devices. It is wrong. It would not happen in the domain of matter. Had I purchased a paperback copy of Nineteen Eight Four from Amazon, and the company had subsequently discovered that Penguin Books had in fact infringed copyright by printing and selling the book to Amazon, this would not have given Amazon the right to come into my house, retrieve the book and leave a fiver in the hall on the way out by way of compensation. In the domain of atoms and matter, we have long established ways of dealing with these sorts of situations, and they tend to involve the police, courts, warrants, discussion and notions of natural justice. I can think of nothing in the established world of atoms even vaguely similar to Amazon’s casual unselling of its Kindle editions.

So the first thing we need is for it simply to be illegal for Amazon or anyone else to alter information on our devices (or, for that matter, in our “cloud” storage) without either our explicit, meaningful and uncoerced consent, or the benefit of some suitable instrument of law (by which, for avoidance of doubt, I do not mean a clickthrough licence agreement).

DRM: Tame it or Kill It¶

The state of DRM systems is a kind of Wild West where companies can and do try almost anything. Sony has been installed rootkits (a form of malware) on computers as part of its DRM efforts; DVDs are deliberately “region encoded” so that they can only be played on certain players; music is tied to a particular computer and can only be transferred to a limited number (if any) of others; printers will only use ink from cartridges with the right code on a chip; some e-books from Amazon cannot be read aloud by the dalek-voiced Kindle because the audio right is not included with the ebook.

Mummy, will you read me a story please?

Certainly dear. What story would you like?

Thomas the Tank Engine! Thomas the Tank Engine!

Oh, I’m sorry, but your Thomas the Tank Engine books don’t include the audio right so I can’t read that one to you. How about the King James edition bible? There’s no DRM on that. Or the GNU Public License, perhaps? I’m sure there wouldn’t be a problem reading that aloud!

Even as basic an operation as backing up a DVD is prohibited, as is ripping it to play it on an iPad or a phone. Legally purchased DVDs are so painful to use that on the rare occasions I have bought them I have sometimes given up before getting to the film. Some others I know purchase DVDs and then watch the painless torrent. This widely shared imgur image summarizes the absurdity of today’s situation well.

Microsoft’s “Genuine Advantage” program so annoyed customers who had purchased legitimate licenses for Microsoft products but found themselves unable to use them that Microsoft has now quietly dropped the scheme.

Where will it end?

Lessons from Music¶

As everyone knows, DRM didn’t go to plan for the music industry, and interestingly is now seems to be disappearing. It was always odd. The industry was terrified by the threat of perfect digital copies proliferating without money flowing back to them, so for years resisted selling music “digitally”. Except that it didn’t. The record industry happily sold high-quality, non-DRM infected music all the time they claimed the skies would fall in if they did just that. It’s just they sold them in the reassuringly familiar and solid form of matter. They sold their bits as CDs.

We will never know what would have happened if instead of burying its head in the ground, adopting DRM, dragging its heels, suing its customers and making up numbers about lost sales that not even a fool a hurry would be taken in by, the record industry had embraced the inevitable and started doing something more like Apple did with iTunes. My guess is that it would probably be in considerably better shape than it is now.

It amazes me that the book industry, having watched what happened in music, seems to have drawn the conclusion that DRM is the way forward. I think it is deluded, and that things will probably play out much as they did in music, i.e. in a few years time the DRM will get dropped, and the sky will not fall in. There will be some piracy, probably more that there was in print, just as there is in music. But there will also be savings and long-tail benefits, and as with music, eventually the book industry will realise that, in general, people understand that writers and their support infrastructures need to be paid; as long as e-books are convenient and reasonably priced and of high quality, most people will be happy to pay for them.

I have the highest respect and regard for those who produce the music, the books the films and art that form the bedrock of our culture, and very much want to see them properly rewarded. One of the exciting possibilities arising from new technologies is that artists can do more with less, and can retain more direct control with smaller support structures. It is quite possible for the digital revolution to provide more income for creators and their aides while lowering prices overall or tolerating a degree of piracy. The DRM systems being peddled, mostly not by creators, but by those who end up distributing their works, are wholly disproportionate; their insidious effects and potential for abuses even more severe than we have seen thus far are too serious for us to ignore. They also carry long-term risks, such as information becoming unusable as hardware and software changes and as companies go out of business. This danger also exists with open formats, but is far smaller and dramatically easier to protect against.

For our own sakes, our children’s sakes, and our culture’s sake, we need, as a society, to face up to the serious dangers that are intrinsic to DRM and its fellow travellers.

[1]	There’s no word “unsell” you say? There is now.

If books were like e-books...