Showing posts with label shell-fish. Show all posts
Showing posts with label shell-fish. Show all posts

15 January 2012

Updates to the Web Apps Shell-Fish and About Tag

I just pushed new versions of Shell-Fish (the online version of Fish) and the About Tag app.

There are some quite significant changes, which I will summarize briefly.

  • Shell-Fish has been updated to the latest version 4.26. It has sat at 4.00.0 for a ridiculously long time, and that became self-sustaining because non-trivial changes were needed to the web version to make it use the new one.

  • The About Tag app now includes a clone of Shell-Fish. Click the bottom tag on the logo to use this.

  • The search functionality in About Tag has been overhauled and remodelled on the much better version used at yet another of my sites, Art of Tagging, which is hosted on the intriguing PythonAnywhere platform. Changes include:

    • When you click a link, the diagram now appears inline
    • If you re-click, it refreshes
    • If you’re signed in, you can tag items and the diagram updates.
  • The changes are quite major, and I haven’t tested them as thoroughly as I would wish, so I’ve probably broken some thing. Sorry. Let me know and I’ll try to fix them.

  • The three sites all use the same model of requiring your Fluidinfo credentials; unfortunately they don’t share databases, so if you want to use all three, you need put them in three times. This is crazy; all three sites should merge, but that will take time.

  • Although the search functionality on the About Tag site mostly works well, Google’s hard 10-second time limit on queries means that queries that match a lot of objects tend to time out. The version of Art of Tagging doesn’t suffer from this limit, but unfortunately that site is quite often down for maintenance.

  • There are many small changes to Shell-Fish but two really major bits of new functionality:

    • Aliases and syncing now work. If you create aliases on a local version, type sync in Shell-Fish and they will appear there, and vice-versa. Among other things, this makes it easy to use sequences and access them from both sites.

    • You can drop the -i and -a in almost all cases. So (in the most common case), rather than:

      fish> tag -a "artist:melody gardot" rating=10

      you can simply use:

      fish> tag "artist:melody gardot" rating=10

      though the old form will, of course, continue to work, along with -q and the new -@ (for anonymous objects).

  • My process of turning everything from XHTML to HTML5 continues. Shell-Fish and the About Tag we app are now pure HTML5, like this blog. Previously, they were all XHTML, which worked for everything under the Sun except Microsoft Browsers; at least now Internet Explorer 9 should be able to use some of the functionality. Users of Internet Explorer 1–8, and therefore of Windows XP, remain out of luck (in more ways than I can possibly enumerate, most of which are rather more significant than not being able to use these web sites).

07 July 2011

About Tags In Fish

I’ve added a new command to fish (and updated the online version, Shell-Fish accordingly) to allow easy construction of standardized about tags using the conventions from the abouttag library. They make use of a new abouttag function, available in the new generic.py file in the abouttag library, which takes the object type as its first parameter, and the usual parameters as a variable parameter list.
The new fish command is abouttag, though can also be abbreviated to about and its general form is:
fish abouttag <object type> <object specifiers>
The object type is something like book, album or fi-user and the object specifiers are the key parameters used to describe that object, in the same order as they are used in the corresponding function from the abouttag library.
The easiest way to illustrate and define these is with examples. The following examples are taken from a Unix system; on Windows, use double quotes rather than single around parameters. In the online version (Shell-Fish), and on Unix, single or double quotes work. In the online version, you don’t need the fish prefix (though it does work).
I should note that part of the motivation for adding this functionality is a desire to allow the command to be used to specify objects without knowing the exact form of their about tags. In Unix-like systems (Linux, Mac OS X, Solaris etc.), this is possible by using left quotes, which can be placed inside double quotes. Thus, the following, slightly ungainly command (using all three forms of quote) works, at least in bash:
$ fish show -F -a "`fish abouttag book 'Gödel, Escher, Bach: An Eternal Golden Braid' 'Douglas R. Hofstader'`" njr/rating
Object with about="book:gödel escher bach an eternal golden braid (douglas r hofstader)":
  njr/rating = 10
I will leave it to the reader to judge whether this is easier than using cut and paste. For those who don’t know about left quotes in Unix shells, a command enclosed in left quotes within another command is evaluted before its enclosing command; its output replaces the left-quoted phrase on the original command line. So in the case above, we first run the command
fish abouttag book 'Gödel, Escher, Bach: An Eternal Golden Braid' 'Douglas R. Hofstader'
which generates
book:gödel escher bach an eternal golden braid (douglas r hofstader)
as its output. In effect, the outer command is then transformed to
fish show -F -a "book:gödel escher bach an eternal golden braid (douglas r hofstader)" njr/rating
I hope to extend shell-fish, the on-line version of fish, to support left quotes, but that may take a little while.
The following examples are taken from the fish documentation, which is available online from http://fluiddb.fluidinfo.com/about/fish/fish/index.html.
  1. Books and related items using the book-u convention (book, author)
    $ fish abouttag book 'Gödel, Escher, Bach: An Eternal Golden Braid' 'Douglas R. Hofstader'
    book:gödel escher bach an eternal golden braid (douglas r hofstader)
    
    $ fish abouttag book 'The Feynman Lectures on Physics' 'Richard P. Feynman' 'Robert B. Leighton' 'Matthew Sands'
    book:the feynman lectures on physics (richard p feynman; robert b leighton; matthew sands)
    
    $ fish abouttag book 'The Oxford English Dictionary: second edition, volume 3', 'John Simpson', 'Edmund Weiner'
    book:the oxford english dictionary second edition volume 3 (john simpson; edmund weiner)
    
    $ fish abouttag author 'Douglas R. Hofstadter' 1945 2  15
    author:douglas r hofstadter (1945-02-15)
  2. Music-related items (track, album, artist, isrc-recording)
    $ fish abouttag track 'Bamboulé' 'Bensusan and Malherbe'
    track:bamboulé (bensusan and malherbe)
    
    $ fish abouttag album 'Solilaï' 'Pierre Bensusan'
    album:solilaï (pierre bensusan)
    
    $ fish abouttag artist 'Crosby, Stills, Nash & Young'
    artist:crosby stills nash & young
    
    $ fish abouttag isrc-recording 'US-PR3-73-00012'
    isrc:USPR37300012
  3. URLs and URIs (URI, URL)
    $ fish abouttag uri FluidDB.fluidinfo.com
    http://fluiddb.fluidinfo.com
    
    $ fish abouttag url https://FluidDB.fluidinfo.com/one/two/
    https://fluiddb.fluidinfo.com/one/two
    
    $ fish abouttag URI http://fluiddb.fluidinfo.com/one/two/
    http://fluiddb.fluidinfo.com/one/two
    
    $ fish abouttag URL 'http://test.com/one/two/?referrer=http://a.b/c'
    http://test.com/one/two/?referrer=http://a.b/c
  4. Fluidinfo objects (fi-user, fi-namespace, fi-tag)
    $ fish abouttag fi-user njr
    Object for the user named njr
    
    $ fish abouttag fi-namespace njr/misc
    Object for the namespace njr/misc
    
    $ fish abouttag fi-ns njr/private
    Object for the namespace njr/private
    
    $ fish abouttag fi-tag terrycojones/private/rating
    Object for the attribute terrycojones/private/rating
  5. Database components (db-table, db-field)
    $ fish abouttag db-table 'elements'
    table:elements
    
    $ fish abouttag db-field 'name' 'elements'
    field:name in table:elements
  6. Miscellaneous (planet, element)
    $ fish abouttag planet 'Mars'
    planet:Mars
    
    $ fish abouttag element 'Helium'
    element:Helium

25 June 2011

Like A Scrolling Terminal

I have just updated Shell-Fish, the on-line version of the Fluidinfo Shell, fish, so that it acts rather more like a scrolling terminal, and rather less like a search engine.

Instead of being a “one-shot” application, where each time you type a fish command the screen is wiped and you lose all previous context, fish now simply scrolls the output, just a like a terminal. If you’re a command-line kind-of-a person, it will all look deeply familiar, except for the merest hint of AJAX spinniness.

The original inspiration for this was Stefan Grothcop’s wonderful Google Shell, goosh, which I’ve long used as my main interface to Google on Firefox. If you haven’t tried goosh, what are you waiting for?

Although the new scrolling version of Shell-Fish appears to work fine, I expect there are bugs: let me know if you find any. It still lacks a few basics, most obviously a history, but I plan to add that. Globbing too.

All in good time.

19 June 2011

Securing shell-fish with fish: Even More on Fluidinfo Permissions

I mentioned in my last post that I’ve extended the range of permissions that can be set with the perms command to cover the whole gamut. Here, I’ll describe this in the context of showing how I have set up permissions for the fish user to make them suitable for use with the online version of fishshell-fish.

My goal was this. Shell-fish, the online version of fish, allows users to log in with their Google account and effectively link it to one or more Fluidinfo accounts. Users who do this can use fish to perform arbitrary Fluidinfo operations using their own tags. But I also wanted to allow users to try out Fluidinfo and fish without setting up an account. Initially, I did this using the Fluidinfo test user, but that’s probably not a good long-term solution. What I’d prefer to do is to provide a locked down account that anyone can use for certain things—not only for reading but also to try out tagging. The process of locking down a Fluidinfo account is also useful as an illustration of the extended perms command in fish, so that is the subject of this post.

I created the Fluidinfo fish user and switched to it in fish.

$ fish su fish
Credentials set to user fish.

$ fish ls -ld fish
nrwcr--r--   fish/

As you can see the permissions on fish’s namespace are the defaults, with the user having read, write and control permission, and everyone else having read only. (See this post or the fish documentation for the ls command).

The first thing I wanted to do was to transfer control permission from the fish user to me (njr) so that a user of Shell-Fish can’t do arbitrary things. This wasn’t possible with the original perms command, but I’ve added three new forms of the command, each of which follows the general template:

fish perms perms-class [open|closed] [except list+of+users] list of tags or namespaces

where perms-class is one of read, write or control. This simply lets the user set the explicit FLuidinfo permissions, as open or closed, with an optional exception list. The exception list is specfied as a list of users, separated by + signs. The command only changes the permissions class specified and changes all the permissions in that class for the tags or namespaces given. So valid examples operating on a tag fish/rating and a namespace fish/private include:

fish perms read closed except fish fish/rating fish/private
fish perms write closed fish/rating fish/private
fish perms control closed except fish+njr fish/rating fish/private

To transfer the control permissions to njr for the top-level fish namespace, I used the command (still as the fish user at this point):

$ fish perms -f control closed except njr fish

The only extra thing to notice here is the -f flag, which forces fish to make a change it would otherwise resist. Any change to permissions that results in the owner of a tag or namespace not having control permissions falls into this category, and since this is usually undesirable, fish requires the -f flag to force this to happen. (Tranferring permission to njr is, of course, irreversable for fish.)

Having made this change, if we repeat the ls command we see this:

fish ls -ld fish
(denied)    fish/

Fluidinfo allows only users with control permission to see the permissions on tags and namespaces, so the fish user can no longer see them. (I suppose this is slightly odd: it means that you can’t even see that you have write permission if you don’t have control permission. But that’s not a huge problem.)

Let’s switch to njr and try.

$ fish su njr
Credentials set to user njr.

$ fish ls -ld fish
nrw-r-cr--   fish/

This is as expected. The owner—fish—no longer has control permission on the fish namespace, but someone in a group does. We know that someone is njr. (We could verify this with ls -L, as we will after making some more changes.)

So far so good.

In terms of writing, my initial idea is to provide a set tags with single-letter names (a to z) in fish‘s top-level namespace that fish can write, but not to allow other tags or namespace to be written.

First, let’s create those tags as the fish user.

$ fish su fish
Credentials set to user fish.

$ fish -U touch a b c d e f g h i j k l m n o p q r s t u v w x y z

The touch command on a non-existent tag creates the tag, and the -U says use Unix-style paths, i.e. assume they’re in the authenticated user’s namespace unless introduced with a leading /. So this creates the 26 single-letter tags I want.

Next, we need to remove write permission from fish on the fish namespace, so that it can’t create new tags or namespaces.

$ fish su njr
Credentials set to user njr.

$ fish perms write closed except njr fish

$ fish ls -ld fish
nr--rwcr--   fish/

Now let’s think about the permissions on the tags we have just created. There is currently nothing hierarchical about Fluidinfo’s permissions. So switching back to the fish user, we see:

$ fish su fish
Credentials set to user fish.

$ fish ls -l
trwcr--r--   fish/a
trwcr--r--   fish/b
...
trwcr--r--   fish/z

So even though njr has control of the fish namespace, the fish user has control of the tags it created. I don’t want that, so let’s transfer them over:

$ fish -U perms -f control closed except njr a b c d e f g h i j k l m n o p q r s t u v w x y z

$ fish ls -l a
trw-r-cr--   fish/a

[Again, the -U saves me having to prefixed each tag with njr/. Of course, adding globbing (wild-carding) to fish would be even more helpful; I’ll probably do that soon, though interaction with the Unix shell’s globbing will be a slight issue there.]

So this is pretty good: the fish user has read and write permissions, njr has read and control, and everyone else has read only.

The one remaining problem is that there are several different write permissions on tags—permission to tag things with the tag (in native Fluidinfo API terms, create permission on /tag-values), permission to untag objects currently tagged with the tag (delete permission on /tag-values) permission to change the description of the tag (update permission on /tags) and permission to delete the tag itself (delete permission on /tags). The last of these is the problem: I don’t really want a random user of the Shell-fish tags to be able to delete any of fish/a to fish/z. To start with, fish couldn’t recreate them (because I’ve removed write permission from fish on the fish namespace) and if it did, fish would get control permission over that tag, which is what I’ve been trying to avoid. So I need to remove the fine-grained Fluidinfo permission to annihilate the tags from fish.

In order to allow this, I added a -X option to the perms command. This “eXtra” specification restricts which particular permissions within the class specified should actually be modified. -X is followed by the fish name for a low-level Fluidinfo permission, as shown in the ls -L “longer” listing. Let’s look at the detailed permissions on one of our tags:

$ fish su njr
Credentials set to user njr.

$ fish ls -L /fish/a

fish/a:

ABSTRACT TAG (/tags)
  Write
    update (metadata):  policy: closed; exceptions = [fish]
    delete (delete):    policy: closed; exceptions = [fish]
  Control
    control (acontrol): policy: closed; exceptions = [njr]

TAG (/tag-values)
  Read
    read (read):        policy: open; exceptions = []
  Write
    create (tag):       policy: closed; exceptions = [fish]
    delete (untag):     policy: closed; exceptions = [fish]
  Control
    control (tcontrol): policy: closed; exceptions = [njr]

Here, the Fluidinfo name for the low-level permission is shown first, and then parentheses, the fish name is shown. The fish names are unique for each kind of permission that exists on tags, and the one we are concerned with is the ability to delete the tag itself—the abstract tag. This has the same name, delete, in both systems. So I used the command:

$ fish perms -f write -X delete closed except njr /fish/a

$ fish ls -L fish/a

fish/a:

ABSTRACT TAG (/tags)
  Write
    update (metadata):  policy: closed; exceptions = [fish]
    delete (delete):    policy: closed; exceptions = [njr]
  Control
    control (acontrol): policy: closed; exceptions = [njr]

TAG (/tag-values)
  Read
    read (read):        policy: open; exceptions = []
  Write
    create (tag):       policy: closed; exceptions = [fish]
    delete (untag):     policy: closed; exceptions = [fish]
  Control
    control (tcontrol): policy: closed; exceptions = [njr]

As you can see, this changed only the delete permission from the specified write class, which is what we wanted. (If I had wanted to change several, I could have repeated the -X option.) I repeated this using the other 25 tags. If we now look at the result using ls -g, we get this:

$ fish ls -g /fish
tr/-r/cr--   fish/a
tr/-r/cr--   fish/b
...
tr/-r/cr--   fish/z

Reading across:

  • The t means that this is tag
  • The r/- means that the tag’s owner, fish, has read and some write permissions on the tag—in this case, all except the delete permission on the abstract tag
  • The r/c means that there is some group of users with control and read permissions, but only some write permissions. That group is in fact the singleton [njr], and he has delete permission only. (I could have actually just made the delete policy closed—no one needs to be able to delete the tag, and I have control anyway. Equally, I could have given myself all the other write permissions; but I didn’t.)
  • The final r-- means that the world has read permission, but not write or control.

So that’s it. The fish user can now tag and untag things using the tags fish/a through fish/z but has no ability to create other tags or namespaces, or delete the tag itself, and does not have control permission. The fish user can also in principle change tag descriptions, though fish doesn’t provide a way of doing that for existing tags right now.

I have just switched over shell-fish to use the fish user for non-authenticated users, so you can try this out now. In passing, if you use ls to list the fish namespace, you’ll see this:

_static/        genindex.html   mkdir.html      s               unixlike.html
a               h               mkns.html       search.html     untag.html
b               help.html       n               searchindex.js  v
c               i               o               show.html       version.html
cli.html        index.html      p               su.html         w
commands.html   install.html    perms.html      t               whoami.html
count.html      j               pwd.html        tag.html        x
d               k               pwn.html        tags.html       y
e               l               q               test.html       z
f               ls.html         r               touch.html
g               m               rm.html         u

If you’re wondering what all these tags-that-look-like-web-pages are, they’re tags that I use to store the HTML fish documentation in Fluidinfo itself. If you look carefully at the documentation path — http://fluiddb.fluidinfo.com/about/fish/fish/index.html — you will see that its root is a tag on the Fluidinfo object having the about tag fish. The name of that tag is fish/index.html, and its content is the HTML for the index page. The trick lies partly in the tag name and partly in setting its content type to text/html.

I have a script that does that, which will probably become a fish command called Something like upload or publish or files2fi.

15 June 2011

Of Fish, Shell-Fish and Fish Py

FluidDB is dead; long live Fluidinfo.

Whither fdb?

Obviously, fdb should become fi; it’s perfect. Thirty-three-and-a-third per cent shorter is 33⅓% better for a command-line command. And fi is just so beautiful. It could almost become a ligature: how perfect would fi be?

Except, of course, there’s one tiny problem. In Unix shells, fi is reserved as the closing counterpart to if. Even if I could make fi kinda, sorta work, I wouldn’t want to. The closing counterpart to if should be fi; it’s part of the cosmic order.

So what to do? The procrastinator’s dictum to the rescue:

Why put off till tomorrow that which doesn’t really need to be done until the day after that?

Why does fdb need to change at all? It could be a throwback, a reminder of glories past, a piece of Fluidinfo’s cultural legacy (along with the fluiddb superuser).

That’s what I thought.

Until I decided to put fdb into the sky. I’ve long thought it would be cool to have a browser-based version of fdb that anyone could simply use without installation. “No software”, as Salesforce.com likes to say.

For better or for worse, I tend to use Google’s App Engine to write web apps at the moment, so I went to register a new app there.

In Search of a Google App Engine App Name

Unfortunately, registering a new app is a bit like picking a domain; most of the desirable onare are gone already, not helped by the fact that all google usernames are considered taken. Add to that a minimum-of-six-characters requirement, and fdb looks to be in trouble.

As I was doing all this, I was chatting online with Terry (@terrycojones), who is the leading advocate of taking the DB out of FluidDB, and who, while entirely willing for me to plough my own furrow, had a very clear preference for expunging the db from fdb too.

Clearly, in reality, fdb is a shell for Fluidinfo. Unix has a long history of shells. In roughly chronological order I have used sh (the original “Bourne” shell), csh (the C shell), tcsh, ssh (Simon’s shell; not the Secure Shell; though I use that daily too), and now, always, nearly exclusively, bash, the truly wonderful Bourne-Again Shell. I’ve also dabbled with ksh, zsh and no doubt various others that fall into the large things-I-used-to-know category.

So give this, what would you call a shell for Fluidinfo? It just has to be fish. It’s screaming out to be fish. The only problem is that it’s thirty-three per cent worse (33% more typing).

Well, the fact that it’s 33% worse and a bit fishy.

Well, the fact that it’s 33% worse and a bit fishy and isn’t actually long enough to be a Google App Engine ID. (Too long and yet too short; not long enough and yet altogether too long. Such a paradox.)

FDB’s Fate Sealed by a Typo

As I was checking availability on App ID after App ID, I eventually got to shell-fish. Now shell-fish is just silly. I mean, it’s redundant (shell-Fluidinfo shell?). It’s hyphenated. It’s even fishier than fish. It sounds like a drunken version of selfish. Clearly, no person in his right mind was never going to choose shell-fish.

But then, instead of clicking the “Check Availability Button”, I typed return. And discovered that shell-fish was available. And that I had registered it.

Now, give me some credit. I do appreciate that this wasn’t really it. I could have changed it. But it could have been worse. I checked availability of fishnet (taken) and fish-net (available) and countless dozens I’ve have to go back to the IRC logs to recall so memorable were they. But in the end, I wasn’t convinced that I was gong to do better than shell-fish. And it does lock in fish, which is the perfect name for the Fluidinfo Shell—well, except for being 33% worse, and fishy, and not available as a Google App Engine ID, and . . .

Shell-Fish

So there it is. If you wish to be a guinea pig, head on over to http://shell-fish.appspot.com, where you can try fish online. It’s mostly the same as fdb was, and fish is, except that

  • you don’t need to prefix commands with fdb (or fish), obviously.
  • you are subject to the Google App Engine, 5–10 second maximum for an HTTP request. This can be an issue; timeouts are not uncommon, especially for complex queries, and when Fluidinfo is under load.

It’s almost certainly buggy and subject to change.

Right now, if you don’t log in, you will use the Fluidinfo test user. Before too long (when registrations are fixed), it’ll be a different user. But you can log in using your own Fluidinfo credentials if you like.

The way you do that is that you log into the appliction using a Google Account. (My app doesn’t get to see your Google password.)

Then, if you go into settings, you can add one or more Fluidinfo accounts by specifying your username and password. (You can also choose whether to use the default, Fluidinfo-style full paths for all tags and namespaces (njr/rating etc.) or whether your own tags and namespaces will be abbreviated to rating etc., at the cost of having to use a leading / for other people’s (/ntoll/rating etc.).

IMPORTANT: PASSWORD SECURITY

If you register a Fluidinfo username and password, shell-fish will store these in Google’s data store. I’m not particularly comfortable either with asking people for their passwords or with storing them, but I don’t think there’s much alternative at the moment. (There may be in an OAuth future.)

Obviously, before you hand over your password, you need to consider a few things:

  • Do you trust me? I could steal your password.
  • Do you trust fish? Even if you think me worthy of your trust, do you consider me competent? [Disclosure: sometimes, I make mistakes. See the discussion above on how shell-fish got its name.]
  • Are you happy with your password living in Google’s data store?

On the last point, I have taken what might be called minimal precautions. I do not store your password in plain text, partly so that should anyone happen to gain access to Google’s data store, they won’t just be able to read your password, and even more so that if I browse the shell-fish Google data store, I won’t inadvertantly see your password. (I’d have to decide to be evil.)

But I should also admit that what I’ve done to the password, while presumably technically qualifying as encryption, would probably be more accurately termed obfuscation. Let’s put it this way: it’s better than ROT-13, but it’s not as strong as PGP.

The other thing to know is that when you remove a user from your shell-fish settings, I simply the datastore (well, shell-fish tells the data store) to delete the record. I certainly don’t have access to it after that; whether a DrEvil@google.com could recover it, I know not.

Of Fish, Shell-Fish and Fish Py

So there it is.

I am in the process of changing the name of fdb to fish. (In fact, I’ve done it locally, but I don’t plan to push it to Github for a few days.)

The web app shell-fish is available at http://shell-fish.appspot.com for brave early adopters. I’m fiddling with it all the time. It will change a lot (most importantly, I hope it will end up looking more like a scrolling terminal than a one-shot search engine—think Goosh rather than Google. But right now, it’s very Google.

As fdb becomes fish, so will fdb.py become fish.py (geddit?).

One more thing . . . Amazon Product Pages

I’ll blog about this separately, but even if you don’t want to use fish per se, you might be interested in one neat little experimental feature.

At the top of the page in shell-fish, there’s a link called az-fish (though it might become amazon-fish). This is bookmarklet. Don’t click on it on the page; rather drag it to your tool bar. (It works even if you don’t give fish details of your Fluidinfo account.)

Then, click it when you are on an Amazon product page for a book, an eBook, a CD or an MP3 track. (I’ve only tested it on Amazon UK and Amazon US; it will probably need to tweaked for others, especially for non-English others.) It will take the URL and give it to fish (at shell-fish), which will attempt to figure out the about tag for the corresponding book, album or track in Fludiinfo, using its new amazon command (which may eventually become a thing command.)

How cool is that?

Labels