31 July 2011

The End of the Fish Wars: The Fluidinfo Shell Version 4.0

A funny thing happened. Terry (Jones; Fluidinfo creator, founder, CEO etc.) started using Fish and declared that he preferred using Unix-style paths in it. Given that the whole reason for adding support for non-Unix-style paths was the distress Unix-style paths appeared to cause Terry, this was a happy surprise. It turns out that I slightly misunderstood his position.
So with his complete blessing, the latest version of Fish defaults to Unix-style paths, i.e. except in queries, if a tag path does not start with a leading /, it is assumed to be in the authenticated user’s namespace; paths for tags and namespaces of other users are introduced with a leading /. So if I use fish, my rating tag is rating, but Terry’s is /terrycojones/rating.
If you already use fish, this affects you only if you have been using Fluidinfo-style paths without setting that explicitly in your ~/.fluidDBcredentials file. If that’s the case, and you want to continue using Fluidinfo-style long paths, add a third line to that file as follows:
username
password
unix-style-paths false
Users of te online version, Shell-Fish, won’t see any difference when they’re logged in as the preference will be remembered. But if you use the fish user, it now uses Unix-style paths.
The new consensual goal is to make Fish report as much as it can using Fluidinfo-style paths, but to default to unix-style for input. So the abbreviated form is seen primarily as a shortcut for input (which is, indeed, what it is). The complication is that this makes it harder for Fish to consume its own output; so I will make changes slowly, trying to ensure that things like command substitution with left quoting are facilitated as much as possible.
There are a few minor improvements to Fish in 4.00 as well as the paths change. It’s gained exit and quit commands to leave the interactive shell (though ^D (unix) and ^Z (Windows) still work). And it’s gained a get command, which is like show, but more minimal, returning the information you request with no extras. Compare and contrast:
$ fish show -a Paris rating /about
Object with about="Paris":
  /njr/rating = 10
  /fluiddb/about = "Paris"

$ fish get -a Paris rating /about
10
"Paris"
The get form is motivated partly, again by, a desire to make it as simple as possible for Fish (and other things) to consume the output.
There’s also better mobile support in the online version, Shell-Fish, but that’s another blog post (coming shortly).

26 July 2011

Movement and Renaming In Fluidinfo and Fish

The Fluidinfo Shell, Fish, already includes Unix-inspired commands for many tag-management tasks including listing tags and namespaces (ls), removing them (rm), creating them (touch and mkns/mkdir) as well an analogue of the Unix chmod permissions-changing command (perms). What about copying and renaming?

The Unix command for renaming files is mv (short for move). I imagine this is briefly confusing for newcomers to Unix, but the command is actually well named when all its functions are considered. In Unix, to rename a file from foo to bar one says:

mv foo bar

The same form can be used to rename a directory, say from private to secret:

mv private secret

The true appropriateness of the name mv becomes apparent only when the command is used for it third form, in which the destination is an existing directory. If etc is such an existing directory then

mv foo etc

will move foo into the directory etc; or, if you prefer, it renames foo as etc/foo. The same thing happens with a directory, i.e.

mv private etc

will move the directory private into etc as etc/private. We can also move several files at the same time:

mv foo private etc

will move both foo and private to etc., which is handy.

Should Fish do something similar?

Moving Tags and Values in Fish

There seems to be every reason to support a similar mv command in Fish. As with files, we may wish to rename a tag or namespace, or to move it within the tag hierarchy, and Unix provides a well-thought-through, powerful and convenient template for this. Tags would behave like files, and namespaces like directories. But there is a complication, as we shall see.

In Fluidinfo, a named tag, such as njr/rating is an abstract entity: it doesn’t really store any significant information itself. The tag may be attached to many objects, with different values or none on each. We can think of those tags-attached-to-objects as concrete tags or tag instances.

So if Fish implemented an mv command as above, then if authenticated as user njr (and using Unix-style paths in Fish), the command

mv rating star-rating

would change the name of the tag njr/rating to njr/star-rating not just in one place, but everywhere it occurs in the system. Similarly, if njr/private is a namespace,

mv private secret

would rename not only the tags in njr/private, but on every object to which those tags are attached.

This is not so much a problem as an opportunity. There is definitely a need for an easy way to rename and move whole named abstract tags and namespaces, but it would also be useful to be able to move and rename individual concrete tags. For example, maybe I want to rename my njr/rating tag on Citizen Kane from njr/rating to njr/star-rating (preserving its value). (After all, film ratings are all about stars, right?)

The obvious thing to do there would be to add a -a to specify the about tag for the object:

mv -a 'film:citizen kane (1941)' rating star-rating

It could also accept -i, to specify by ID, or -q to choose a set of objects for which to apply the renaming with a Fluidinfo query.

So far, so good. But what if it is a different kind of movement that I desire? What if I want to move the tag to a different object? Perhaps I’ve used the wrong object, or wish to swap to a different convention for about tags. As an example, let’s suppose I missed out the year required in the film-u convention when constructing the about tag for the film Citizen Kane. How would I move the tag from one object to another?

Perhaps the most obvious approach would be to repeat the -a flag to specify two different about tags. The library Fish uses for processing command-line options does allow them to be repeated, so the command could be

mv -a 'film:citizen kane' -a 'film:citizen kane (1941)' rating

Notice that there tag appears only once, because here, it’s not being renamed. However, I really don’t like repeated tags; they are quite rare in Unix and feel sloppy to me. I’d be more tempted to use -b for the second one (from a to b; geddit?). So then it would be:

mv -a 'film:citizen kane' -b 'film:citizen kane (1941)' rating

You could also perform a renaming at the same time:

mv -a 'film:citizen kane' -b 'film:citizen kane (1941)' rating star-rating

which you might prefer to express as

mv -a 'film:citizen kane'/rating -b 'film:citizen kane (1941)' star-rating

We could do something similar with -i (-j for the second, I suppose); it doesn’t really apply in the case of -q since it would be optimistic in the extreme to expect pairs of about tags from two different queries to align correctly.

What I’ve described so far seems fairly workable, and is my current best guess at what I’ll try to implement. A cp command would be fairly similar, and would probably require a -R flag for recursion.

But there is an alternative that seems worth discussing, even if only to reject it.

The endpoint approach

I generally think of tags as being attached to objects, sometimes with values, pretty much exactly as the diagrams generated by http://abouttag.appspot.com show them. Here’s an example for Citizen Kane.

citizen-kane-object.png

(You can generate this live, in a modern browser, by visiting here).

And I think of the (abstract) tags and namespaces as living in a tree, exactly like the directory structure in a hierarchical file system. Something like this, where the yellow folder icons represent namespaces and the tags are red.

FluidinfoTagHierarchy.png

The analogy can even be extended further by thinking of the value on a tag as being like a file’s contents; this seems particularly apt in cases where the tag values really do correspond to file contents, as is the case with Fish’s documentation. Using a pair of tricks stolen from Nicholas Tollervery (@ntoll), the documentation is actually stored in Fluidinfo, with the content of the index.html file stored in a tag called index.html under the fish namespace. So the full path for the tag that stores the front page of Fish’s documentation is fish/index.html and this (concrete) tag is stored on the object with the about tag fish. The URL for the documentation is then http://fluiddb.fluidinfo.com/about/fish/fish/index.html. The second “trick” is to set the MIME type of the value of fish/index.html on the fish object to text/html, which causes Fluidinfo to serve the content with that MIME type, so that as far as the internet is concerned, http://fluiddb.fluidinfo.com/about/fish/fish/index.html is regular HTML web content. The same applies to all the other pages. It will come as no surprise to learn that the fish documentation is published programmatically, and that I have vague plans to add a publish command to Fish to allow a directory tree of files to be uploaded to Fluidinfo as a set of tags on namespaces on a nominated object.

It will not have escaped your attention in this digression, that a crucial difference between a file in a filesystem and a tag in Fluidinfo is that the tag may have arbitrarily many different values, each attached to a different object. It’s like having your file system partially replicated on every computer in the world, with different content on the subset of files stored on each.

This suggests a different way of thinking about the relationship between the tag hierarchy and objects, which is really the way that Fluidinfo’s end-points use them. In this view, each object has a partial copy of the (abstract) tag hierarchy instantiated on it, with values that are specific (and hopefully appropriate) to that object.

This is neatly illustrated by looking at a recursive listing of the fish user’s top-level namespace:

$ fish ls -R

fish:
_static/            h                   p                   tags.html
a                   help.html           perms.html          test.html
about.html          i                   pwd.html            touch.html
abouttag.html       index.html          pwn.html            u
amazon.html         install.html        q                   unixlike.html
b                   j                   r                   untag.html
c                   k                   rm.html             v
cli.html            l                   s                   version.html
commands.html       ls.html             search.html         w
count.html          m                   searchindex.js      whoami.html
d                   mkdir.html          shell-fish.html     x
e                   mkns.html           show.html           y
f                   n                   su.html             z
g                   normalize.html      t
genindex.html       o                   tag.html

fish/_static:
basic.css           default.css         fish-doc-logo.png   pygments.css
def.css             doctools.js         jquery.js           searchtools.js

All the tags that look like filenames (the ones containing a dot) are tags that exist in exactly one place—on the object with the about tag fish. (They could occur in multiple places, however. For example, I could publish the full set of documentation to version-numbered objects fish v3.12, fish v3.11 etc., rather than constantly replacing the old with the new, as I do now.)

The other tags (those with single-letter names) are there to allow anyone playing with the online version of Fish (shell-fish) to tag things with a set of different tags, and might exist on any set of objects in Fluidinfo.

In this view, the full path for a concrete tag can be expressed either in terms of its about tag (if it has one) or its ID. In the case of fish/index.html, the only concrete instantiation taht exists today lives under the /about endpoint, at

/about/fish/fish/index.html

or, for those of you who prefer UUIDs, under the /objects endpoint, at

/objects/0e7b94c7-1b37-41c2-bd7f-1a376786aa4e/fish/index.html

This suggests an alternative possible syntax for moving concrete tags. We could recast

mv -a 'film:citizen kane' /rating -b 'film:citizen kane (1941)' star-rating

as

mv '/about/film:citizen kane/njr/rating' '/about/film:citizen kane (1941)/star-rating'

This approach is not without its attractions. It is rather elegant and unambiguous, and it completely eliminates the need for flags in this case, which can hardly fail to be a benefit. On balance, however, I think it’s probably not the way to go. It feels somewhat inconsistent with thre rest of Fish, it forces me to put the username in (which I don’t really want to do) and—to me—it feels slightly inverted (if logical). There’s also a potential problem that there’s technically ambiguity with respect to a potential user called about, but that probably isn’t such a worry.

But the more important reason is that I dont want to have to repeat the object specification if I do something like

mv '/about/film:citizen kane (1941)/njr/rating' '/about/film:citizen kane (1941)/star-rating'

In that case, the -a tag feels much better, and that seems like a strong reason for with with either -a and -b or repeated -a flags when shifting between different objects.

So at the moment I’m leaning strongly towards the first suggestions introduced above, using -a and -b as required. As ever, however, I’m interested in opinions. I expect it will be a while before I get around to implementing this anyway.

21 July 2011

Command Substitution and the Interactive Fish Shell

 
The Fluidinfo Shell, Fish (documented here) has a gained a few important new features.   In summary, these are
  1. Interactive Shell.   If invoked without any parameters,
    Fish will start an interactive session that allows you to issue a sequences of Fish commands (without prefixing them by fish).
    $ fish
    This is fish version 3.12.
    > whoami
    alice
    > ls
    books/              favourite-things    private/            test-fish/
    comment             has-read            rating              to-read
    > show -a 'book:animal farm (george orwell)' alice/rating
    Object with about="book:animal farm (george orwell)":
      alice/rating = 2
    > show -a "`about book 'animal farm' 'george orwell'`" alice/rating
    Object with about="book:animal farm (george orwell)":
      alice/rating = 2
    > ^D
    $
    Points to note include:
    • If the readline library is available to python, it will be used, allowing Emacs key bindings to be used to edit the command line (as well as arrow keys) and saving a command history, which may be navigated with CTRL-P and CTRL-N or up and down arrows.  You can also view the history by typing history or h.
    • The interactive session is exited by typing the end-of-file
      character—CTRL-D on Unix and CTRL-Z on Windows.
    • Command substitution with left quoting (“backticks”) is supported
      as illustraed in the last example of the interactive session shown (explanation below).
    • Single and double quotes may be used interchangeably, and
      are protected by each other.   Quotes and other special characters may be escaped using backslash, which also escapes itself.   So, for example, Alice's Adventures in Wonderland may be passed as a parameter with
      "Alice's Adventures in Wonderland"
      
      or
      'Alice\'s Adventures in Wonderland'
      
    • Obviously, on Unix, when using the interactive shell, the
      Unix shell does not process the command line, meaning, among other things, that there is no globbing (wildcard expansion).
  2. Command Substitution.    Unix users will be familiar with
    command substitution, which now comes to Fish.   The way this works in Unix shells is that any part of a command enclosed in left quotes (“backticks”) is evaluated before the enclosing command, and its output replaces it in that enclosing command. This is particularly relevant to the commands that build Fluidinfo about tags—abouttag, amazon and the new normalize command (see below). In Unix shells, it has always been possible to say:
    $ fish show -a "`fish abouttag book 'animal farm' 'george orwell'`" alice/rating
    The shell first evaluates the subcommand, which produces the output book:animal farm (george orwell) and then, after substitution, the shell executes:
    $ fish show -a "book:animal farm (george orwell)" alice/rating
    In Unix, double quotes are weaker than single quotes, and command substitution can still occur inside them, which is exactly what we need in this case. Fish now makes almost identical functionality available in its new interactive mode.    In the interactive shell you can say
    > show -a "`about book 'animal farm' 'george orwell'`" alice/rating
    This command is identical except that the leading fish is no longer needed within the substitution command (though it can be used), just as it not needed in the outer show command.
  3. Command substitution in Shell-Fish.   The online version of
    Fish, Shell-Fish also benefits from the new command substution capabilities, but does not yet include the history feature.   Whether readline-like command line editing is available is a feature of the browser (today).
  4. The new normalize command.   A new normalize command has
    been added to Fish.   This simply performs some standardization on text, consisting primarily of mapping to lower case, removing most punctuation and standardizing whitespace; accents are preserved. If multiple arguments are passed, the components are joined with colons.   For example:
    $ fish normalize 'A *very* strange (and    wonderful) fish'
    a very strange and wonderful fish
    
    $ fish normalize golfer 'Darren Clarke'
    golfer:darren clarke
Happy fishing!

11 July 2011

Fish 3.07: More compact longer listings

I made a minor update to Fish, available from Github. This fixed a rather nasty bug that had broken many aspects of the perms command and which also modified the behaviour of the ls -L command (longer listing, including Fluidinfo-style permissions summary). The behaviour that was previously generated by ls -L is still available with ls -G, which is exemplified by
$ fish ls -G njr/rating

njr/rating:

ABSTRACT TAG (/tags)
  Write
    update (metadata):   policy: closed; exceptions = [njr, miro]
    delete (delete):     policy: closed; exceptions = [njr, miro]
  Control
    control (acontrol):  policy: closed; exceptions = [njr, miro]

TAG (/tag-values)
  Read
    read (read):         policy: open; exceptions = []
  Write
    create (tag):        policy: closed; exceptions = [njr, miro]
    delete (untag):      policy: closed; exceptions = [njr, miro]
  Control
    control (tcontrol):  policy: closed; exceptions = [njr, miro]
The new behaviour for ls -L will present a more compact summary in the large majority of cases where the four write permissions for tags are identical and the two control permissions are the same. This simplified view is:
$ fish ls -L njr/rating

njr/rating:
     read: policy: open; exceptions = []
    write: policy: closed; exceptions = [njr, miro]
  control: policy: closed; exceptions = [njr, miro]
The longer view is generated when the permissions cannot be shown accurately in this more compact form. Analogous behaviour occurs for namespaces when their three write permissions are identical. For example:
$ fish ls -Gn njr/fi

njr/fi/:

NAMESPACE (/namespaces)
  Read
    list (read):        policy: closed; exceptions = [njr, miro]
  Write
    create (create):    policy: closed; exceptions = [njr, miro]
    update (metadata):  policy: closed; exceptions = [njr, miro]
    delete (delete):    policy: closed; exceptions = [njr, miro]
  Control
    control (control):  policy: closed; exceptions = [njr]
reduces to:
$ fish ls -Ln njr/fi

njr/fi/:
     read: policy: closed; exceptions = [njr, miro]
    write: policy: closed; exceptions = [njr, miro]
  control: policy: closed; exceptions = [njr]
The very perceptive reader will also notice that the longest listing now includes the write update (metadata) permission that controls access to writing the namespace description; this was missing in previous releases as a result of a typographical error in the source. (Computers can be so picky.)

07 July 2011

About Tags In Fish

I’ve added a new command to fish (and updated the online version, Shell-Fish accordingly) to allow easy construction of standardized about tags using the conventions from the abouttag library. They make use of a new abouttag function, available in the new generic.py file in the abouttag library, which takes the object type as its first parameter, and the usual parameters as a variable parameter list.
The new fish command is abouttag, though can also be abbreviated to about and its general form is:
fish abouttag <object type> <object specifiers>
The object type is something like book, album or fi-user and the object specifiers are the key parameters used to describe that object, in the same order as they are used in the corresponding function from the abouttag library.
The easiest way to illustrate and define these is with examples. The following examples are taken from a Unix system; on Windows, use double quotes rather than single around parameters. In the online version (Shell-Fish), and on Unix, single or double quotes work. In the online version, you don’t need the fish prefix (though it does work).
I should note that part of the motivation for adding this functionality is a desire to allow the command to be used to specify objects without knowing the exact form of their about tags. In Unix-like systems (Linux, Mac OS X, Solaris etc.), this is possible by using left quotes, which can be placed inside double quotes. Thus, the following, slightly ungainly command (using all three forms of quote) works, at least in bash:
$ fish show -F -a "`fish abouttag book 'Gödel, Escher, Bach: An Eternal Golden Braid' 'Douglas R. Hofstader'`" njr/rating
Object with about="book:gödel escher bach an eternal golden braid (douglas r hofstader)":
  njr/rating = 10
I will leave it to the reader to judge whether this is easier than using cut and paste. For those who don’t know about left quotes in Unix shells, a command enclosed in left quotes within another command is evaluted before its enclosing command; its output replaces the left-quoted phrase on the original command line. So in the case above, we first run the command
fish abouttag book 'Gödel, Escher, Bach: An Eternal Golden Braid' 'Douglas R. Hofstader'
which generates
book:gödel escher bach an eternal golden braid (douglas r hofstader)
as its output. In effect, the outer command is then transformed to
fish show -F -a "book:gödel escher bach an eternal golden braid (douglas r hofstader)" njr/rating
I hope to extend shell-fish, the on-line version of fish, to support left quotes, but that may take a little while.
The following examples are taken from the fish documentation, which is available online from http://fluiddb.fluidinfo.com/about/fish/fish/index.html.
  1. Books and related items using the book-u convention (book, author)
    $ fish abouttag book 'Gödel, Escher, Bach: An Eternal Golden Braid' 'Douglas R. Hofstader'
    book:gödel escher bach an eternal golden braid (douglas r hofstader)
    
    $ fish abouttag book 'The Feynman Lectures on Physics' 'Richard P. Feynman' 'Robert B. Leighton' 'Matthew Sands'
    book:the feynman lectures on physics (richard p feynman; robert b leighton; matthew sands)
    
    $ fish abouttag book 'The Oxford English Dictionary: second edition, volume 3', 'John Simpson', 'Edmund Weiner'
    book:the oxford english dictionary second edition volume 3 (john simpson; edmund weiner)
    
    $ fish abouttag author 'Douglas R. Hofstadter' 1945 2  15
    author:douglas r hofstadter (1945-02-15)
  2. Music-related items (track, album, artist, isrc-recording)
    $ fish abouttag track 'Bamboulé' 'Bensusan and Malherbe'
    track:bamboulé (bensusan and malherbe)
    
    $ fish abouttag album 'Solilaï' 'Pierre Bensusan'
    album:solilaï (pierre bensusan)
    
    $ fish abouttag artist 'Crosby, Stills, Nash & Young'
    artist:crosby stills nash & young
    
    $ fish abouttag isrc-recording 'US-PR3-73-00012'
    isrc:USPR37300012
  3. URLs and URIs (URI, URL)
    $ fish abouttag uri FluidDB.fluidinfo.com
    http://fluiddb.fluidinfo.com
    
    $ fish abouttag url https://FluidDB.fluidinfo.com/one/two/
    https://fluiddb.fluidinfo.com/one/two
    
    $ fish abouttag URI http://fluiddb.fluidinfo.com/one/two/
    http://fluiddb.fluidinfo.com/one/two
    
    $ fish abouttag URL 'http://test.com/one/two/?referrer=http://a.b/c'
    http://test.com/one/two/?referrer=http://a.b/c
  4. Fluidinfo objects (fi-user, fi-namespace, fi-tag)
    $ fish abouttag fi-user njr
    Object for the user named njr
    
    $ fish abouttag fi-namespace njr/misc
    Object for the namespace njr/misc
    
    $ fish abouttag fi-ns njr/private
    Object for the namespace njr/private
    
    $ fish abouttag fi-tag terrycojones/private/rating
    Object for the attribute terrycojones/private/rating
  5. Database components (db-table, db-field)
    $ fish abouttag db-table 'elements'
    table:elements
    
    $ fish abouttag db-field 'name' 'elements'
    field:name in table:elements
  6. Miscellaneous (planet, element)
    $ fish abouttag planet 'Mars'
    planet:Mars
    
    $ fish abouttag element 'Helium'
    element:Helium

Labels