12 March 2010

abouttag: A Package for Normalizing FluidDB about tags

The previous post discussed the small number of conventions already in use for about tags in FluidDB. It occurred to me that I could usefully write some code to create about tags following each of these conventions. I’ve just done this (in python) and it’s available from github.

In addition to simply producing the about tags, the package has some normalization capabilities.

I may blog about it more, but you get most of the idea from the following examples.

from abouttag.fluiddb import FluidDB
from abouttag.uri import URI
from abouttag.database import Database
from abouttag.objects import planet, element
from abouttag.location import GEOnet

db = FluidDB()
print db.user(u'njr')
print db.namespace(u'/njr/misc')
print db.tag(u'terrycojones/private/rating/')

print URI(u'FluidDB.fluidinfo.com')
print URI(u'FluidDB.fluidinfo.com/one/two/')
print URI(u'https://FluidDB.fluidinfo.com/one/two/')
print URI(u'http://fluiddb.fluidinfo.com/one/two/')
print URI(u'http://test.com/one/two/?referrer=http://a.b/c')

db = Database()
print db.table(u'elements ')
print db.field(u'Name', u'elements')

print planet(u'mars')

print element(u' helium')

print GEOnet(-2601490, -3577649)

which produces the following:

Object for the user named njr
Object for the namespace njr/misc
Object for the attribute terrycojones/private/rating
http://fluiddb.fluidinfo.com
http://fluiddb.fluidinfo.com/one/two
https://fluiddb.fluidinfo.com/one/two
http://fluiddb.fluidinfo.com/one/two
http://test.com/one/two/?referrer=http://a.b/c
table:elements
field:Name in table:elements
planet:Mars
element:Helium
GEOnet-2601490_-3577649

These examples all have normalization turned on, though it can be turned off by adding normalization=False to any of the relevant function calls.

This will become more significant in some other cases where the key for the about tag may require more transformation from the raw input. (For example, I’ve been reading about NACO normalization, which is used by librarians to increase the probability of matching titles and authors successfully; I’m thinking of devising a schema for about tags for books based on this.)

11 March 2010

About Tag Conventions in FluidDB

This page documents the conventions for about tags in FluidDB that I am aware of. I will update it periodically when I learn of more conventions or discover errors.

For want of a better alternative, I define each about tag format using a python string format string and as many parameters as are required. I then give real examples, illustrated with fdb (Not that, unlike FluidDB itself, fdb uses leading slashes to introduce tags and interprets /about as /fluiddb/about and /id as an object’s UUID.)

UPDATE 2010/03/12: Rather than just showing the python string format here, I realised it would be useful to write some code to produce about tags following the conventions below. I have just committed a python library called abouttag at github. In addition to performing the rather trivial formatting described below this also does a degree of (optional) normalization of string input, to further reduce the likelihood of unintentional differences. I will gradualy add examples below using the library and will also do a separate post about it.

UPDATE 2010/12/29: I’ve also added in links to some visualization software available at abouttag.com, which shows objects in FluidDB and the tags on them graphically. This software should work with any modern, standards-compiant web brower, which in practice means almost anything except Internet Explorer.

1. FluidDB (fluiddb-1)

The following conventions are used by FluidDB itself when it creates the objects that form the very basis of FluidDB.

1.1. Users

The about tag for the user with name username is:

"Object for the user named %s" % username

$ fdb show -i 8cb0e83b-fa48-490c-a95f-958053db598c /about
Object 8cb0e83b-fa48-490c-a95f-958053db598c:
/fluiddb/about = "Object for the user named njr"

Graphical Example: Object for the user named njr.

1.2. Namespaces

The about tag for the namespace whose name is ns is:

"Object for the namespace %s" % ns
$ fdb show -i fb2e7545-1dd4-4432-8cfc-2fd24db7384b /about
Object fb2e7545-1dd4-4432-8cfc-2fd24db7384b:
/fluiddb/about = "Object for the namespace njr"

$ fdb show -i 9b4846ad-6639-46c2-adb1-e0d87649413d /about
Object 9b4846ad-6639-46c2-adb1-e0d87649413d:
/fluiddb/about = "Object for the namespace miro/elements"

Graphical example: Object for the namespace miro/elements

1.3. Tags

The about tag for the tag tag is:

"Object for the attribute %s" % tag

$ fdb show -i 4f6550a5-7eed-4f45-8046-56f0ac6c4f43 /about
Object 4f6550a5-7eed-4f45-8046-56f0ac6c4f43:
  /fluiddb/about = "Object for the attribute njr/rating"

$ fdb show -a "Object for the attribute fluiddb/about" /about /id
Object with about="Object for the attribute fluiddb/about":
  /fluiddb/about = "Object for the attribute fluiddb/about"
  /id = "873e0311-07d2-42bd-b67f-739d731332da"

Graphical example: Object for the attribute fluiddb/about

Find about tags for the objects FluidDB uses itself here.

2. URLs and URIs (uri-1)

Perhaps the most obvious and common entities for which objects are created in FluidDB are URLs and more generally URIs. So far, probably with barely a conscious decision, almost everyone is using the obvious convention:

URLs and URIs

The about tag for a URL or URI is itself in standard form, for example

Here, standard form most often seems to mean lower case for the domain name, no trailing slash, and inclusion of the protocol specifier (http://) etc. So:

http://fluidinfo.com/fluiddb
not fluidinfo.com/fluiddb
not http://fluidinfo.com/fluiddb/

For example, there are a couple of ways of importing bookmarks from del.icio.us to FluidDB (here and here), both of which use the standardized URL of the web page as the object on which to place imported tags and other metadata.

fdb show -a 'http://www.google.com' /about /id
Object with about="http://www.google.com":
  /fluiddb/about = "http://www.google.com"
  /id = "e75abb22-2f3c-45dd-9429-fce0a78f82f2"

Graphical example: http://www.google.com.

Find about tags for URLs and URIs here.

3. Tables (database-1)

Miró is a data analysis package produced by Stochastic Solutions. Miró has direct integration with FluidDB including the ability to publish a dataset (in the form of a table) straight to FluidDB.

Miró is in the process of defining conventions for about tags for the objects it used to represent tables.

There is a post describing the approach to tables available here.

3.1. Tables

The about tag for a table called tablename in is:

"table:%s" % tablename

Object with about="table:elements":
  /fluiddb/about = "table:elements"
  /id = "b898b875-89ad-4b69-bf89-b6a605b79697"

3.1.1. Fields

The about tag for a field named fieldname in a table tablename is:

"field:%s in table:%s" % (fieldname, tablename)

$ fdb show -q 'miro/field-name = "RelativeAtomicMass"' /about
1 object matched
Object 261a1ce8-4a42-45dc-b4be-4206d95b0bbe:
  /fluiddb/about = "field:RelativeAtomicMass in table:elements"

3.1.2. Records

There is no fixed recommendation for the about tag for a database record. Usually, if the record pertains to something, it will be appropriate to store the record data on an on object naturally relating to that thing.

Miró intends always to use about tags of the general for entity:description, where entity is a category of entity, and a description is a (standard-form) description of that entity.

At the time of writing, two tables have been published to FluidDB by Miró, pertaining to elements of the periodic table and to planets (and dwarf planets) in the solar system.

4. Elements of the Periodic Table (element-1)

Miró has published a table of all of the elements of the periodic table to FluidbDB usin the table conventions above.

The about tag it uses for an named Element is:

"element:%s"' % Element

The elements can be identified by querying FluidDB for objects with (among others) the tag miro/elements/Name

$ fdb count -q "has miro/elements/Name"
118 objects matched
Total: 118 objects
$ fdb show -q 'miro/elements/Name = "Hydrogen"' /about
1 object matched
Object 270a8269-f02d-4925-b152-da3934edaa43:
  /fluiddb/about = "element:Hydrogen"

$ fdb show -q 'miro/elements/Z = 92' /about
1 object matched
Object adde1cbf-286b-4d0f-9c8c-57976a473321:
  /fluiddb/about = "element:Uranium"

Graphical example: element:Uranium.

Find about tags for elements here.

5. Planets (planet-1)

Miró has published a table containing the thirteen recognized planets and dwarf planets.

The about tag for an planet (or similar) named Planet is:

"planet:%s" % Planet

This dataset is described in detail at here.

$ fdb show -q 'miro/planets/Name = "Earth"' /about
1 object matched
Object 87551a22-5ef9-49e0-9d74-99f2d4af6776:
  /fluiddb/about = "planet:Earth"

$ fdb show -q 'miro/planets/Name = "Saturn"' /about
1 object matched
Object e06bea33-a000-4294-a7b2-d3245f1481ca:
  /fluiddb/about = "planet:Saturn"

Graphical example: planet:Saturn.

Find about tags for planets here.

6. Location (geonet-1)

Nicholas Tollervey (@ntoll) has been tagging places using the geo FluidDB user. He is using conventions from GEONet, which (among other things) defines a Unique Feature Identifier (UFI) and a Unique Name Identifier (UNI).

The about tag for a place with a given UNI and UFI Nicholas is using is:

"GEOnet%s_%s" % (UNI, UFI)

Objects corresponding to such places can be identified by having a geo/name tag. At the time of writing:

$ fdb count -q "has geo/name"
18950 objects matched
Total: 18950 objects

$ fdb show -q 'geo/name = "Fitful Head"' /about
1 object matched
Object 0a3dfbc6-e6e2-4d8b-8cbc-1c1920fa082b:
  /fluiddb/about = "GEOnet-2596258_-3571222"

Graphical example: GEOnet-2596258_-3571222.

Find about tags for locations here.

7.Twitter Users

The Tickery application stores information about twitter users on objects identified by their twitter ID. A twitter ID is a numeric identifier that remains the same even if twitter users change their twitter name. For example, the twitter ID for @terrycojones is 42983, so tickery stores information about this user on the about having the about tag twitter.com:uid:42983.

You can find the twitter ID for a user using idfromuser.com.

Find about tags for twitter users here

More?

If you are systematically using a convention for about tags in FluidDB, let me know. I plan to try to update this page as more conventions emerge.

09 March 2010

Order from Chaos: Tagging and Tabulating the Solar System in FluidDB

Planets-Tagged.png

One of Terry’s [1] mantras is that it’s easy to build structure on unstructured foundations, but very hard to build something unstructured from a structured base. Those familiar with the second law of thermodynamics or young children may question the second half of this, and those who have ever tried dry-stone walling or making something comprehensible from certain developers’ code may disagree with first half; but let’s try.

Our task today is to publish the following table, taken from the Wikipedia entry on Planets in our Solar System [2], in FluidDB.

_images/planets-wikipedia.png

Obviously, this is a reasonably structured textual table of data that contains a lot of useful information that we, as humans, can interpret.

Here it is in Miró, a data analysis package that my (other) company, Stochastic Solutions produces:

_images/planets-miro.png

All I did was extract the data from Wikipedia and “standardized” it a bit, getting rid of hyperlinks and things, replacing ranges with central values, separating out metadata and so forth. (The apparent loss of precision is illusory; Miró is only showing two decimal places.)

There are many possible ways to structure this data in FluidDB, but let’s start by showing you what I did, and then we’ll discuss the rationale for the choices I made.

I spent some of last weekend baking in FluidDB integration to Miró, with quite pleasing results. (I used both the my fdb library as well as Ali Afshar’s fom to do this: thanks for fom Ali: it certainly helped.) This is the log of what I did in Miró: as you can see, the upload took about three and a half minutes, which is obviously longer than we’d like; but that will improve.

$ time miro -f scripts/planets-publish
This is Miro, version 0.3.59.
Copyright (c) Stochastic Solutions 2008-2010.
Seed: 1268155418
Logs started at 2010/03/09 17:23:38 host zero.
Logging to /miro/log/2010/03/09/session100.

[1]> load planets
planets.miro: 13 records; 13 (100%) selected.
[2]> def about (+ 'planet:' Name)
Defining field about (+ planet: Name) . . . done.
Field "about" (string) planet:Ceres planet:Venus nulls = 0
[3]> fdb publish 13
Creating FluidDB table with name "planets"

Creating field objects for table "planets"
11 10 9 8 7 6 5 4 3 2 1 0
0

Creating record objects for table "planets"
12 11 10 9 8 7 6 5 4 3 2 1 0
0

Completing table entry for Miro table planets.
Table planets complete; id=5ff094b0-72be-4786-82db-078e9fa38623.

5ff094b0-72be-4786-82db-078e9fa38623

Job completed after a total of 209.3469 seconds
Logs closed at 2010/03/09 17:27:07 host zero.
Logs written to /miro/log/2010/03/09/session100.

real        3m29.640s
user        0m0.596s
sys 0m0.364s

So what did it do? We can use the fdb command-line utility, available from github, to find out. (Remember, in what follows, that—unlike fluiddb itself—fdb uses a leading slash to indicate full paths; I could have omitted all the /miro/ prefixes here and got the same result, but included them for clarity.)

The command below asks fdb to show the various nominated attributes about the FluidDB object whose about tag is table:planets.

$ fdb show -q 'fluiddb/about = "table:planets"' \
    /about \
    /miro/class \
    /miro/n-fields \
    /miro/n-records \
    /miro/first-record-about \
    /miro/first-field-about \
    /miro/has-about-links \
    /miro/has-id-links \
    /miro/has-record-numbers \
    /miro/has-field-numbers \
    /miro/consistent
Object 5ff094b0-72be-4786-82db-078e9fa38623:
  /fluiddb/about = "table:planets"
  /miro/class = "table"
  /miro/n-fields = 12
  /miro/n-records = 13
  /miro/first-record-about = "planet:Mercury"
  /miro/first-field-about = "field:Name in table:planets"
  /miro/has-about-links = True
  /miro/has-id-links = False
  /miro/has-record-numbers = True
  /miro/has-field-numbers = True
  /miro/consistent = True

Note that the object ID returned (5ff094b0-72be-4786-82db-078e9fa38623) matches the one reported as the result of Miró’s publish operation. (We could have got the same result by saying

$ fdb show -i 5ff094b0-72be-4786-82db-078e9fa38623 ...

Obviously all that’s here is some metadata. Key points are:

  • The top-level table metadata is stored in the object whose about tag is table:planets. Here planets is the name of the dataset.

  • n-fields and n-records obviously store the number of fields and records in the dataset, respectively.

  • The next two entries are used for navigation and are pointers to linked lists—one for the fields and one for the records. In this case the linking is done on about tags, so the first record is stored in the object about planet:Mercury, and metadata about the first field is stored in the object whose about tag is field:Name in table:planets. (It’s worth noting that another application or user wanting to store a field called Name in a table called planets, and using the same conventions, could and absolutely should use the same object: since that user or application will inevitably have a different namespace, there will be no conflict but the collecting together if information on the same object will be useful.)

    Note that the prefix planet is not the same as the table name (planets) and no convention is being proposed about any relation between table names and about prefixes. Indeed, there is no requirement that there should even be an about tag on the records, though (as discussed below) my very strong preference is that in most cases there should be. Obviously, about-tag linking can only be used if there is an about field on each record.

  • The next four items all describe what extra metadata is stored in the table (if any). I’ll discuss these later.

  • Finally, the consistent tag is a boolean that indicates whether the table is in a consistent state or not. When Miró created the table object (about="table:planets") the very next thing it did was to set consistent=False to indicate that the data is not in a consistent state. The last operation Miró performs in publishing the table is to set it to True.

So much for the table metadata. What about the data? Again, we can use fdb (or anything else that can talk to FluidDB) to look at it. Here it all is:

$ time fdb show -q "has miro/planets/Name"  \
     /about \
     /miro/planets/Name \
     /miro/planets/Category \
     /miro/planets/EquatorialDiameter \
     /miro/planets/Mass \
     /miro/planets/OrbitalRadius \
     /miro/planets/OrbitalPeriod \
     /miro/planets/Inclination \
     /miro/planets/OrbitalEccentricity \
     /miro/planets/RotationPeriod \
     /miro/planets/Moons \
     /miro/planets/HasRings \
     /miro/planets/Atmosphere \
     /miro/planets/db-record-number \
     /miro/planets/db-next-record-about
13 objects matched
Object 2994f561-8efe-4e13-9374-bf3f9436eac6:
  /fluiddb/about = "planet:Jupiter"
  /miro/planets/Name = "Jupiter"
  /miro/planets/Category = "gas giant"
  /miro/planets/EquatorialDiameter = 11.209
  /miro/planets/Mass = 317.8
  /miro/planets/OrbitalRadius = 5.2
  /miro/planets/OrbitalPeriod = 11.86
  /miro/planets/Inclination = 6.09
  /miro/planets/OrbitalEccentricity = 0.048
  /miro/planets/RotationPeriod = 0.41
  /miro/planets/Moons = 49
  /miro/planets/HasRings = True
  /miro/planets/Atmosphere = "hydrogen and helium"
  /miro/planets/db-record-number = 5
  /miro/planets/db-next-record-about = "planet:Saturn"
Object c74807ca-8090-4f0b-a45c-404b3e31d6b0:
  /fluiddb/about = "planet:Mercury"
  /miro/planets/Name = "Mercury"
  /miro/planets/Category = "terrestrial"
  /miro/planets/EquatorialDiameter = 0.382
  /miro/planets/Mass = 0.06
  /miro/planets/OrbitalRadius = 0.39
  /miro/planets/OrbitalPeriod = 0.24
  /miro/planets/Inclination = 3.38
  /miro/planets/OrbitalEccentricity = 0.206
  /miro/planets/RotationPeriod = 58.64
  /miro/planets/Moons = 0
  /miro/planets/HasRings = False
  /miro/planets/Atmosphere = "minimal"
  /miro/planets/db-record-number = 1
  /miro/planets/db-next-record-about = "planet:Venus"
Object 6c590ab3-fa20-486e-a6ce-026b0e1d3a8c:
  /fluiddb/about = "planet:Eris"
  /miro/planets/Name = "Eris"
  /miro/planets/Category = "dwarf planet"
  /miro/planets/EquatorialDiameter = 0.19
  /miro/planets/Mass = 0.0025
  /miro/planets/OrbitalRadius = 67.7
  /miro/planets/OrbitalPeriod = 557.0
  /miro/planets/Inclination = 44.19
  /miro/planets/OrbitalEccentricity = 0.442
  /miro/planets/RotationPeriod = 0.3
  /miro/planets/Moons = 1
  /miro/planets/HasRings
  /miro/planets/Atmosphere
  /miro/planets/db-record-number = 13
  /miro/planets/db-next-record-about
Object bc910875-ef5c-4b6a-b75f-bb80eff7a4ca:
  /fluiddb/about = "planet:Mars"
  /miro/planets/Name = "Mars"
  /miro/planets/Category = "terrestrial"
  /miro/planets/EquatorialDiameter = 0.532
  /miro/planets/Mass = 0.11
  /miro/planets/OrbitalRadius = 1.52
  /miro/planets/OrbitalPeriod = 1.88
  /miro/planets/Inclination = 5.65
  /miro/planets/OrbitalEccentricity = 0.093
  /miro/planets/RotationPeriod = 1.03
  /miro/planets/Moons = 2
  /miro/planets/HasRings = False
  /miro/planets/Atmosphere = "carbon dioxide and nitrogen"
  /miro/planets/db-record-number = 4
  /miro/planets/db-next-record-about = "planet:Jupiter"
Object 88969e67-4834-477d-8cc8-f819ffbad6dc:
  /fluiddb/about = "planet:Pluto"
  /miro/planets/Name = "Pluto"
  /miro/planets/Category = "dwarf planet"
  /miro/planets/EquatorialDiameter = 0.19
  /miro/planets/Mass = 0.0022
  /miro/planets/OrbitalRadius = 40.0
  /miro/planets/OrbitalPeriod = 248.09
  /miro/planets/Inclination = 17.14
  /miro/planets/OrbitalEccentricity = 0.249
  /miro/planets/RotationPeriod = -6.39
  /miro/planets/Moons = 3
  /miro/planets/HasRings = False
  /miro/planets/Atmosphere = "temporary"
  /miro/planets/db-record-number = 10
  /miro/planets/db-next-record-about = "planet:Haumea"
Object 5af2d080-9098-4e9c-9f60-3b85eb06f6d5:
  /fluiddb/about = "planet:Haumea"
  /miro/planets/Name = "Haumea"
  /miro/planets/Category = "dwarf planet"
  /miro/planets/EquatorialDiameter = 3.7e-16
  /miro/planets/Mass = 0.0007
  /miro/planets/OrbitalRadius = 43.35
  /miro/planets/OrbitalPeriod = 282.76
  /miro/planets/Inclination = 28.19
  /miro/planets/OrbitalEccentricity = 0.189
  /miro/planets/RotationPeriod = 0.16
  /miro/planets/Moons = 2
  /miro/planets/HasRings
  /miro/planets/Atmosphere
  /miro/planets/db-record-number = 11
  /miro/planets/db-next-record-about = "planet:Makemake"
Object 72144788-a59e-4819-a9c9-6b8577e2695b:
  /fluiddb/about = "planet:Neptune"
  /miro/planets/Name = "Neptune"
  /miro/planets/Category = "gas giant"
  /miro/planets/EquatorialDiameter = 3.883
  /miro/planets/Mass = 17.2
  /miro/planets/OrbitalRadius = 30.06
  /miro/planets/OrbitalPeriod = 164.8
  /miro/planets/Inclination = 6.43
  /miro/planets/OrbitalEccentricity = 0.009
  /miro/planets/RotationPeriod = 0.67
  /miro/planets/Moons = 13
  /miro/planets/HasRings = True
  /miro/planets/Atmosphere = "hydrogen and helium"
  /miro/planets/db-record-number = 8
  /miro/planets/db-next-record-about = "planet:Ceres"
Object e9b022e6-c770-44ad-abaa-1a2cde9a3224:
  /fluiddb/about = "planet:Uranus"
  /miro/planets/Name = "Uranus"
  /miro/planets/Category = "gas giant"
  /miro/planets/EquatorialDiameter = 4.007
  /miro/planets/Mass = 14.6
  /miro/planets/OrbitalRadius = 19.22
  /miro/planets/OrbitalPeriod = 84.01
  /miro/planets/Inclination = 6.48
  /miro/planets/OrbitalEccentricity = 0.047
  /miro/planets/RotationPeriod = -0.72
  /miro/planets/Moons = 27
  /miro/planets/HasRings = True
  /miro/planets/Atmosphere = "hydrogen and helium"
  /miro/planets/db-record-number = 7
  /miro/planets/db-next-record-about = "planet:Neptune"
Object 818d8f3f-f3a1-4b10-9dc8-11514e468283:
  /fluiddb/about = "planet:Makemake"
  /miro/planets/Name = "Makemake"
  /miro/planets/Category = "dwarf planet"
  /miro/planets/EquatorialDiameter = 0.12
  /miro/planets/Mass = 0.0007
  /miro/planets/OrbitalRadius = 45.8
  /miro/planets/OrbitalPeriod = 309.88
  /miro/planets/Inclination = 28.96
  /miro/planets/OrbitalEccentricity = 0.159
  /miro/planets/RotationPeriod
  /miro/planets/Moons = 0
  /miro/planets/HasRings
  /miro/planets/Atmosphere
  /miro/planets/db-record-number = 12
  /miro/planets/db-next-record-about = "planet:Eris"
Object e06bea33-a000-4294-a7b2-d3245f1481ca:
  /fluiddb/about = "planet:Saturn"
  /miro/planets/Name = "Saturn"
  /miro/planets/Category = "gas giant"
  /miro/planets/EquatorialDiameter = 9.449
  /miro/planets/Mass = 95.2
  /miro/planets/OrbitalRadius = 9.54
  /miro/planets/OrbitalPeriod = 29.46
  /miro/planets/Inclination = 5.51
  /miro/planets/OrbitalEccentricity = 0.054
  /miro/planets/RotationPeriod = 0.43
  /miro/planets/Moons = 52
  /miro/planets/HasRings = True
  /miro/planets/Atmosphere = "hydrogen and helium"
  /miro/planets/db-record-number = 6
  /miro/planets/db-next-record-about = "planet:Uranus"
Object c81ab53a-9f69-4d24-bc2e-98f08298298f:
  /fluiddb/about = "planet:Ceres"
  /miro/planets/Name = "Ceres"
  /miro/planets/Category = "dwarf planet"
  /miro/planets/EquatorialDiameter = 0.08
  /miro/planets/Mass = 0.0002
  /miro/planets/OrbitalRadius = 2.75
  /miro/planets/OrbitalPeriod = 4.6
  /miro/planets/Inclination = 10.59
  /miro/planets/OrbitalEccentricity = 0.08
  /miro/planets/RotationPeriod = 0.38
  /miro/planets/Moons = 0
  /miro/planets/HasRings = False
  /miro/planets/Atmosphere = "none"
  /miro/planets/db-record-number = 9
  /miro/planets/db-next-record-about = "planet:Pluto"
Object 6d75ee02-c64e-45c6-b167-49083ab3739a:
  /fluiddb/about = "planet:Venus"
  /miro/planets/Name = "Venus"
  /miro/planets/Category = "terrestrial"
  /miro/planets/EquatorialDiameter = 0.949
  /miro/planets/Mass = 0.82
  /miro/planets/OrbitalRadius = 0.72
  /miro/planets/OrbitalPeriod = 0.62
  /miro/planets/Inclination = 3.86
  /miro/planets/OrbitalEccentricity = 0.007
  /miro/planets/RotationPeriod = -243.02
  /miro/planets/Moons = 0
  /miro/planets/HasRings = False
  /miro/planets/Atmosphere = "carbon dioxide and nitrogen"
  /miro/planets/db-record-number = 2
  /miro/planets/db-next-record-about = "planet:Earth"
Object 87551a22-5ef9-49e0-9d74-99f2d4af6776:
  /fluiddb/about = "planet:Earth"
  /miro/planets/Name = "Earth"
  /miro/planets/Category = "terrestrial"
  /miro/planets/EquatorialDiameter = 1.0
  /miro/planets/Mass = 1.0
  /miro/planets/OrbitalRadius = 1.0
  /miro/planets/OrbitalPeriod = 1.0
  /miro/planets/Inclination = 7.25
  /miro/planets/OrbitalEccentricity = 0.017
  /miro/planets/RotationPeriod = 1.0
  /miro/planets/Moons = 1
  /miro/planets/HasRings = False
  /miro/planets/Atmosphere = "nitrogen and oxygen"
  /miro/planets/db-record-number = 3
  /miro/planets/db-next-record-about = "planet:Mars"

real        1m8.985s
user        0m0.313s
sys 0m0.346s

OK, so that’s too much data for a blog post, but since it’s a first I wanted to include it all.

There are two points to note here.

  • The convention I’m using (and proposing) is that miro (the FluidDB user) uses a separate namespace for each of its tables, and that the name of the namespace is the same as the name of the table, in this case planets. That namespace includes a tag for each field, and the name of the tag is the name of the field, i.e. the field Moons in the table planets is stored by Miró as values of the tag miro/planets/Moons.

  • In this case, there are two extra tags in the same namespace, both containing structure/navigation information. These tags have deliberately been chose to include hyphens and a db- prefix to try to minimize the likelihood of clashes with actual field names. As you have guessed, db-record-number is the record number (starting from 1), and db-next-record-about is a link to the about field of the next record in sequence. (The last record has this set to null.)

    Clearly this is a deliberately redundant representation.

So we’ve seen the overall table metadata, and we’ve seen the data. There’s just one more thing, and that’s the field metadata. Again, we can use fdb to look at this:

$ fdb show -q 'miro/class = "field" and miro/table-name = "planets"' /about /miro/field-name /miro/type /miro/class /miro/table-name /miro/description /miro/next-field-about /miro/field-number12 objects matchedObject 27341750-121c-417d-a229-ed9112e4e182:  /fluiddb/about = "field:Name in table:planets"
  /miro/field-name = "Name"
  /miro/type = "string"
  /miro/class = "field"
  /miro/table-name = "planets"
  /miro/description = "Name of planet"
  /miro/next-field-about = "field:Category in table:planets"
  /miro/field-number = 1
Object 84e64f2a-ce44-4e72-8fab-fade19250034:
  /fluiddb/about = "field:EquatorialDiameter in table:planets"
  /miro/field-name = "EquatorialDiameter"
  /miro/type = "float"
  /miro/class = "field"
  /miro/table-name = "planets"
  /miro/description = "Diameter at the equator in units of the Earth's equatorial diameter, which is about 12,756.2km, or about 7,926.34 miles."
  /miro/next-field-about = "field:Mass in table:planets"
  /miro/field-number = 3
Object c153ce50-2216-49aa-8238-41e910e3ac4a:
  /fluiddb/about = "field:Category in table:planets"
  /miro/field-name = "Category"
  /miro/type = "string"
  /miro/class = "field"
  /miro/table-name = "planets"
  /miro/description = "Kind of planet. One of terrestrial, gas giant or dwarf planet"
  /miro/next-field-about = "field:EquatorialDiameter in table:planets"
  /miro/field-number = 2
Object 060881dc-5f4f-481a-9f12-699ac7d50dbd:
  /fluiddb/about = "field:OrbitalRadius in table:planets"
  /miro/field-name = "OrbitalRadius"
  /miro/type = "float"
  /miro/class = "field"
  /miro/table-name = "planets"
  /miro/description = "Orbital radius in astronomical units.   The Earth's mean distance from the sun is defined as 1AU, and is about 150 million km or 93 million miles. In the case of the dwarf planets, ranges have been converted to approximate central values."
  /miro/next-field-about = "field:OrbitalPeriod in table:planets"
  /miro/field-number = 5
Object fc8fc84f-5ab0-4e2b-942c-f653de08ac74:
  /fluiddb/about = "field:OrbitalEccentricity in table:planets"
  /miro/field-name = "OrbitalEccentricity"
  /miro/type = "float"
  /miro/class = "field"
  /miro/table-name = "planets"
  /miro/description = "The eccentricity of the planet's orbit."
  /miro/next-field-about = "field:RotationPeriod in table:planets"
  /miro/field-number = 8
Object 7ae04a54-e104-4a20-88d1-22bcfeacfdb9:
  /fluiddb/about = "field:Inclination in table:planets"
  /miro/field-name = "Inclination"
  /miro/type = "float"
  /miro/class = "field"
  /miro/table-name = "planets"
  /miro/description = "Incliniation to the Sun's equator in degrees."
  /miro/next-field-about = "field:OrbitalEccentricity in table:planets"
  /miro/field-number = 7
Object 5081c010-d99f-4e95-97ae-1bedc5b22a97:
  /fluiddb/about = "field:HasRings in table:planets"
  /miro/field-name = "HasRings"
  /miro/type = "bool"
  /miro/class = "field"
  /miro/table-name = "planets"
  /miro/description = "True if the planet has rings."
  /miro/next-field-about = "field:Atmosphere in table:planets"
  /miro/field-number = 11
Object 1a653242-7049-468d-816d-e41d88deb491:
  /fluiddb/about = "field:OrbitalPeriod in table:planets"
  /miro/field-name = "OrbitalPeriod"
  /miro/type = "float"
  /miro/class = "field"
  /miro/table-name = "planets"
  /miro/description = "Orbital period in (Earth) years."
  /miro/next-field-about = "field:Inclination in table:planets"
  /miro/field-number = 6
Object c4fae320-e9eb-4905-8c8e-9857dafecc34:
  /fluiddb/about = "field:RotationPeriod in table:planets"
  /miro/field-name = "RotationPeriod"
  /miro/type = "float"
  /miro/class = "field"
  /miro/table-name = "planets"
  /miro/description = "The period of rotation in (Earth) days."
  /miro/next-field-about = "field:Moons in table:planets"
  /miro/field-number = 9
Object 5a665b6e-ea63-41a6-9f27-ff1ed4960288:
  /fluiddb/about = "field:Moons in table:planets"
  /miro/field-name = "Moons"
  /miro/type = "int"
  /miro/class = "field"
  /miro/table-name = "planets"
  /miro/description = "Number of (named) moons."
  /miro/next-field-about = "field:HasRings in table:planets"
  /miro/field-number = 10
Object f35e65bc-a595-48d4-9da7-3ccac02b0893:
  /fluiddb/about = "field:Mass in table:planets"
  /miro/field-name = "Mass"
  /miro/type = "float"
  /miro/class = "field"
  /miro/table-name = "planets"
  /miro/description = "Mass of planet in units of Earth Mass.   The Earth's mass is roughly 5.9736 x 10^24 kg."
  /miro/next-field-about = "field:OrbitalRadius in table:planets"
  /miro/field-number = 4
Object baf7d11c-4821-4862-8af1-21c69b141dca:
  /fluiddb/about = "field:Atmosphere in table:planets"
  /miro/field-name = "Atmosphere"
  /miro/type = "string"
  /miro/class = "field"
  /miro/table-name = "planets"
  /miro/description = "Atmospheric composition (main constituents)."
  /miro/next-field-about
  /miro/field-number = 12

So:

  • The fields have names (tagged field-name) and those names (as we saw above) are also used as tags in a namespace whose name is the table name (e.g. miro/planets/Moons).

  • The fields also have types (tagged type). Miró is currently supporting five types for fields, four of which are base FluidDB types

    • bool
    • int
    • float
    • string

    The fifth supported type (not included in this table) is

    • revdate

    which is a string exemplified by 2010/03/09 22:22:22. There is no timezone information and the format must be exactly as shown (i.e. %Y/%m/%d %H:%M:%S). No doubt others will be needed.

    NOTE:

    • In FluidDB values have types but tags do not: different objects may be tagged with values of different types for the same tag. The schema being used here prohibits that: in effect Miró is making a guarantee that FluidDB does not.
    • Nulls are allowed in all fields. While in FluidDB, null is considered to be a type, in the schema used here, null is simply the absence of a value; and all values can be missing.
  • There is a separate object for each field, and the object has an about tag of the form "field:Name of table:tablename".

  • Fields can optionally have a description. In this case, I’ve includeded various information in that including units; I will probably add a separate units tag later.

Discussion

There’s much that could be discussed about this schema, but this has been a long post so I’ll restrict myself to a few words now.

Tag names:
In general, I’ve tried to choose tag names that are relatively neutral. I’ve used hyphens as separators, which some people won’t like, but this is a detail. More difficult was the decision about whether to include a prefix on db-next-record-about, db-next-record-id, and db-record-number. The reason I did is that these pollute a space that really should be for arbitrary field names. There were alternatives, such as using a subname space (miro/planets/navigation/record-number etc.), but that seemed like overkill. It’s unpleasant that these two have the db- prefix where no others do (not even next-field-about), and some will see this is inconsistent; it is not, however, capricious: there is no possible conflict in the other cases, so I’ve sought to restrict unnaturalness to the only three tags in which is serves a purpose.
Navigation schemes:
Whether any navigation information is needed at all depends on whether you take the view that the order of rows and columns is significant or not. In practice, there is a very strong convention that tables have records and columns in particular orders, and in rare cases information is actually contained in the order that is not stored in any other way; so I at least wanted to support this. Including the record and field numbers is probably the simplest scheme. Miró also supports adding forward links based on either or both of the about field (if they are used) or the object IDs. Clearly, linked lists are particularly good if you want to insert records at arbitrary places (and the record numbers become a problem in that case), but in general they are probably harder to work with. Miró currently defaults to including record and field numbers and also linked-lists based on about tags, as shown here, but I consider all to be optional and non-exclusive.
About Tags vs. UUIDs
Objects have two unique and immutable attributes in FluidDB—their IDs (UUIDs), which every object has, and their about tags, where used. If the goal, as it surely is, is to have different users and applications join information by tagging the same objects, it is desirable to tie those objects as firmly as possible to the things that they represent. Clearly, there are no established conventions for about tags yet (though perhaps this post constitues a baby step towards one small set of conventions in the realm of structured data), but over time I am confident some will emerge.

The Periodic Table of the Elements

For anyone interested, I’ve also published the elements from the periodic table using identical schema conventions. I’ll blog about that in a separate post, but if you want to look, the key information is:

$ fdb show -a "table:elements" /id
Object with about="table:elements":
  /id = "b898b875-89ad-4b69-bf89-b6a605b79697"

$ fdb count -q 'miro/class = "record" and has miro/elements/Z' /about
118 objects matched
Total: 118 objects

[1]That’s Terry Jones, @terrycojones, the founder of Fluidinfo, and the most fluid of all of us.
[2]http://en.wikipedia.org/wiki/Planets#Solar_System as at 7th March 2010.

07 September 2009

The Permissions Sketch

Hey, Terry, Can I borrow your tags?

njr: Hi Terry

terrycojones: Hi Nick

njr: Seeing as we’re friends, could I see your, ratings please?

terrycojones: Sure. I’ve just set an exception on see for terrycojones/rating so njr can see it.

njr: Great thanks. Except, it’s odd. I can see that you have a rating on The Hitchhiker’s Guide to the Galaxy, but I can’t see its value.

terrycojones: Oh, you mean you want read permission. No problem. There I’ve set an exception for njr to be able to read terrycojones/rating too.

njr: Excellent. Yes, I can see it now. Just an 8, eh?

terrycojones: Yeah, well . . .

[A bit later]

njr: Terry, you know how we’re really good friends.

terrycojones: Sure.

njr: Well, I was wondering if it wouldn’t be useful if I couldn’t actually set ratings for you. For instance, when we were talking the other day, you were saying how maybe you didn’t even think THHGTTG deserved an 8, and you planned to take it down to a 6; only there you were working on three tickets and the net was down where you were, and basically you couldn’t be arsed?

terrycojones: Yeah, that makes sense. OK, I’ve given you `update permission on terrycojones/rating now, so you can change it.

njr: Wow, you’re really fast at that. Are you using some cool FluidDB client?

terrycojones: No, I just use curl and send the raw HTTP. The API’s so RESTful . . .

njr: [Rolls eyes]. OK, let me try that. No, not curl—changing your rating. Yeah, that worked. Cool. And I know you also wanted to give Pärt’s Tabular Rasa a 10, so I’ll just do that for you.

terrycojones: Great. Hang on, I’ll just give you permission.

njr: Eh? You’re going senile, mate: you just did. I changed your rating on Hitchhiker’s Guide to the Galaxy, remember?

terrycojones: Yes. But you only needed update permission for that. Now you want to tag a new object with terrycojones/rating. You need create permission to do that. But don’t worry, I’ve given you that too.

njr: Oh wow. These permissions are pretty-fined grained aren’t they? Yeah, I’ve done it. Except—fool!—I tagged the wrong object. I forgot the umlaut on Pärt in the about tag. I know you’re fussy about your accents, being Australian. (Mind you, Pärt might be fussy about it too.) So there I’ve put it on the right one now. Except—that’s weird, I can’t seem to untag the first one. Surely if I have update and create permission, that must allow me to remove a tag, right?

terrycojones: Of course not! Deletion is completely different! But no problem. I’ve given you delete permission on terrycojones/rating now.

njr: Right. So now I can do anything with terrycojones/rating, right? I can see it, read it, tag things with it (create), change tag values (update) and even untag things (delete). Truly, I have power over your ratings.

terrycojones: Yup. You could even make the world’s biggest Mariah Carey fan. But of course, I’d have to kill you if you did that. And you’re the only one apart from me who can set my ratings, so I’ll know.

njr: Consider it done.

But there’s more . . .

terrycojones: Of course, there are still things you can’t do with my ratings.

njr: There are?

terrycojones: Sure. You can’t do anything to the tag itself.

njr: You mean apart from see it, read it, apply it, change it and update it?

terrycojones: I said the tag itself.

njr: The tag . . . itself?

terrycojones: Yup.

njr: That sounds a bit abstract to me, Terry. I’m just a simple physicist. What are you talking about?

terrycojones: See, you change set my ratings, and change my ratings, but you can’t change terrycojones/rating itself. You can’t change what it means. And you can’t delete it.

njr: What it means?

terrycojones: Yes. If you look at the properties of the tag, you’ll find that the description of it is "terrycojones's ratings (Spinal Tap) scale".

njr: Spinal Tap scale?

terrycojones: Sure. Zero to eleven.

njr: Of course. Well, eleven’s nice, if a bit odd; and prime. What do you rate eleven?

terrycojones: Oh not much. The FluidDB permissions system. And Esteve.

njr: Esteve? Better not tell him. He’ll want a raise.

terrycojones: It’s OK; he’s on the exception list. He can’t read my ratings.

njr: But he writes the code!

terrycojones: Yeah, but you should see his principles. He’s uncorruptable.

njr: Alright, alright.

terrycojones: Anyway, the point is, you can’t change that.

njr: The incorruptable genius of Esteve?

terrycojones: Well, that too. But I meant the meaning of my ratings.

njr: Even though I have have every conceivable write permission on the tag?

terrycojones: Yeah, but not on the tag itself.

njr: (If you say itself in that meaningful tone one more time . . .)

terrycojones: Yeah, well, anyway, there’s a separate permission for updating the tag itself.

njr: [Expletives deleted.] Of course there is. And what’s that called?

terrycojones: update.

njr: No, see, you already gave me update permission, wise guy

terrycojones: On the tag. Not the tag itself.

njr: Oh, update on the tag itself. I see. And what about delete? You said I couldn’t delete the tag. But I’ve already removed that Part tag from the object without the umlaut.

terrycojones: Ah yes; but you haven’t deleted the tag itself

njr: [Further colourful expletives deleted] Right. So I can delete every terrycojones/rating you’ve ever put on anything, and indeed, any terrycojones/rating anyone else has ever put on anything for you. But I can’t delete the essence of terrycojones/rating, the meta-data about terrycojones/rating, the terrycojones/rating itself.

terrycojones: That’s right. (And it’s not meta-data; it’s data. All data is equal in FluidDB.)

njr: Whatever. So it that it? Is that really it?

terrycojones: Yup.

njr: So let me see if I have this straight. There are permissions for seeing, creating, reading, updating and deleting tags. And then there are some special administrative permissions for updating and deleting the tag itself.

terrycojones: That’s right.

njr: And that’s absolutely it? If you gave me those, I really would own you. I’d have complete control of terrycojones/rating.

terrycojones: Ah, well, it’s funny you should say that.

njr: There’s more isn’t there?

terrycojones: Well, there’s control.

njr: There’s control.

terrycojones: Yes, there’s control.

njr: Meaning . . .?

terrycojones: Well, who do you think’s been giving you all these permissions to tell the world about my secret infatuation with Mariah Carey?

njr: Ah, yes, Of course. There’s control of tags. Kind of like ownership of files. But surely, you own all the ``terrycojones tags, don’t you?

terrycojones: Sure I do. But I could given them to you if I wanted to. Or I could even let us share them: so we both controlled them.

njr: And that’s called control?

terrycojones: Right.

njr: And that works the same way? With an open/closed policy and an exception list?

terrycojones: Sure does.

njr: Wow. So you could you set it to be closed and not have any exceptions?

terrycojones: Yup.

njr: And then no one would be able to change it?

terrycojones: Not even God.

njr: Really? Is God subject to the FluidDB permissions system.

terrycojones: Well, not yet. She doesn’t have an account. But if she ever gets on, it’s the same rules for her as you, and me, and FluidDB. No exceptions.

njr: Wow, so you if you closed off all the permissions on your ratings and then took away control, your ratings would be like digital tatoos. No one could ever change them.

terrycojones: That’s right. I’m going to rate my mum 10 and then do that.

njr: Aaahhh . . .

Some months later

terrycojones: Hi Nick

njr: Hi Terry

terrycojones: You know how we’re great friends and all that shit?

njr: Sure

terrycojones: And how I trusted you with my ratings, and you mostly didn’t abuse except for that whole rating-Mariah-Carey-11 business.

njr: You love her really.

terrycojones: Whatever. The thing is, I thought it would be good if you have me permission on your njr/guardian-1000 namespace, so I can add some stuff and fix all the unicode you screwed up.

njr: Ah, unicode, yes. OK, what do I need to do.

terrycojones: Well, you could just give me control of it; then I could do anything.

njr: Control of njr/guardian-1000? So I guess control on a namespace is like control on a tag?

terrycojones: On a tag itself, right.

njr: Well, you know, Terry, I trust you and everything, but . . .

terrycojones: Yeah, OK. I don’t really need control. But create permission would be useful.

njr: OK, done. Is that it?

terrycojones: Well, actually, update would be useful too. Since you screwed its description.

njr: So update permission on a namespace is like the administrative update permission on a tag?

terrycojones: On the tag itself. Right. It lets you change the data about the namespace. Like the description.

njr: OK, I’ve done that too. Anything else.

terrycojones: Well you have an open policy on list, so I can see what’s there. But there seems to be some junk. I mean you have a sub-namespace njr/guardian-1000/best-FluidDB-UUIDs. I don’t think the Guardian has published its list of all-time best FluidDB-UUIDs yet. And even if it did, I think you’d want a tag not a namespace for that. So I think we could lose it. You know how I love deleting things.

njr: So delete it!

terrycojones: I will. But you need to give me delete permission on the namespace for that

njr: Right, so that would be another administrative permission on the namespace. Like delete on the abstract tag itself.

terrycojones: You’re finally getting this.

njr: Slowly, slowly, the egg walks, (as they say in Addis Ababa).

terrycojones: You ever been to Addis Ababa?

njr: No. But I know a woman who rates it an 11.

terrycojones: OK. It’s in the njr/guardian-1000/cities now too.

njr: It is?

terrycojones: It is!

njr: OK, Ciao, Terry. I’ll let you get back to your Mariah Carey records.

terrycojones: [Expletives I’ve never even heard before deleted.]

02 September 2009

Permissions Worth Getting Excited About

Permissions. Doesn’t the very subject set your heart racing?

No?

Well, me neither; normally. But, rather to my surprise, my thesis today is that one of the most revolutionary and remarkable features of FluidDB is its permissions system, and that this, more than anything else, explains why it has a shot at blowing apart web applications and social data.

I accept it’s an unlikely claim. But let’s see if I can make a case.

WTF?

If you ever hear Terry talk about FluidDB, or read his stuff, the guy speaks in riddles. One minute he sounds like Richard Stallman, banging on about freedom and how when he rules the world no one will have to ask permission to write data, no one will own objects, everything will be shared and social and anarchic and messy and we’ll all be able to do exactly what we want. He even calls FluidDB the database with the heart of a wiki, whatever that’s supposed to mean. And the very next next minute, he’s talking about how at the heart of FluidDB lies a really strong permissions system that allows us to control exactly what goes in FluidDB.

The guy’s clearly schizophrenic, if not paranoid delusional. Right?

In Defence of Terry

FluidDB is a remarkably simple conception. At its heart, it’s just a collection of initially empty containers to which we can attach tags. In best computer-science tradition, we call the containers objects and they really are 100% shared.

If you can find an object, you can tag it . . . and so can anyone else.
http://StochasticSolutions.com/fluiddb/image/bare-objects.png

As you can see, the objects all have rather long, individual identifiers—actually 128-bit numbers sometimes known as UUIDs [1]. Some objects also have a second identifier, known as the about tag. About tags are unique and never change, so there can only be one object in FluidDB whose about tag is set to the (exact) string oxygen, though there can be any number with no about tag.

http://StochasticSolutions.com/fluiddb/image/objects.png

So objects are not owned, but shared, and once created, are never destroyed [2].

So one half of the claim is good: objects are shared, and anyone can tag them with whatever they like. Where do all the permissions and control come in?

Tags and Values

When we say that anyone can write to any objects in the system, that’s true. Anyone can stick any tag they like onto any object they can find [3]. The tag has a name, such rating or colour or Ulysses and may have a value—such as 7, or interesting or a picture of some leaves. Like this:

http://StochasticSolutions.com/fluiddb/image/LuminousLeaves.png

The only restriction your tag names all start with your FluidDB username. For example, my FluidDB username is njr, so my tag names start with njr/. So I can have njr/rating, njr/colour njr/Ulysses, or anything else I like. And you, Josephine, can have tags called josephine/rating, josephine/colour (or even josephine/color, if you’re that way inclined) and so on.

http://StochasticSolutions.com/fluiddb/image/hitchhiker-njr-josephine.png

Placing tags on the same object in FluidDB is, of course, a way of relating them, but that isn’t the focus of this article.

Permissions

So finally, we come to the bit.

By default, I can only create and set tags starting njr, but can read any tags, while Josephine, naturally enough, can only create and set tags starting josephine. Unlike objects, tags are owned.

But there is a permissions system that means that I, as the owner of the owner of tags starting njr, can control exactly what Josephine can do with my tags and vice versa.

So if I want to, I can hide my ratings from Josephine, but let her see my colours. Or I can hide them from everyone except Josephine and Terry.

Slightly more unusually, I can also choose to allow some, or even all other users to write some of my tags if I like. For example, I might decide that I trust Terry enough to allow him to set some of my tags.

This ability to control who can do what extends not only to users, but to applications as well. So suppose someone writes an application called Read-Planner that allows people to tag any book they run across on the web (or anywhere, in fact) with a to-read tag. If I want to use this application, I don’t have to give it my password or open up access to all the rights that I have: I can simply give Read-Planner the ability to create to-read tags for me. And if at some point I become unhappy with it, of course I can revoke that permission. I don’t even need something like OAuth: I can just do it.

Even better, I can also choose to allow other applications whatever form of access to my to-read tags I want—maybe read only, or maybe some of them have write access too. It’s up to me.

And this is why we think FluidDB has the potential to be so revolutionary: it is truly social, while leaving each user firmly in control of his or her data.

New Rules for Web Applications

At the moment, any of us who use web applications tend to spend a lot of time and effort populating application databases to make them useful to us. But when we do so, we tend to lose control of our data. They go into a private database schema, and what access we have to that depends entirely on what the application allows us to do. Sometimes there are reasonable ways to get the data back out (some kind of an XML dump perhaps), sometimes not. But always the application is in control. And linking data across applications is, in general, somewhere between hard and impossible.

FluidDB can change all that by leaving the user in control of his or her data, granting the application only such permissions as necessary or desired, and ensuring that the user retains flexability and control.

Now obviously, this only works if applications agree to work with data in this way, and equally obviously, a lot of them are going to be extremely reluctant to do so. After all, in the well-worn phrase, information is power. So it’s more than possible that far from embracing the openness and intrinsic cross-application, cross-user interoperability championed and facilitated by FluidDB, many applications will seek to hold onto the data. We fully expect this.

But we also expect that there will be some applications, perhaps new ones, perhaps small ones initially, that will embrace the idea. And over time, a movement can grow, perceptions can change, and maybe in ten years time the idea of an application owning and controlling your data will seems as antiquated as the notion that you shouldn’t be allowed to see your own medical records.

A Tiny Bit of Detail

I’ll close by talking in slightly more detail about the way the permissions system in FluidDB operates. Further articles will go into the gory details.

Briefly, you can set permissions for each group of tags sharing a name. [4] So for example, if I use a rating tag, I can control exactly who can read, change and create this njr/ratings tag. And I can control each of those aspects independently. [5]

Similarly, if I choose to use names that contain slashes, like maybe njr/book/rating and njr/book/own I can choose control who can do what at the level of a stem like njr/rating if I choose. So I could allow some book applications, or a bookish friends, to manipulate my book tags, maybe even creating new ones for me, but not my other tags (especially not my njr/private ones!)

Conclusion

Unlikely as it sounds, one of the key innovations in FluidDB is its combining a completely shared set of information containers (the objects) with the ability for users to tightly control, at a granular level, precisely who can read, write and create the tags used to store information on those objects. This applies not only to users, but also to applications, giving a simple way for users to grant applications broad or narrow access to read or manipulate some or all of their data while, ultimately retaining complete control of it.

In FluidDB, you never need to ask permission to write to an object, but you always need permission to use someone else’s tags.

It’s a powerful combination.


[1]So-called Universal Unique Identifiers.
[2]In fact, we sometimes like to say that objects for every possible about tag already exist, in rather the same way that Plato believed that all numbers and other mathematical objects existed in perpetuity in what we now call his platonic universe. It’s simply that we only bother to allocate storage for objects with any particular about tag when someone actually wants to use it.
[3]If it doesn’t have an about tag, an object might be very hard to find, essentially requiring you to guess a 128-bit number. But it doesn’t really matter, as we’ll see.
[4]We sometimes call the set of tags having, or potentially having a particular name (like njr/rating) as an abstract tag. We can think of the (concrete) tags that we actually attach to FluidDB objects as concrete instantiation of a canonical, abstract, platonic, master tag of the same name.
[5]There are actually more than three aspects I can control, but this the essence of it.

01 September 2009

FDB 1.21

I just pushed fdb.py 1.21 to GitHub (http://github.com/njr0/fdb.py).

The command line utilities and the API now work consistently with subnamespaces using the funcitonality introduced in 1.19.

So

fdb tag -a DADGAD /njr/rather/deep/space/rating=9

will now work, creating namespaces for as many of rather, deep, space and rating as required.

FDB 1.20: Namespace Functions Added to API

I’ve added some new functions to the python API to FluidDB in the fdb.py library. This has been pushed to GitHub (http://github.com/njr0/fdb.py).

At the moment, the functionality is somewhat embryonic, but useful. Basically there are new functions to:

  • Create a namespace, e.g.

    id = db.create_namespace ('/njr/bas/bar/foo',  'three levels deep',
                              verbose=True)
    

    This is recursive, and will create bas under /njr and bar under bas if required.

  • Delete a namespace:

    status = db.delete_namespace ('bas/bar/foo')
    

    This is not recursive, though arguments for recurse and force have been added to the function signature. (They are casually ignored at present.)

  • Fetch the description of a namespace, e.g.

    print db.describe_namespace ('bas/bar/foo')
    

These follow FDB’s usual convention that if paths start with a ‘/’ they are taken to be absolute, and if not they are taken to be relative to the user’s top-level namespace. So for me (njr), /njr/foo/bar is the same as foo/bar.

Points to note:

  • There are no unit tests for this functionality yet (lazy, lazy, bad njr!)
  • There is, however, a set of examples in nstest.py that illustrate and, to some extent, test the functionality (I know, I know, why not just make them tests. I will, I will.)
  • There are no new commands in the CLI to leverage this APi functionality. (As you might imagine, I don’t intend that to be a permanent state of affairs.)
  • Not only are there no new commands, but the existing commands have not been extended to take advantage of the new functionality properly, So, for example, fdb tag still can’t use the new namespace created. Ridiculous? Yes.

The following code shows the API:

import fdb
import simplejson as json

db = fdb.FluidDB ()
user = db.credentials.username

db.create_namespace ('subs', 'Subnamespace of njr', verbose=True)
print
db.delete_namespace ('/%s/subs' % user, verbose=True)
print

db.create_namespace ('bas/bar/foo',  'three levels deep', verbose=True)
print

print db.describe_namespace ('/%s/bas/bar/foo' % user), '\n'
print db.describe_namespace ('/%s/bas/bar' % user), '\n'
print db.describe_namespace ('/%s/bas' % user), '\n\n'

db.delete_namespace ('/%s/bas/bar/foo' % user, verbose=True)
db.delete_namespace ('bas/bar', verbose=True)
db.delete_namespace ('/%s/bas' % user, verbose=True)
print

assert db.describe_namespace ('bas/bar/foo') == 404
assert db.describe_namespace ('bas/bar') == 404
assert db.describe_namespace ('bas') == 404

print 'All namespaces verified to have been deleted.'

It produces the following output (starting from a clean sheet, anyway).

Created namespace /njr/subs with ID 1671aa1f-9cdb-44a0-b55e-ed4ca78553cd

Removed namespace /njr/subs

Created namespace /njr/bas with ID 2ce1aa8e-d069-4d3b-960c-40685c47868f
Created namespace /njr/bas/bar with ID 4328b9ec-d96a-40e4-9a30-6906d360d9a8
Created namespace /njr/bas/bar/foo with ID d83240e7-2863-4cac-95d2-42d61b2050d4

         description: three levels deep
                  id: d83240e7-2863-4cac-95d2-42d61b2050d4
      namespaceNames: []
            tagNames: []

         description: None
                  id: 4328b9ec-d96a-40e4-9a30-6906d360d9a8
      namespaceNames: ['foo']
            tagNames: []

         description: None
                  id: 2ce1aa8e-d069-4d3b-960c-40685c47868f
      namespaceNames: ['bar']
            tagNames: []


Removed namespace /njr/bas/bar/foo
Removed namespace /njr/bas/bar
Removed namespace /njr/bas

All namespaces verified to have been deleted.

Labels