Fish Fixes and setuptools

I have a few pieces of good news. And one piece of not-so-good news.

1. I’ve pushed Fish 4.33 to Github. This version includes two main bug fixes and some useful reorganization. The bug fixes are:

   • One test was failing under Python 2.7 (but not 2.6 or 2.5) That turns out to be because I was lazily doing some json encoding “by hand” and failing to percent-encode non-ASCII characters. httplib in the earlier versions didn’t seem to mind, but in Python 2.7 it does. I am now using a proper json serializer and things are better.

   • It transpires that starting Fish interactively was failing for new users. This is because the interactive version of Fish (the one you get if you just type fish <return>) was doing a sync, and that sync assumes that you have a tag called .fish/alias. New users tend not to have that. Obviously, the solution is to create it, which Fish now does. (It makes it private, too. You can change it to public if you prefer.

     fish perms public .fish/alias

     will do the job.)

   My thanks to Rodrigo Barnes (@rodrigobarnes) for noticing and pointing out both problems.

2. I have switched the default/preferred HTTP library used by Fish from httplib2, which had always caused me problems, to Kenneth Reitz’s requests. I don’t know whether httplib2 is just broken or whether it’s a case of user error, but I have never managed to persuade it to work reliably. It took under 15 minutes to swap it out and replace it with requests, and the process fixed all the problems I knew about with httplib2, so that was pretty fantastic.

   I have talked to various people about requests and the most common response has been “It’s awesome”. Do not misunderstand me when I say, I disagree. My experience merely that requests is well-designed, functional, pythonic and easy-to-use. In those respects it is like most Python libraries. I suspect that the main reason requests generates reactions of awe is that all the other main Python libraries for handling HTTP requests are sub-par [1]. As Kenneth Reitz himself says (specifically of urllib2):

   Things shouldn’t be this way. Not in Python.

   Requests is excellent and I strongly recommend you use it, with one caveat:

   Version 0.9.2, which I used to make the change, works perfectly. Use that.

   I have not yet been able to make version 1.0.1 work with Fish. It is entirely possible that is because I have done something stupid. But at this point, using 1.0.1 will not move you forward.

   The way I have upgraded Fish is first to try to import requests and then to check that the version is less that 1.0.0. If requests is unavailable, or you have 1.0.0 or newer, Fish falls back (silently) to httplib2. To be fair, that mostly works; but requests works better.

   [“What does that mean?”, you ask. Well, I have two problems with httplib2. First, it consistently fails with request bodies over slightly less than 64KB. This is a problem with some file uploads to Fluidinfo. Secondly, when performing bulk uploads, particularly if I use multiple threads, I get occasional failures with httplib2. Retrying sometimes, but not always, works. (This was a major pain uploading, for example, the 2.5 million items in the British National Bibliography. The upload spent about 50% of the time uploading the first 98%, and the other 50% retrying the troublesome 2%. I have yet to see any random failures with requests.]

3. I finally got round to packaging up Fish with setuptools. So if you download it now, you should be able to do a standard

   python setup.py install

   in the package directory and standard, good things will happen. These include installing the fish script, so if you used an alias or specially added it to your path before, this is no longer necessary.

   I also cleaned up the import structure a bit to make this work.

So much for the good news.

The bad news is that I can’t upload it to PyPI, which had been my intention, because there is already a package called fish on PyPI.

Animating fish (and birds) for progress bars

Obviously, I could rename Fish (for a second time), but that seems undesirable when it is about to appear in print in the form of an O’Reilly book (blog post upcoming). So I guess we’ll have to do things the old way.

Needless to say, in the process of all these lovely upgrades, it is possible I broke something. Please let me know if I did and I will endeavour to fix it.

[1]	Funny term, “subpar”. Every golfer aspires to shoot under par: in golf, less is definitely more. But I didn’t feel using the term “super-par” would really have conveyed my meaning.