Release Notes/Change History¶

Release 2.18.1 (r8379, Feb 3, 2014)¶

Bug fixes:

geoip-lite-update: This tool reloads Apache, but the systemctl call added in 2.18.0 for operating systems with systemd was incorrect. This has been fixed and was the reason for this release.
mod_mirrorbrain: Some obsolete configuration directives have been removed (MirrorBrainGeoIPFile, MirrorBrainHandleDirectoryIndexLocally, MirrorBrainMetalinkHashesPathPrefix). MirrorBrainGeoIPFile was also still present in the example configuration file. (issue 146)
mb scan: an obsolete reference to a -f switch was removed that exists only in the behind-the-scene scanner perl script, but not in mb scan. Thanks, Bart!

In the documentation about system tuning, a reference to the PostgreSQL docs has been added.

Release 2.18.0 (r8365, Feb 2, 2014)¶

(Pleae note that a new version of mod_asn was also issued recently.)

New features:

mod_mirrorbrain: Nick Schermer from Xfce contributed a wonderful patch to improve the HTML output for the details pages that MirrorBrain generates (see issue 123). As a consequence, in some installations the web design needs to be adjusted, but hopefully people will value the better possibilities. The full list of changes can be viewed here.
mod_mirrorbrain: If multiple instances of MirrorBrain run in Apache (or you have multiple vhosts using one MirrorBrain configuration), you would have multiple DBDParams statements which prepare SQL statements when Apache starts processes. Peculiarly, Apache doesn’t allow the same connection string used more than once. (To make the connection strings unique, a possible workaround is to use differing connect_timeout values.) Anyhow, to help users running into this problem, MirrorBrain used to log a warning (added in 2009). This warning was removed when the DBD error handling was reworked in 2.16.1 (2012). It replaced with much more detailed error logging, but the helpful one-liner was missing hence. This release re-adds the helpful one-liner, and it’ll also show the workaround at the same time. In addition, the documentation was enhanced. Thanks Stephan Jauernick.
The installation documentation was updated in many places.
mb: new command line option for configuration file path. Patch kindly provided by Gökdeniz Karadağ (see also issue 114)
mb update: The geoiplookup and geoiplookup_city binaries are now also looked for in /usr/share/mirrorbrain. Helps to solve the packaging cleanup issue 110.
geoip-lite-update: It is now possible to run this script without reloading Apache. On the other hand, it can now reload Apache on openSUSE, Ubuntu, Debian, Fedora, and via systemd. Thanks Andrea Veri for the report.

The create_timestamp script no longer contains openSUSE specifics. Usage:

create_timestamp [username:groupname] timestampfile1 [timestampfile2...]

geoiplookup_city and geoiplookup_continent tools: The path to the GeoIP database files is now configurable at compile time, so distribution builders could use preprocessor definitions to define them instead of patching the code. Patch by Dagobert Michelsen. Thanks! (issue 130)
mirrorprobe: in the --help output, display the default value for the network timeout (20 seconds)

Bug fixes:

There’s a version table in the database since recently (created since 2.17.0), but it didn’t contain an id column as primary key, so SQLObject couldn’t work with it. So now we add an id column as primary key to the table (and an existing table from 2.17.0 is migrated by simply recreating it from scratch).
The database SQL scheme for new installations has been updated to add the new column named ipv6_only. Thanks George Koutras, Raphael Hertzog and others for the report (and for their patience)! (issue 119)
The SQL schema was updated to remove obsolete quotes around language names on function declarations: 'plpgsql' -> plpgsql; 'SQL' -> SQL. PostgreSQL 9.2 and newer no longer ignore these wrong quotes.
mod_mirrorbrain: Compiler warnings about using %d for size_t were silenced, by now using APR_SIZE_T_FMT where appropriate. (issue 82)
mb / mirrorbrain.conf: Trailing(!) spaces in passwords were taken literally so far, but were very hard to see and hard to debug. Now, trailing spaces are rightfully ignored (issue 112). Thanks to patch from Pat Riehecky!
mb now gives sane error messages when a config statement is missing/misspelled in /etc/mirrorbrain.conf.
mb: when mod_asn is not installed, an additional ProgrammingError exception from the sqlobject.dberrors can occur. This is now also caught. Thanks Gökdeniz very much for the patch!
mb makehashes: “permission denied” errors are now handled gracefully, fixing issue 105. Thanks Tom Albers for report & patch!
metalink-hasher.py: This very old (backward) compatibility wrapper has been removed from the tools directory, since it is long obsolete.
mb iplookup: On the Solaris/OpenCSW platform, using socket.getaddrinfo() in Python for DNS lookups doesn’t work with port 0. Using None instead seems to be more correct and hopefully work on all platforms. Thanks Dagobert for the fix! (issue 135)
mb edit: A mistyped dash in the commands help output was fixed. Thanks Dago! (issue 136)
mb update: This command now handles errors that lead to Null as prefix or Null as AS number, so the command doesn’t crash anymore under these conditions. (issue 137)
mb dirs: When using the -d or --missing option, only enabled mirrors are shown now. Thanks Florian! (issue 116)
tools/geoiplookup_*: They no longer segfault when opening a GeoIP database database fails. Patch courtesy of Dagobert Michelsen. (issue 138)
mb scan:
- When FTP URLs are not correct, and the directory is not found on an FTP server, the scanner bailed out. Fixed with patch from Dago. (issue 139)
- when in verbose mode, don’t wrongly log symlinks as directories (issue 141)
The file mb/countries.py was never used; remove it so it doesn’t confuse anyone. Thanks Gokdeniz for the hint.
The build on openSUSE 13.1 was fixed.
The build on RHEL6 was fixed, with a patch courtesy of jcpunk. (issue 125)

Release 2.17.0 (r8289, Apr 21, 2012)¶

New features:

mod_mirrorbrain: IPv6 geolocation for IPv6 clients is now enabled. This requires GeoIP 1.4.8 and mod_geoip 1.2.7 or newer (which add experimental support for IPv6 resolution) (issue 106).
mb update, mb iplookup: DNS resolution now works with IPv4 + IPv6.
Support for Metalink/HTTP (RFC 6249) has been implemented (issue 15). This was long on my todo list! This makes MirrorBrain include in its server HTTP responses useful metadata like cryptohashes, mirror URLs and links to alternate representations. There’s support for RFC 5988 Web Linking, for RFC 6249 Metalink/HTTP: Mirrors and Hashes, and for RFC 3230 HTTP Instance Digests (including updates from RFC 5843). Here’s an example (screenshot).
mb edit: An editor set via the environmental variable $VISUAL is now used, if none is set in $EDITOR. This fixed issue 96.
mb db vacuum: A new option -q allows to silence the commands output (issue 99).

New platforms:

Ubuntu 11.10 packages are now built and tested.
Debian 6.0 packages have been tested.

Bug fixes:

mb makehashes: A problem was fixed with filenames containing characters that could be interpreted as magic characters in regular expressions (issue 94) Thanks, KDE sysadmins, for your help!
mirrorprobe: Incomplete responses returned by mirrors (less bytes sent than announced) spawned an annoying error message. Now this error is just logged, as it should.
mb scan: A typo has been fixed (patch by Oliver Beattie)

Internally, a way to migrate the database after updates has been implemented. A table named version keeps info about the state of the database. Thus, database schema upgrades can hopefully be done automatically when possible. This release adds a new flag to the database called ipv6_only to denote mirrors that are not reachable by IPv4. Once this new flag is used, it will allow to redirect clients to this type of mirrors. (And IPv6-only mirrors won’t go away – rather the contrary :-)

Release 2.16.1 (r8261, Mar 25, 2012)¶

Bugs fixed:

mb makehashes: It didn’t work with PGP signature files that were not detached signatures. Non-detached (attached) signatures are now ignored because they could be very large (file size of the original file plus signature) (issue 102). Thanks to Tom Albers for his help here.
mb makehashes: It no longer writes metalink data / cryptohashes into files. All hashes are stored in the database since 2.13.0. The obsolete storage in files had been kept only for backwards compatibility with 2.12 and earlier. (Which are outdated since ~18 months now. So let’s avoid confusion.)
mod_mirrorbrain: Reworked error handling regarding the acquisition of database connections, including more detailed logging of errors. This fixes a crash that affected only setups with MirrorBrainFallback configuration. The crash could occur when no database connection was available – because the logging code wrongly tried to log details about the (unavailable) connection.

New features:

mb mirrorlist: Path names can now contain wildcards. (Edited via mb markers -e). Very nice improvement, thanks to idea and patch from Stephan Jauernick.
There is a little new tool: tools/push2mirrors, an example script to run rsync processes in parallel to push content to mirrors.

Release 2.16.0 (r8251, Feb 21, 2012)¶

This release sums up small fixes that piled up slowly, over a good year.

URL signing is no longer regarded experimental. See Configuring URL signatures for more information on this interesting feature.

mb dirs: A new option was added to list all mirrors which don’t have a specified directory: mb dirs --missing DIR

The following bugs were fixed:

mod_mirrorbrain: The server could crash if there was configuration for fallback mirrors in place and acquisition of a database connection failed. This has been fixed (issue 84).
mb scan:
- If some directories on a mirror return 404, the scanner crashed. This is fixed with a patch kindly contributed by Thorsten Behrens.
- As adjunct to r8180 (terse logging), logging messages in the large file check have been silenced (and their formatting improved, while at it)
mb: A compatibility issue in Python has been fixed, by updating the way how exceptions are raised. Thanks to Christian Lohmaier for bringing this up.
mb makehashes: Handling of non-availability of SHA256 cipher was fixed for old Python versions (issue 85).

In the documentation, the section about Configuring URL signatures has been added, and a general build problem has been documented: On some newer platforms, there is a need for linking the math library (as in -lm).

Release 2.15.0 (r8232, Nov 13, 2010)¶

This release comes with a new feature useful for RPM-based Linux distributions: generation of Yum mirror lists. Another new feature is that nginx directory indexes can be scanned. In addition, there are several bug fixes and improvements, and new documentation on tuning your database server for optimal performance.

Yum-style mirror list support is configured with a new Apache configuration directive which creates a mapping of Yum’s query arguments to directories in the file tree. Please refer to the complete instructions in Serving Yum-style mirror lists.

To make this possible, the main handler function in mod_mirrorbrain is now run before all other configured handlers from other modules, not as the very last one. This means we can run before mod_autoindex, which would otherwise handle a request on a directory, despite the presence of query arguments requesting a yum mirror list. It also means that we run before mod_php (most modules’ handlers run as middle hook and therefore not in strictly defined order).

A small bugfix is that, for generated torrent files, hashes from the database were retrieved twice from the database. This has been fixed.

The mirror scanner (mb scan) underwent the following small improvements, other than implementing support for scanning Nginx directories:

When scanning only a subdirectory, the calculation of added/removed files was wrong. (It functinally did the right thing, but the logging was wrong.)
The pre-scanning check for existance of a subdirectory is now skipped, when scanning only a single mirror.
The messages logged when encoungering unparseable HTML index when scanning over HTTP have been improved.
Logging messages about directories “not in top_include_list” have been silenced (but can be brought up again by increasing verbosity).
Displaying of file sizes > 4GB when scanning over rsync has been fixed (a finding from issue 8).

In mb makehashes, issue 72 has been fixed: If specified path names contain duplicated slashes, these were introduced as wrong filenames into the database.

The documentation on Tuning PostgreSQL has been extensively reworked and gives a complete set of instructions now.

Release 2.14.0 (r8210, Nov 6, 2010)¶

This release brings a number of new features, and also some bug fixes.

On the precondition that the “GeoLite City” GeoIP database is used, MirrorBrain now uses geographical distance as additional criterion in mirror selection. This is useful in
1. large countries (like the US), and probably any countries with many mirrors
2. countries without mirrors, where only a random mirror from the continent could be selected otherwise. (Defining fallback mirrors for such countries worked before, and still has precedence.)
This implements issue 34. To take advantage of this feature, the free GeoLite City GeoIP database needs to be used. See the 2.14.0 upgrade notes for instructions.
Per-file mirror lists have been improved by showing data in a better readable way, and by embedding a link to a Google map visualizing the 9 closest mirrors.
When running behind a load balancer or other reverse proxy, prefix detection (for containment in network prefixes of mirrors) did not work because mod_mirrorbrain only saw the connecting IP address, and didn’t look at an address passed via HTTP headers from the proxy. This has been fixed. (AS, country and continent comparisons already did this.)
Experimental support for restricted downloads has been implemented, by redirecting to temporary URLs whose validity can be verified by the mirrors. See http://www.mail-archive.com/mirrorbrain@mirrorbrain.org/msg00011.html This a prototype implementation that might still be changed, hence the new Apache config directive is called MirrorBrainRedirectStampKey_EXPERIMENTAL at the moment.
MirrorBrain did not accept requests when access was restricted with authentication (e.g. Basic Authentication), due to a broken check which simply needed to be removed. (A bit of code inherited from mod_offload, and likely still dating back to old Apache 1.3 API.)
MirrorBrain has been tested (successfully) against the latest zsync release (0.6.2) and the documentation updated.
Minor optimizations and code cleanups have been done.

Please read the 2.14.0 upgrade notes before upgrading!

Release 2.13.4 (r8188, Oct 19, 2010)¶

This is a maintenance release with improvements in the mirror scan reporting, and small fixes and improved usability. In addition the documentation were enhanced and added to in some places.

Noteworthy are the added instructions on setting up automatic GeoIP database updates (see below).

mb scan:
- The output of the scanner has been improved, by introducing a -q|--quiet option. Used once, only a summary line per scanned mirror will be shown. Used twice, no output will be produced except errors.
- When a scan via rsync ran into a timeout, the name of the affected mirror was not reported. The error message was only “rsync timeout”, and while there normally were other messages giving a hint, output is now improved to include the mirror identifier.
- When enabling a mirror after successful scanning, the scanner now makes sure that the mirror is not only marked “enabled” but also marked being “online”. Mirrors are normally marked online by the mirrorprobe (which is typically run once per minute), but it is much more logical when a mirror is really directly available after scanning with --enable.
mb scan and mirrorprobe:
- There was a case of a quirky web server software that ignores requests without Accept header. The mirrorprobe and the scanner now send an Accept header with value ‘/’, because sending this header in general should not harm.
geoip-lite-update:
- This script now works on Ubuntu. It no longer relies on a command named ftp being capable of doing HTTP downloads, and prefers curl or wget if available.
- The script is quiet now, producing no output if no error is encountered.

Documentation improvements:

The logging configuration example has been updated (See Setting up logging)
The instructions to update the GeoIP databases on Ubuntu have been updated. (See Installation on Debian/Ubuntu Linux)
Documentation (for all platforms) about setting up automatic updates of the GeoIP database was blatantly missing.
A possibly disturbing ‘-’ in front of cron examples has been removed, which work with Vixie cron but not with Anacron as used by Ubuntu.
Ubuntu install docs for 10.04 have been updated.
The example for using the geoiplookup_continent tool now shows how to specify the path to a GeoIP database.

Release 2.13.3 (r8166, Sep 26, 2010)¶

This is a release that fixes two important bugs in the Metalink generator. In addition, it includes a number of compatibility fixes for Torrents.

mod_mirrorbrain:
- The Magnet links embedded in Metalinks could cause the Metalink client aria2c to wait a long time on P2P connections, and not try the listed mirrors anymore (issue 73). These links are no longer included at the moment, unless MirrorBrainMetalinkMagnetLinks On is set in the Apache configuration.
- Under the conditions that
  - an Accept header with application/metalink+xml or metalink4+xml is sent,
  - and the request goes to a path that doesn’t exist,
  - but some extension (.foo) could be split off,
  - and a corresponding path without extension exists,
  mod_mirrorbrain delivered the file matching the path with the extension split off, instead of replying with a 404 Not found. This affected aria2c when it requested non-existing files. The bug was found and fixed by Michael Schröder and closes issue 75.
- When generating Torrents, the order of keys was not obeyed, which should be lexicographical. This is now the case, so the Torrents should be valid also for clients that insist on correct ordering. This should improve the compatibility to some clients, notably rtorrent. Tracked in issue 74 and issue 78.
- The MD5 sum in Torrent info hashes was wrongly sent in binary form, instead of being hex-encoded. In addition, the key was wrongly named md5 while md5sum is the correct name. Fixing issue 77.
- Not a bugfix, but a hopefully useful addition is that Torrents now contain a “created by” key, indicating the generator of the torrent, and the version number (e.g. MirrorBrain/2.13.3). Suggested in issue 65.

Please read the 2.13.3 upgrade notes before upgrading.

Thanks for all kind help and contribution!

Release 2.13.2 (r8153, Sep 19, 2010)¶

This release adds worthwhile new features to the mirror list generator that you will enjoy:

mod_mirrorbrain:
- The content of the mirror lists (details pages) are now wrapped into a XHTML/HTML DIV container with id="mirrorbrain-details". This improves the possibilities for styling in conjunction with a stylesheet linked in via the MirrorBrainMirrorlistStyleSheet directive (issue 63).
- Further individual design can now be achieved by specifying the XHTML/HTML header and footer which are placed around the page body instead of the built-in XHTML (issue 63). This is configured with two new Apache configuration directives.
  
  This is documented here: Styling the mirrorlist / details pages.
- Hashes can now be requested without a filename being included in the response, to simplify parsing (issue 68). This is done by sending the query string only_hash. This works with different ways to request a hash:
```
http://host.example.com/foo.md5?only_hash
http://host.example.com/foo?md5&only_hash
```
  Instead of 99eaed37390ba0571f8d285829ff63fc du.list, the server will just return 99eaed37390ba0571f8d285829ff63fc.
- The filename in hashes can also be suppressed site-wide (and therewith, on the server side) with a new Apache config directive MirrorBrainHashesSuppressFilenames On. It goes into virtualhost context.
- When sending out a hash to a client (as requested by appending e.g. .md5), there is now a double space between hash and filename – just like as the familiar tools like md5sum and sha1sum do it. This should avoid confusion and extra effort in parsing.
- The mirror list’s content type header now comes with UTF-8 as character set, instead of ISO-8859-1, which should make more sense.
mb export --format=mirmon:
- Exporting a mirror list for mirmon has been adjusted to the default in mirmon-2.3 of its option list_style=plain. The other format (list_style=apache) can also be generated, if mb export is used with --format=mirmon-apache. This fixes issue 62.
  
  The documentation Exporting in mirmon format has been updated to reflect this.

Release 2.13.1 (r8136, Sep 18, 2010)¶

This is a minor release, adding some improvements and fixing a bug that sneaked into the last release.

mb edit:
- A problem was fixed that made it impossible to remove an URL by setting it to an empty string. The fix for issue 30 was the culprit. This was a regression that came with the last release (2.13.0).
mb list/edit/show/...:
- In some situations, the fuzzy-matching on mirror identifiers made it impossible to select certain mirrors. Phillip Smith reported this issue and submitted a clever patch, which retains the convenient behaviour, but also allows for selection mirrors by their full name. This fixes issue 61.
mb scan:
- Scanning lighttpd web servers is now supported. Thanks to patch contributed by Phillip Smith. This fixes issue 60.
Changes regarding packaging:
- Thanks to the work of Phillip Smith, there are now packages for Arch Linux and the ArchServer distribution.
- On Debian and Ubuntu, the mirrorbrain user and group are now automatically created by the package, as well as /var/log/mirrorbrain. This simplifies the installation procedure and fixes issue 4.
- Thanks to the help of Cory Fields, the 2.12 -> 2.13.0 upgrade now works seamlessly on Debian/Ubuntu. Fixing issue 57.

Release 2.13.0 (r8123, Sep 6, 2010)¶

This is a big release, with many new features, and lots of bugs fixed. Big effort has also been put in to ensure a seamless upgrade.

Please read the 2.13.0 upgrade notes.

New features:

This release fully supports IETF Metalinks, as finalized in RFC 5854 early in 2010. The extension .meta4 triggers the IETF Metalink response. An HTTP Accept header containing metalink4+xml also elicits this kind of response. This closes issue 14. The old (v3) Metalinks are still supported, and transparent content negotiation (TCN) is supported with both variants.
As the cache of hashes needed to be restructured for this feature, it became possible to implement a number of additional features. Inclusion of various metadata in the mirror lists is supported now (issue 41):
- file size and modification time
- SHA256 hash
- SHA1 hash
- MD5 hashes
- BitTorrent infohash
- link to Metalink
- link to Torrent
- zsync link
- Magnet link (needs testing)
- link to PGP signature (if available)
These metadata pages resp. mirror lists can now be requested by appending .mirrorlist to an URL. The previous way, using a question mark (&mirrorlist) continues to be supported for backwards compatibility.
Thus, MirrorBrain is now a feature-rich hash/metadata server. A so-called “top hash” (cryptographic hash of the complete file) can now be requested. Depending on the extension added to the URL, like .md5, .sha1, or .sha256, the respective representation is returned. This closes issue 42.

Like before, MirrorBrain also stores piece-wise hashes for chunks of the files. The chunk size is now configurable via /etc/mirrorbrain.conf, see Generating Torrents.

All hashes are now stored in the database. (See Database hash store design notes.)

A fallback mechanism is in place to read existing hashes from disk, if the database doesn’t have the new hashes yet (useful for the migration period).
Even though more hashes are calculated, and hashes stored in the database, hashing is twice as fast as before, not relying the external metalink binary any longer. All functionality of the metalink-hasher tool has been integrated into mb makehashes, which makes sure to never read data from disk more than once, regardless of how many hashes are calculated.

The external tool names metalink is no longer used, and the package dependency on the metalink package is no longer there.
MirrorBrain now has a torrent generator embedded. Torrents are generated in realtime (from hashes cached in the database). See Generating Torrents for details. This resolves issue 37.
MirrorBrain now has basic zsync support. The zsync distribution method is rsync over HTTP, so to speak, and MirrorBrain can generate zsync files on-the-fly. MirrorBrain supports the simpler variant which doesn’t look into compressed content. It is compatible to the current zsync release (0.6.1).

See Configuring zsync support for details.

This feature is off by default, because Apache allocates large amounts of memory for large rows from database; this may be worked around in the future.
Initial support for Magnet links. This largely closes issue 38, but requires further testing/finetuning. See Magnet links for documentation.
Ubuntu 10.04 (Lucid) support! (Issue 6 had to be fixed for this.)

While these are the main news, there is a number of smaller feature updates to be listed:

mb makehashes:
- This is the new tool for hashing files. It supersedes the previously used metalink-hasher and the external metalink tool.
- metalink-hasher is a wrapper now, for backwards compatibility, to avoid breaking existing setups.
- A --force option has been added to force refreshing existing hashes.
- The usage example with --base-dir has been improved.
mb list:
- A new option -N|--number-of-files has been added, which displays the number of files that a mirror is known to have.
  
  To achieve this, a new stored procedure mirr_get_nfiles() has been implemented, which retrieves this number, given either a mirror id or its name. It is added automatically when migrating from previous versions, and made available in through the mb.core.mirror_get_nfiles method.
- mb list <mirror identifier> did not work due to a missing module import in the Python script. This has been amended.
mb update:
- This command can now also update country & region info in mirror records (from GeoIP). Before, it updated only the network prefix and AS number, and geographical coordinates. But country and region assignments occasionally change as well.
- A --dry-run option has been added, to allow seeing the changes before applying them.
- An --all option has been added, which updates all metadata, same as when giving -c -a -p --country --region all at once.
- The command now properly takes notice of hostnames that don’t resolve in the DNS (so further action cannot be taken).
mb db sizes:
- The output of this command now includes also the size of the new hashes table.
mb db vacuum:
- The database cleanup now takes into account that files in the filearr table might not exist on any mirror, but only locally - so they could be referenced in the hash table.
mod_mirrorbrain:
- There is an additional logging handle which provides details about the request and the response. The Apache module takes note in the subprocess environment what the client requested and which representation of the file was actually sent as response. Those variables can be used for logging with standard Apache CustomLog configuration with e.g. want:%{WANT}e give:%{GIVE}e.
mod_autoindex_mb:
- The link “Metalink” is no longer displayed. Instead, the link “Mirrors” has been renamed to “Details”.

Bug fixes:

mod_mirrorbrain:
- When a client IP’s network prefix did not match a mirror’s network prefix exactly, the assignment of the client to this mirror would fail, even though the client IP was (also) contained in the mirror’s network prefix. This has been rectified by properly checking for containment of the IP, fixing issue 52.
- Requests with PATH_INFO were not ignored, as they should be. The default behaviour of Apache is to ignore such requests, and CGI or script handler deviate from that. mod_mirrorbrain now also correctly returns 404 Not Found for such requests. This fixes issue 18, as well as openSUSE bug #546396 (which is not publicly readable).
- When the only available mirror(s) had a limitation flag set (such as region_only), and a metalink was transparently negotiated, an empty metalink would result. This is now prevented, and the file delivered directly instead. Other representations (mirror lists, non-negotiated metalinks, torrents, hashes) are generated also if there is no mirror. This was tracked in openSUSE bug #602434. The mirrorlist is improved when there’s no mirror, and can still list all hashes, and give the direct download URL.
- The module now works when the path used in the Apache <Directory> block contains symlinks, fixing issue 17.
- Errors from the database adapter (lower DBD layer) are now resolved to strings, where available.
- Some variable types have been corrected from int to apr_off_t, using apr_atoi64() instead of atoi(). This applies to: min_size, file_maxsize, and the database identifier of a hash row. This at least fixes the info message given when a file is excluded from redirection due to its size. The checks seemed to work nevertheless, because the min_size numbers were small and file_maxsize numbers large, which helped to get the correct result when comparing.
mb scan:
- Usage of FTP authentication was fixed (with credentials encoded into the URL). The change done in January http://svn.mirrorbrain.org/viewvc/mirrorbrain/trunk/tools/scanner.pl?r1=7911&r2=7945 was incomplete in so far that the FTP client used a wrong path now when cd’ing into a directory (complete URL instead of only the path component). This may have worked with some FTP servers, but it definitely didn’t work with vsftpd. Thanks to Deepak Gupta for raising this issue and providing means to analyse it.
- When using the scanner with --enable, to enable a mirror after scanning, it was counter-intuitive that the redirection to the mirror was not immediately happening. The mirrorprobe first needs to mark the mirror online. The scan tool now does this right away. This issue (issue 59) had repeatedly puzzled people.
mb edit:
- Problems that occurred when copying and pasting data on the editing window have been fixed (reported in issue 30).
mirrorprobe:
- A hard-to-catch exception is now handled. If Python’s socket module ran into a timeout while reading a chunked response, the exception would not be passed correctly to the upper layer, so it could not be caught by its name. We now wrap the entire thread into another exception, which would otherwise be bad practice, but is probably okay here, since we already catch all other exceptions. This should fix issue 46.
- In case of exceptions we run into, allow logging the affected mirror’s name.
- If an unhandled exception occurs, a note is printed.
null-rsync:
- Broken links that are replaced by a directory, and point outside the tree, are now correctly removed in the destination tree. (A very special case.)
- Some error messages were improved.

Internal changes:

mod_mirrorbrain:
- Code was generally cleaned up and logging improved.
- A hex decoder for efficient handling of binary data from PostgreSQL was added.
- Old obsolete code has been removed, which was needed before 2009 when mod_geoip didn’t support continent codes yet. Since then, compiling with GeoIP support built-in was still optionally possible, but this old code is now removed.
- The code path has been cleaned up a lot for easier handling of different representation, like hashes that are requested.
- The message which is logged when no hashes where found in the database has been enhanced.
- The obsolete support for generation of plaintext mirror lists (application/mirrorlist-txt) has been removed.
mb:
- Interruptions by Ctrl-C and various other signals are now properly caught.
- The error classes have been revamped and modernized for Python 2.6.
- The script mirrordoctor.py has been renamed to mb.py, in order to avoid confusion. The tool should now be installed with its own name now, and no further symlinking is needed upon installation.
mb makehashes:
- Hashes are also stored for files which exists only locally, and not on any mirror (and which weren’t present in the filearr table yet, therefore). The cleanup mechanism had to be reworked to take this into account.

Documentations improvements:

The installation docs have been restructured: Now there’s a new section explaining the Initial configuration steps on all platforms, and this part is linked from all platform-specific sections as “next step” at their end. This should avoid some confusion. Hand in hand with this change, a cleanup of things scattered in all places is in progress.
A few hints about Tuning PostgreSQL were added to the Tuning guide.
Setting up logging is described in more detail.
Notes about the necessity of Creating a file tree have been added, and alternatives explained.
Reasons why or why not to use mod_asn are discussed in Installing mod_asn.
Installing from Debian packages: There is now a note about expired keys, and how to renew them.
The obsolete MySQL database schema has been removed, which could theoretically be useful for people aiming to run only mod_mirrorbrain, but not the rest of the framework - but is confusing and may cause people assume that MySQL is supported as backend.

Other improvements:

rsyncinfo:

This script is easier to use now. Instead of the arkward syntax it now also takes simple rsync URLs. Before:

rsyncinfo size gd.tuwien.ac.at -m openoffice

Now:

rsyncinfo size gd.tuwien.ac.at::openoffice
rsyncinfo size rsync://gd.tuwien.ac.at/openoffice

bdecode:

A new tool bdecode to parse a Torrent file (or other BEncoded input), and pretty-print it. Useful mainly to work on the Torrent generator in mod_mirrorbrain, but also to compare the generated torrents with torrents that you get from other generators. The tool can take an argument, or read from standard input:
```
bdecode foo.torrent
curl -s <url> | bdecode
```

Please read the 2.13.0 upgrade notes before upgrading.

Thanks for all the help!

Release 2.12.0 (r7957, Feb 10, 2010)¶

This release contains several important bug fixes, a new feature, and documentation fixes.

The new feature is that geographical coordinates of mirrors are stored. This affects newly created mirrors, as well as mirrors whose metadata is updated with mb update -c. The data are obtained from the GeoIP database, if available. Note that only the GeoIP city (lite) database contains this kind of data. The coordinates aren’t used for anything yet, but it’s easily possible now to display mirrors on a map, or to use them to aid mirror selection (which seems helpful in some cases; see issue 34 for a proposal).

For that, mb update got a new option --coordinates to insert (or update) geographical coordinates in the mirror’s database records. The command can be used to add the data to existing mirrors. Just use mb update --coordinates --asn --prefix to update all mirror records with the coordinates, as well as refreshing asn and prefix data.

Bug fixes:

mb scan
- If rsync is 3.0.0 or newer, mb now uses the --contimeout option in addition to --timeout. This fixes issue 12, where problems during opening the connection could lead to an infinite hang, because that period isn’t covered by rsync’s --timeout option. The additional option to configure this timeout became available with rsync 3.0.0.
- Scanning with FTP authentication has been implemented (URLs in the format ftp://user:pass@hostname/path).
mb mirrorlist
- When generating mirror lists, authentication data (in the form of user:password@) is now removed from URLs. The assumption is that if URLs contain such data, it will almost surely be not the intention to publish them.
mod_mirrorbrain
- On some platforms, mod_mirrorbrain didn’t construct proper filenames for the metalink hash cache. The bug was reported for Debian Lenny, and probably also affected some version of Ubuntu (issue 35). This is fixed by using the APR library function apr_off_t_toa() instead of %llu in the format string fix. Thanks Cory for reporting and tracking this down!
- When Metalinks contained FTP URLs, the URL scheme (url type in the XML) was incorrectly set to http. (issue 23). This has been fixed.
mb db shell
- This new command to spawn a database shell turned out to work only by accident – os.execlp() was used wrongly (missing its 0th argument). This has been correected.
mb file ls -u
- When using the -u option with this command to display URLs, broken URLs could result if a base URL doesn’t end in a slash (issue 36). Thanks Vittorio for reporting!
mb new and mb update
- A stupid error in the selection of the best GeoIP database has been fixed. A forgotten break in the code caused the least preferable database to be chosen, of more than one acceptable database file was available.
- Geographical coordinates are saved to mirror database records.
- The readability of DNSrr warnings is improved.

Since when the metalink hash cache had been reimplemented with release 2.10.0 and 2.10.1, there remained a migration path in mod_mirrorbrain and metalink-hasher for reusing the existing hash files. Since this is several versions away (or 5 months), this migration path has been cleaned up in both mod_mirrorbrain and metalink-hasher.

Backward compatibility and migration support (added around r7794) for old filename scheme (.inode_$INODE) in the metalink hash cache removed.
Backward compatibility (added in r7787) for old filename scheme (.metalink-hashes) in the metalink hash cache removed.

When updating from an installation older than 2.10.1, that is no problem – it just means that metalink hashes will be regenerated before they can be used again.

The documentation was enhanced in the following places:

A few examples for using cURL for testing have been added.
The example for creating metalink hashes was wrong. This was fixed, and some more details added.
The usage info of mb update was improved.
The mb update command has been documented (Editing a mirrors network location).

Release 2.11.3 (r7933, Dec 16, 2009)¶

This release contains a number of small improvements in the toolchain, plus small documentation fixes.

null-rsync:
- IO errors returned by rsync are handled now
- remote errors from rsync are ignored now, and we let rsync continue with dry-run deletions.
mb db sizes:
- Sizes of tables from mod_stats are now shown in addition to MirrorBrain’s own tables.
mb db shell:
- The script now uses os.execlp() instead of os.system() to spawn the database commandline interpreter, because the latter doesn’t reliably pass SIGCONT to the subprocess when resuming.
mb list:
- New options -H, -F, -R to display HTTP/FTP/rsync base URLs have been added.
mb mirrorlist:
- The script now tries harder to not leave temp files – also in case of a crash (which may happen when working with templates).
- Add a link to our project in the footer.

Changes in the documentation were:

The new MirrorBrainFallback directive is now documented in the example mod_mirrorbrain.conf.
The -t 20 option has been removed from the mirrorprobe call, since that is the default now. The scan cronjob also has been simplified.
A hint about ulimits has been removed, which turned out to be a band-aid for a purely local problem.
A hint how to load a database dump with mb db shell has been added.

Release 2.11.2 (r7917, Dec 5, 2009)¶

This release improves scanning via FTP and adds a few small features:

mb scan:
- When scanning via FTP, filenames containing whitespace would not be recognized. The regular expression that parses the FTP directory listing has been extended. In addition, a warning is now printed when a line can’t be parsed. This hopefully fixes issue 31.
- when using the FTP protocol for probing for a file or directory, the wrong use of a variable let the result always be negative. This affected subdirectory scans (using mb scan -d path/to/dir), which would igore some mirrors.
mb db:
- new command for database maintenance tasks:
  - mb db sizes — shows sizes of all relations
  - mb db shell — conveniently open a shell for the database
  - mb db vacuum — cleans up dead references (previously: mb vacuum, which still can be used for backwards compatibility.)
mirrorprobe:
- 60 seconds as timeout have always been a bit long. Change the default timeout to 20 seconds, which is also the value suggested in the documentation.

Release 2.11.1 (r7899, Dec 3, 2009)¶

This release fixes a regression in mod_mirrorbrain that was introduced with the 2.11.0 release. It affected Debian and Ubuntu, or more generally all platforms where the APR (Apache Portable Runtime) is version 1.2, not 1.3. The version detection at compile time was not working. This has been corrected, fixing issue 29. Thanks to Cory Fields in tracking down this bug!

Release 2.11.0 (r7896, Dec 2, 2009)¶

A new feature and lots of bug fixes and minor corrections come with this release.

It’s now possible to configure fallback mirrors, via Apache config, in the following form:

MirrorBrainFallback na us ftp://linuxfreedom.com/ultimate/
MirrorBrainFallback eu de http://www.ultimate-edition.org/~ue/

Those mirrors are used when no reachable mirror is found in the database. Thus, these mirrors get all those requests that MirrorBrain would normally deliver itself (you know, the default fallback behaviour).

They are also used in the mirror lists (with priority 1) and metalinks, and country/region selection is done like for normal mirrors. They are used blindly, without knowing their file lists.

This actually allows to run a MirrorBrain instance with a pseudo file tree (cf. recently added null-rsync script.)

A “degraded mode” that continues to work in case of database complete outages is easily achievable now, however for now the code path is less robust in that regard (if fallback mirrors are configured. Otherwise, it shouldn’t). This should be fixed later.

This new feature is still its infancy, but ready to be tested. It may be subject to refinement, based on future discussion.

Other changes in mod_mirrorbrain are:
- The module now automatically makes sure at compile time that its usage of the DBD database API fits to the APR (Apache Portable Runtime) version. The issue was that the semantics of reading result rows was with APR 1.3. With older APR, different semantics need to be used, which hits Debian and Ubuntu. This fixes issue 7.
- The MirrorBrainHandleDirectoryIndexLocally directive has been removed. It was never actually useful, because we never did (and could) redirect to directory listings. For one, a listing might not be available at each URL that we might redirect to. What’s more, since the database only stores file paths and not directories, we can’t actually look up directories. Thus, the directive is now removed, and a warning issued where it is still found in the config.
- The default of MirrorBrainHandleHEADRequestLocally has been changed to Off, and it has been made clearer (in the Apache-internal help text) what the default is. This change mainly has the effect that the directive does not need to be given anymore, in most scenarios.
- The default setting of the MirrorBrainMinSize directive has been documented in its help text.
The documentation for installation on Debian Lenny was tested and corrected where needed. Thanks, TheUni! Minor issues in the Debian packages have been improved, to further simplify the installation. Ubuntu benefits from this as well.
mb
- Parse errors in the configuration file are not caught and and reported nicely.
- Special characters occurring in the password are escaped before passing them to SQLObject/psycopg2, thus fixing issue 27. A remaining issue is that double quotes can’t be used; a warning is issued if it’s attempted.
mb scan:
- A warning that appeared since the last release has been removed. It was caused by the removal of obsolete code, and purely cosmetic.
null-rsync
- An --exclude commandline option has been implemented, to be passed through to rsync.
- Control over the program output can now be exerted by the two new options --quiet and --verbose.
- Usage info is implemented (--help etc.).
- Interruptions by Ctrl-C and similar signals are intercepted now.
metalink-hasher
- When comparing the modification time of a saved metalink hash with that of a source file, the sub(sub-)second portion of the value could be different from the value that has just been set by os.utime(). (Quite surprisingly.) So now, we compare only the int() portion of the value. This fixed issue 24.

Release 2.10.3 (r7871, Nov 28, 2009)¶

This release adds a new script, which hopefully opens up interesting new use cases, called null-rsync. This is a special rsync wrapper which creates a local file tree from a mirror, where all files contain only zeroes instead of real data. The files are created as sparse files, so only the metadata occupies actual space in the filesystem. Modification times and sizes are fully copied, so that even (native) rsync thinks that the file tree is identical.

This script should allow to create a pseudo mirror of arbitrary size (or several mirrors), in order to host MirrorBrain instances which run under the precondition that they always redirects. (This scenario hasn’t tested yet, but should work.) At any rate, it is a good basis for experimentation.

Then, this release fixes some usability issues in the mb tool:

mb new:
- when creating a new mirror, and detecting that the hostname resolved to multiple addresses (round-robin DNS), a warning about this fact was issued. Now, (short of documentaion in the manual) a reference to http://mirrorbrain.org/archive/mirrorbrain/0042.html is added, where the issue has been discussed in depth.
- A proper error message is now shown if an identifier is chosen that already exists.
mb mirrorlist / mb marker:
- The order in which mirrorlist columns are presented is now kept unchanged, so it appears as it was entered into the database.
- The sort order of mirrorlist entries has been improved. Instead of the priority, the mirror operator name is now given precendence in order, which results in a mirror list that actually looks sorted.

Release 2.10.2 (r7853, Nov 9, 2009)¶

Some non-code changes that should be mentioned:

The documentation was updated in various places. Notably, there are now instructions for Installation on Debian/Ubuntu Linux, which David Farning deserves credits for.
Ubuntu (and Debian) packages have been created. The Ubuntu packages have been tested successfully. (See download page.)
A bug tracking system has been set up: http://mirrorbrain.org/issues/

In the code, the following bugs were fixed:

The mirrorprobe could crash when the sender domain of a configured mail log handler wasn’t resolvable (issue #9). This has been fixed.
When scanning a subdirectory, the mirror scanner (mb scan) could accidentally delete files from the database outside of that directory. This was caused by lack of terminatation (with a slash) of the path expression that is used to grab the list of known files before the scan. Herewith fixing issue #19.
A misleading error message in the mb tool was improved, which was issued when encountering config file with missing sections.

Release 2.10.1 (r7798, Sep 9, 2009)¶

The implementation of the hash cache created by the metalink-hasher tool has been revised again. The reason is that some filesystems (at least the VirtualBox Shared Folder) don’t implement stable inode numbers. Instead of the inode number, now the file size (plus filename and modification time) is used to identify file hashes. (These are the same criteria that rsync uses, by the way.)

Existing hashes are migrated, so that the files don’t need to be hashed again (which could potentially be time-consuming).

The modification time of files is now copied to the hash file, so it is available for comparison when checking if a hash file is up to date.

mod_mirrorbrain has been adapted for the new cache scheme. Also, it is now required that the modification time of the hash file matches the modification time of the file. (For backwards compability, the module still also checks for files matching the old scheme.)

To ease the migration, and since it doesn’t matter otherwise, non-existance of files to be unlinked is ignored now. This occurs for instance in the above mentioned migration scenario, where the hash files are renamed to a different name.
New features in the metalink-hasher tool:
- Per-directory locking was implemented: directories where already a job is running will be skipped. This allows for hassle-free parallel runs of more than one job.
  
  Note that simultaneous spawning of the script still needs to be controlled, to avoid consuming too much I/O or CPU bandwidth for a machine.
- Ctrl-C key presses and common interrupting signals are now handled properly.

Release 2.10.0 (r7789, Sep 4, 2009)¶

The cache of metalink hashes, as created by the metalink-hasher, was changed to more reliably detect changes in the origin files. So far, the file modification time was the criterion to invalidate cached hashes. When files were replaced with older versions (version with smaller mtime), this wasn’t detected, and a cached hash would not be correctly invalidated. https://bugzilla.novell.com/536495 reports this of being an issue.

To fix this, the cache now also uses the file inode as criterion.

mod_mirrorbrain was updated to use the new inode-wise metalink hashes. At the same time, it still knows how to use the previous scheme as fallback. If the new-style hash isn’t found, it looks for the old-style hash file.

Thus, the transition should be seamless, and no special steps should be required when upgrading. Note however that all hashes are regenerated, which could take a while for large file trees, and which could lead to cron jobs stacking up.
There were a number of enhancements, and small bug fixes, in the mb tool (and accompanying Python module):
- mb new:
  - When adding new mirrors, the hostname part in the HTTP base URL might contain a port number. This is now recognized correctly, so the DNS lookup, GeoIP lookup and ASN lookup for the hostname string can work.
  - The commandline options --region-only, --country-only, --as-only, --prefix-only were added, each setting the respective flag.
  - The commandline options --operator-name and --operator-url were added.
  - The --score option is depreciated, since it has been renamed it to --prio.
- mb scan:
  - The passing of arguments to the scanner script was fixed in the case where the -j (--jobs) option was used together with mirror identifier specified on the commandline.
- mb list:
  - Command line options to display the boolean flags were added: --region-only, --country-only, --as-only and --prefix-only.
- mb scan and mb file ls --probe:
  - the lookup whether the multiprocessing or processing module exist was fixed: it could print a false warning that none of them was installed.
The mirrorprobe program no longer logs to the console (stderr). This allows for running the script without redirection its output to /dev/null — which could mean swallowing important errors in the end.

A scenario was documented where the mirrorprobe could fail on machines with little memory and many mirrors to check. The fix is to properly set ulimits to allow a large enough stack size.

Error handling was cleaned up; more errors are handled (e.g. socket timeouts during response reading) and logged properly; and for exceptions yet unhandled, info about the mirror that caused them is printed.

Release 2.9.2 (Aug 21, 2009)¶

Most work happened on the documentation, which includes
- more installation instructions,
- directions for upgrading,
- some tuning hints,
- a quite complete walkthrough through the usage of the mb commandline tool to maintain the mirror database,
- instructions how to set up change notifications (Exporting to a Version Control System (VCS))
- list of known problems, and these release notes.
The documentation is in the docs subdirectory, as well as online at http://mirrorbrain.org/docs/.

Notably, there is a new section How to improve this documentation, which explains how to work on the docs.
New features:
- mb export can now generate a mirmon mirror list. Thus, it is easy to deploy mirmon, automatically scanning the mirrors that are in the database. See Exporting in mirmon format for usage info.
- In mod_autoindex_mb, displaying the “Mirrors” and “Metalink” links was implemented for configurations with Apache’s IndexOptions HTMLTable configured.
Two minor bugs were fixed:
- Missing slash added in mod_autoindex_mb to terminate the XHTML br element in the footer.
- The scanner now ignores rsync temp directories (.~tmp~) also when they occur at the top level of the tree, and not below.

Release 2.9.1 (Jul 30, 2009)¶

mb new
- Now an understandable error message is printed when the geoiplookup_continent couldn’t be executed. Thanks to Daniel Dawidow for providing helpful information to track this down.
mod_mirrorbrain
- Under unusual circumstances it may happen that mod_mirrorbrain can’t retrieve a prepared SQL statement. This occurs when an identical database connection string is being used in different virtual hosts. To ease tracking down this special case, the module now logs additional information that could be useful for debugging. Also, it logs a hint noting that connection strings defined with DBDParams must be unique, and identical strings cannot be used in two virtual hosts.
The mod_mirrorbrain example configuration files were updated to reflect several recent (or not so recent) changes:
- the switch to PostgreSQL
- the now disabled memcache support
- the updated GeoIP database path (/var/lib/GeoIP instead of /usr/share/GeoIP)

Release 2.9.0 (Jul 28, 2009)¶

A very hindering restriction in the mb tool which made it require mod_asn to be installed alongside MirrorBrain has been removed. MirrorBrain can now be installed without installing mod_asn.
The Subversion repository was moved to http://svn.mirrorbrain.org/svn/mirrorbrain/trunk/.
rsync authentication was fixed. Credentials given in rsync URLs in the form of rsync://<username>:<password>@<host>/<module> now work as expected. Patch by Lars Vogdt.
The documentation has been moved into a docs subdirectory, and is rewritten in reStructured Text format, from which HTML is be generated via Sphinx (http://sphinx.pocoo.org/). Whenever the documentation is changed in subversion, the changes automatically get online on http://mirrorbrain.org/docs/
Parallelized mirror probing. Note: for this new feature, the Python modules processing or multiprocessing need to be installed. If none of them is found, the fallback behaviour is to probe serially, like it was done before. This new feature affects the mb probefile and mb file commands, and not actually the mirrorprobe, which has always ran threaded. It also affects the scanner (mb scan) to speed up the checks done when only a subdirectory is scanned.
Various new features were implemented in the mb tool:
- mb probefile
  - Implemented downloading (and displaying) of content.
  - A --urls switch was added, to select the kind of URLs to be probed.
    - --urls=scan probes the URLs that would be used in scanning.
    - --urls=http probes the (HTTP) base URLs used in redirection.
    - --urls=all probes all registered URLs.
  - The usual proxy environment variables are unset before probing (http_proxy, HTTP_PROXY, ftp_proxy, FTP_PROXY)
  - Report the mirror identifier for FTP socket timeouts
- mb scan
  - Logging output was considerably improved, avoiding lots of ugly messages which look like real errors (and tend to cover real ones)
  - The time that a scan took is now shown.
- mb new
  - while looking up a mirror’s location when a new mirror is added, try different geoip database locations (GeoIP database was moved around on openSUSE…).
  - prefer the larger city lite database, if available, and prefer updated copies that were fetched with the geoip-lite-update tool.
- mb list
  - add --other-countries option to allow displaying the countries that a mirror is configured to handle in addition to its own country
mod_mirrorbrain: in the generator tag of metalinks, include mod_mirrorbrain’s version string
The metalink-hasher tool has been revised to implement a number of lacking features:
- Automatic removal of old hashes, which don’t have a pendant in the file tree anymore, is implemented now.
- A summary of deletions is printed after a run.
- A number of things were optimized to run more efficiently on huge trees, mainly by eliminating all redundant stat() calls.
- sha256 was added to the list of digests to generated.
- The need to specify the -b (--base-dir) option was eliminated, which makes the command easier to use.
- The order in which the tool works through the todo list of directories was changed to be alphabetical.
- Using a Python set() builtin type instead of a list can speed up finding obsolete files in the destination directory by 10 times, for huge directories.
- The program output and program help was improved generally.
- Various errors are caught and/or ignored, like vanishing directories and exceptions encountered when recursively removing ignored directories.
- The indentation of verification containers was corrected, so it looks sane in the metalink in the end.
- The version was bumped to 1.2.
geoip-lite-update: This tool to fetch GeoIP databases has been updated to use the path that’s used in the openSUSE package since recently (/var/lib/GeoIP), and which complies better to the Linux Filesystem Hierarchy Standard. It still tries the old location (/usr/share/GeoIP) as well, so to continue to work in a previous setup.
mirrorprobe
- A logrotate snippet was added.
- The mirrorprobe logfile was moved to the /var/log/mirrorbrain/ directory.
The openSUSE RPM package now creates a user and group named mirrorbrain upon installation. Also, it packages a runtime directory /var/run/mirrorbrain (which is cleaned up upon booting) and a log directory /var/log/mirrorbrain. Some additional Requires have been added, on the perl-TimeDate, metalink and libapr-util1-dbd-pgsql packages.

Release 2.8.1 (Jun 5, 2009)¶

Python 2.6 compatibility fixes:
- mb file ls --md5 now uses the hashlib module, if available (this fixes a DepracationWarning given by Python 2.6 when importing the md5 module).
- mb list: The --as option had to be renamed to --asn, because as is a reserved keyword in Python, and Python 2.6 is more strict about noticing this also in cases where just used as an attribute.
- The b64_md5 function was removed, which was no longer used since a while.
mb file ls
- make the --md5 option imply the --probe option
mb export
- when exporting metadata for import into a VCS (version control system), handle additions and deletions
The docs were updated to point to new RPM packages in the openSUSE build service (in a repository named Apache:MirrorBrain). The formerly monolithic package has been split up into subpackages.
perl-Config-IniFiles was added to the list of perl packages required by the scanner (mb scan)

Release 2.8 (Mar 31, 2009)¶

Improvements in the scanner, mainly with regard to the definition of patterns for files (and directories) that are to be included from scanning. Old, hardcoded stuff from the scanner has been removed. Now, excludes can be defined in /etc/mirrorbrain.conf by the scan_exclude and scan_exclude_rsync directives. The former takes regular expressions and is effective for FTP and HTTP scans, while the latter takes rsync patterns, which are passed directly to the remote rsync daemon. See http://mirrorbrain.org/archive/mirrorbrain-commits/0140.html for details. This can decrease the size of the database (>20% for openSUSE), and for many mirrors it considerably shortens the scan time.
Fixed a bug where the scanner aborted when encountering filenames in (valid or invalid) UTF-8 encoding. See https://bugzilla.novell.com/show_bug.cgi?id=490009
Improved the implementation of exclusions as well as the top-level-inclusion pattern, which were not correctly implemented to work in subdir scans.
The documentation was enhanced in some places.
mod_autoindex_mb (which is based on mod_autoindex) was rebased on httpd-2.2.11.
mb dirs: new subcommand for showing directories that the database contains, useful to tune scan exclude patterns.
mb export: implement a new output format, named vcs. Can be used to commit changes to a subversion repository and get change notifications from it. See http://mirrorbrain.org/archive/mirrorbrain-commits/0152.html
Partial deletions (for subdir scans) have been implemented.
mb list accept --country --region --prefix --as --prio options to influence which details are output by it.
mb file: support for probing files, with optional md5 hash check of the downloaded content.
The latter three changes have already been described in more detail at http://mirrorbrain.org/news_items/2.7_mb_toolchain_work

Release 2.7 (Mar 4, 2009)¶

Completely reworked the file database. It is 5x faster and one third the size. Instead of a potentially huge relational table including timestamps (48 bytes per row), files and associations are now in a single table, using smallint arrays for the mirror ids. This makes the table 5x faster and 1/3 the size. In addition, we need only a single index on the path, which is a small and very fast b-tree. This also gives us a good search, and the chance to do partial deletions (e.g. for a subtree).
With this change, MySQL is no longer supported. The core, mod_mirrorbrain, would still work fine, but the toolchain around is quite a bit specific to the PostgreSQL database scheme now. If there’s interest, MySQL support in the toolchain can be maintained as well.
many little improvements in the toolchain were made.
Notably, the scanner has been improved to be more efficient and give better output.
mirror choice can be influenced for testing with a query parameter (as=), specifying the autonomous system number.

Release 2.6 (Feb 13, 2009)¶

supports additional, finer mirror selection, based on network topological criteria, network prefix and autonomous system number, using mod_asn and global routing data.
updated database schemes and toolchain – PostgreSQL support is solid now
work on installation documentation for both MySQL and PostgreSQL (the latter is recommended now, because it allows for nifty features in the future. The mb tool has an mb export subcommand now, perfect to migrate the database.)
toolchain work

Release 2.5 (Feb 3, 2009)¶

working on PostgreSQL support
working on the INSTALL documentation
scanner: 0.22
- more efficient SQL statement handling
- output much improved
- added SQL logging option for debugging
mb (mirrorbrain tool):
- bugfix in the mb file command: make patterns work which have a wildcard as first character.
- extend mb scan to accept -v and --sql-debug and pass it to the scanner

Release 2.4 (Jan 23, 2009)¶

rename mod_zrkadlo to mod_mirrorbrain
use mod_geoip for GeoIP lookups, instead of doing it ourselves. We can now use the GeoIP city database for instance
handle satellite “country” called A2
auto-reenable dead mirrors
geoiplookup_city added, new tool to show details from GeoIP city databases
geoip-lite-update tool updated, with adjusted URL for GeoLite databases. It also downloads the city database now.
deprecate clientip query parameter, which can no longer work once we use mod_geoip. Implement country parameter that can be used instead.
make memcache support optional at compile time

Release 2.3 (Dec 13, 2008)¶

add commandline tool to edit marker files. (Marker files are used to generate mirror lists. Each marker file is used to determine whether a mirror mirrors a certain subtree.)
improvements and few features in the toolchain:
- the mirrorprobe now does GET requests instead of HEAD requests.
- mb, the mirrorbrain tool, has a powerful mb probefile command now that can check for existance of a file on all mirrors, probing all URLs. This is especially useful for checking whether the permission setup for staged content is correct on all mirrors.
new database fields: public_notes, operator_name, operator_url
new database tables: country, region
generate mirror lists

Release 2.2 (Nov 22, 2008)¶

simplified database layout, with additional space save.

Release 2.1 (Nov 9, 2008)¶

simplified the Apache configuration: It is no longer needed to configure a database query. At the same time it’s less error-prone and avoids trouble if one forgets to update the query, when the database schema changes.
specific mirrors can be now configured to get only requests for files < n bytes

Release 2.0 (Nov 3, 2008)¶

implement better fallback mirror selection
add mb file tool to list/add/rm files in the mirror database

Release 1.9 (Oct 26, 2008)¶

add bittorrent links (to all .torrent files that are found) into metalinks
embed PGP signatures (.asc files) into metalinks
add configurable CSS stylesheet to mirror lists
mod_zrkadlo:
- implement the redirection exceptions (file too small, mime type not allowed to be redirected etc) for transparently negotiated metalinks.
- add Vary header on all transparently negotiated resources.
- allow to use the apache module and all tools with multiple instances of the mirrorbrain. Now, one machine / one Apache can host multiple separate instances, each in a vhost.
new, better implementation of rsyncusers tool
bugfixes in the scanner, mainly for scanning via HTML
installation instructions updated
a number of small bugs in the tools were fixed and several improvements added.
added “mirrordoctor”, a commandline tool to maintain mirror entries in the database. Finally!

Release 1.8 (Jun 2, 2008)¶

mod_zrkadlo now uses mod_memcache for the configuration and initialization of memcache
metalink-hasher script added, to prepare hashes for injection into metalink files
rsyncusers analysis tool added
rsyncinfo tool added
scanner bugfix regarding following redirects for large file checks
failover testbed for text mirrorlists implemented
metalinks: switch back to RFC 822 format
new ZrkadloMetalinkPublisher directive
fix issue with <size> element
now there is another (more natural) way to request a metalink: by appending .metalink to the filename.
change metalink negotiation to look for application/metalink+xml in the Accept header (keep Accept-Features for now, but it is going to be removed probably)

Release 1.7 (Apr 21, 2008)¶

new terse text-based mirrorlist
allow clients to use RFC 2295 Accept-Features header to select variants (metalink or mirrorlist-txt)
metalink hash includes can now be out-of-tree
mod_autoindex_mb added
adding a content-disposition header

Older changes¶

Please refer to the subversion changelog: http://svn.mirrorbrain.org/svn/mirrorbrain/trunk respectively http://svn.mirrorbrain.org/viewvc/mirrorbrain/trunk/