Making debug symbols discoverable and fetchable

Michael wrote a few days ago about the experience of debugging programs on Debian.  And he is certainly not the only one, who found it more difficult to find debug symbols on Linux systems in general.

But fortunately, it is a fixable problem.  Basically, we just need a service to map a build-id to a downloadable file containing that build-id.  You can find the source code to my (prototype) of such a dbgsym service on salsa.debian.org.

It exposes one API endpoint, “/api/v1/find-dbgsym”, which accepts a build-id and returns some data about that build-id (or HTTP 404 if we do not know the build-id).  An example:

$ curl --silent http://127.0.0.1:8000/api/v1/find-dbgsym/5e625512829cfa9b98a8d475092658cb561ad0c8/ | python -m json.tool
{
    "package": {
        "architecture": "amd64",
        "build_ids": [
            "5e625512829cfa9b98a8d475092658cb561ad0c8"
        ],
        "checksums": {
            "sha1": "a7a38b49689031dc506790548cd09789769cfad3",
            "sha256": "3706bbdecd0975e8c55b4ba14a481d4326746f1f18adcd1bd8abc7b5a075679b"
        },
        "download_size": 18032,
        "download_urls": [
            "https://snapshot.debian.org/archive/debian-debug/20161028T101957Z/pool/main/6/6tunnel/6tunnel-dbgsym_0.12-1_amd64.deb"
        ],
        "name": "6tunnel-dbgsym",
        "version": "1:0.12-1"
    }
}

Notice how it includes a download URL and a SHA256 checksum, so with this you can download the package containing the build-id directly from this and verify the download. The sample_client.py included in the repo does that and might be a useful basis for others interested in developing a client for this service.

To seed the database, so it can actually answer these queries, there is a bulk importer that parses Packages files from the Debian archive (for people testing: the ones from debian-debug archive are usually more interesting as they have more build-ids).

Possible improvements

• Have this service deployed somewhere on the internet rather than the loopback interface on my machine.
• The concept is basically distribution agnostic (Michael’s post in fact links to a similar service) and this could be a standard service/tool for all Linux distributions (or even just organizations).  I am happy to work with people outside Debian to make the code useful for their distribution (including non-Debian based distros).
  • The prototype was primarily based around Debian because it was my point of reference (plus what I had data for).
• The bulk importer could (hopefully) be faster on new entries.
• The bulk importer could import the download urls as well, so we do not have to fetch the relevant data online the first time when people are waiting.
• Most of the django code / setup could probably have been done better as this has been an excuse to learn django as well.
  • Like some basic content on the service to help people get started.
  • And a test suite to avoid regressions.

Patches and help welcome.

Kudos

This prototype would not have been possible without python3, django, django’s restframework, python’s APT module, Postgresql, PoWA and, of course, snapshot.debian.org (including its API).