Discussion:
[janitorial] CI now builds snapshots with "funny" versions
(too old to reply)
Olaf Meeuwissen
2017-06-17 08:18:40 UTC
Permalink
Raw Message
Dear list,

You may have seen (and even remember!) posts during the preparations for
1.0.27 about using `git` repository info to generate the package version
number for snapshots[1].

[1]: https://lists.alioth.debian.org/pipermail/sane-devel/2017-May/035305.html

After running in semi-circles, chasing dead ends and barking up a wrong
tree, or two, I'm happy to inform you that the CI builds on GitLab.com
now do. If all you care about is getting a tarball for the latest push
to git (that has reached the GitLab.com mirror!), all you need to do is
something like

wget -O - https://gitlab.com/sane-project/backends/builds/artifacts/master/download?job=archive \
| funzip | tar xzf -

to get an extracted sane-backend-*/ directory.

# The URL gives you a zipped archive of all the CI job's artifacts,
# hence, the pipes.

If you're more of a point-and-click type, point your browser at

https://gitlab.com/sane-project/backends

and click the download button (see attachment).

The most recent build as of writing has a version number of

1.0.27-24-ge7d9779-dirty

That's 24 commits since the annotated 1.0.27 tag (added as an alias of
RELEASE_1_0_27), with a 'g'it commit hash of e7d9779. The '-dirty' bit
is due to uncommitted changes to configure.ac and configure that result
from the way things are implemented at present.

If you like to know what's going on "under the covers" and why things
are done the way they are, then please read on.

My first thought was to use the git-version-gen module of gnulib. It
turned out that gnulib was a *FAT* dependency. Too fat for my liking.
A bit of poking around later, I realized that just pulling the script
from the module would do just fine. Tried that and realized after a
while that the script was WAY over-engineered for what I was trying to
achieve.

All that's needed is simply changing the second argument to AC_INIT in
configure.ac to

m4_esyscmd_s([git describe --dirty])

and run `autoconf -f` whenever you want the version info updated. Now
here's the snag. Running autoconf changes configure which we version
control with git. Doh! Infinite loop.

Since I didn't want to kick the status quo (just yet ;-) about keeping
generated files under version control, I implemented the current support
in the `.gitlab-ci.yml` file. That got a little bit more complicated
than necessary because of the change to a version controlled file but it
works without affecting whatever happens on Alioth.

So then, what does happen on Alioth when the daily snapshots are made?
These snapshots are, unsurprisingly, courtesy of a daily cron job. It
runs something close to

git archive --format=tar master | gzip > sane-backends-$DATE.tar.gz

I've tried to run `make dist` on Alioth. No such luck. That bombs
because the gettext tools are not installed. Also, you cannot run
autoconf on Alioth. It's not installed. So, trying to do this git
repository info based versioning thingy on Alioth is a no-go.

The fact you cannot run `make dist` on Alioth is probably why the daily
snapshots are just `git archive`s, but those archives are not made the
way as our release tarballs. And that bugs me. Maybe I should change
that cron job to pull the tarballs from GitLab.com ...

The git archives can be had from GitLab.com as well, BTW. Just point
your favourite download tool at

https://gitlab.com/sane-project/backends/repository/archive.tar.gz?ref=master

Well then, there you have it. We now have tarball snapshots that use
git repository info based versioning on GitLab.com and they are updated
with every push to Alioth shortly after the mirror sync runs on GitLab.

Next thing I would like to do is drop those pesky generated files from
git (perhaps after that daily cronjob starts pulling the snapshots from
GitLab so the unsuspecting user doesn't really notice ;-). Does that
sound like a good idea?

Hope this helps,
--
Olaf Meeuwissen, LPIC-2 FSF Associate Member since 2004-01-27
GnuPG key: F84A2DD9/B3C0 2F47 EA19 64F4 9F13 F43E B8A4 A88A F84A 2DD9
Support Free Software https://my.fsf.org/donate
Join the Free Software Foundation https://my.fsf.org/join
Olaf Meeuwissen
2017-07-01 12:32:12 UTC
Permalink
Raw Message
Hi Allan,
Post by Olaf Meeuwissen
Dear list,
[ ... creating `git describe --dirty` versioned tarballs ...]
So then, what does happen on Alioth when the daily snapshots are made?
These snapshots are, unsurprisingly, courtesy of a daily cron job. It
runs something close to
git archive --format=tar master | gzip > sane-backends-$DATE.tar.gz
I've tried to run `make dist` on Alioth. No such luck. [...]
The fact you cannot run `make dist` on Alioth is probably why the daily
snapshots are just `git archive`s, but those archives are not made the
way as our release tarballs. And that bugs me. Maybe I should change
that cron job to pull the tarballs from GitLab.com ...
Well, I did just that and ran it manually without any hitch. There is
now an extra snapshot[1] that is made the same way we roll our release
tarballs.

[1] http://sane-project.org/snapshots/

This snapshot corresponds to the most current git checkout that passed
the CI build at GitLab.com.

The updated make-git-snapshots.sh should (famous last words!) run
without any trouble and produce no output. If, however, you start
receiving mail from your daily sane cronjob, just forward it to me
and I'll fix it up.

Hope this helps,
--
Olaf Meeuwissen, LPIC-2 FSF Associate Member since 2004-01-27
GnuPG key: F84A2DD9/B3C0 2F47 EA19 64F4 9F13 F43E B8A4 A88A F84A 2DD9
Support Free Software https://my.fsf.org/donate
Join the Free Software Foundation https://my.fsf.org/join
--
sane-devel mailing list: sane-***@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/sane-devel
Unsubscribe: Send mail with subject "unsubscribe your_password"
to sane-devel-***@lists.alioth.debian.org
Olaf Meeuwissen
2017-07-06 12:39:51 UTC
Permalink
Raw Message
Hi All,

I found it most unfair that sane-frontends didn't get the same snapshot
tarball treatment as sane-backends does so I set up a simple CI job over
on GitLab.com. All it does is roll a source tarball. Nothing more. It
doesn't even try to compile the sources. But! We now have a

sane-frontends-1.0.14-31-gc1b7785-dirty.tar.gz

on http://sane-project.org/snapshots/

Now I'm wondering if I should drop the daily git archives ...
# ... and update the SANE - Download web page[1].
# [1]: http://sane-project.org/source.html

For those of you running cron jobs against those git archives, perhaps
now is your chance to speak your mind (or change your cron job ;-)
Post by Olaf Meeuwissen
Hi Allan,
Post by Olaf Meeuwissen
Dear list,
[ ... creating `git describe --dirty` versioned tarballs ...]
So then, what does happen on Alioth when the daily snapshots are made?
These snapshots are, unsurprisingly, courtesy of a daily cron job. It
runs something close to
git archive --format=tar master | gzip > sane-backends-$DATE.tar.gz
I've tried to run `make dist` on Alioth. No such luck. [...]
The fact you cannot run `make dist` on Alioth is probably why the daily
snapshots are just `git archive`s, but those archives are not made the
way as our release tarballs. And that bugs me. Maybe I should change
that cron job to pull the tarballs from GitLab.com ...
Well, I did just that and ran it manually without any hitch. There is
now an extra snapshot[1] that is made the same way we roll our release
tarballs.
[1] http://sane-project.org/snapshots/
This snapshot corresponds to the most current git checkout that passed
the CI build at GitLab.com.
The updated make-git-snapshots.sh should (famous last words!) run
without any trouble and produce no output. If, however, you start
receiving mail from your daily sane cronjob, just forward it to me
and I'll fix it up.
Hope this helps,
--
Olaf Meeuwissen, LPIC-2 FSF Associate Member since 2004-01-27
GnuPG key: F84A2DD9/B3C0 2F47 EA19 64F4 9F13 F43E B8A4 A88A F84A 2DD9
Support Free Software https://my.fsf.org/donate
Join the Free Software Foundation https://my.fsf.org/join
--
sane-devel mailing list: sane-***@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/sane-devel
Unsubscribe: Send mail with subject "unsubscribe your_password"
to sane-devel-***@lists.alioth.debian.org
Loading...