Requests from Microsoft IP Addresses

[Richard Biener]

On Mon, 26 Jun 2023, Mathias De Mar? wrote:

> Hello,
> 
> 
> 
> I?d like to give a little bit of additional information about a Mercurial
> feature that might provide some help here. My apologies if this is old news
> and doesn't add anything useful.
> 
> (Small context: I?m a Mercurial contributor and I maintain a set of
> repositories at our company that are cloned a lot at the same point in time)
> 
> 
> 
> By default, when you do a clone, Mercurial will take the repository,
> compress it and send it across the wire to the client.
> 
> I simulated similar behaviour for the gmp repository by creating a bundle:
> 
>    - it takes about 8 seconds (so approximately 1 core occupied for < 8
>    seconds, as there is a bit of overhead involved)
>    - it generates a bundle of about 14 MB
> 
> That?s not much, but it of course becomes problematic when there are 700
> clones at nearly the exact same moment, like you experienced.
> 
> 
> 
> I cannot help in reducing the bandwidth, but assuming the cpu usage is the
> problem, ?clonebundles? would help resolve the issue. Clonebundles are a
> Mercurial feature that allows the Mercurial server to reuse an existing
> bundle instead of generating a new one for every user. They would basically
> get rid of the cpu issue, the server simply points to the pre-generated
> bundle and does not actually need to do any work. More context from Gregory
> Szorc, who initially implemented this feature:
> https://gregoryszorc.com/blog/2015/05/29/faster-cloning-from-hg.mozilla.org-with-server-provided-bundles/
> 
> 
> 
> To use clonebundles for a repository (let me know if there?s anything
> missing, I have been known to make mistakes):
> 
>    1. Make sure you?re running a fairly recent Mercurial version.
>    2. Put up a cronjob that generates weekly bundles of your repository (hg
>    bundle ?all /path/to/gmp.bundle).
>    3. Make sure the bundle is placed in a location accessible to clients,
>    for example: https://gmplib.org/clonebundles/gmp.bundle.
>    4. In your repository, create a file .hg/clonebundles.manifest that
>    refers to the bundle.
>    To check what the ?BUNDLESPEC? parameter should be, you can run: hg
>    debugbundle ?spec /path/to/gmp.bundle
>    Contents could for example be:
>    https://gmplib.org/clonebundles/gmp.bundle BUNDLESPEC=bzip2-v2
>    5. In your hgrc (either $HOME/.hgrc or /path/to/repository/.hg/hgrc),
>    you will need to enable clonebundles:
>    [extensions]
>    clonebundles =
> 
> 
> The next version of Mercurial (6.5) will contain a feature (implemented by
> Mercurial developer Pierre-Yves David) to auto-generate clonebundles when
> changes are pushed to a repository. This means you don't need the above
> steps, you can purely use configuration options, as described in the help
> of the clonebundles extension:
> https://foss.heptapod.net/mercurial/mercurial-devel/-/blob/branch/default/hgext/clonebundles.py
> 
> Some additional notes:
> 1. Clonebundles reduce server load massively, in exchange for additional
> disk space. Make sure you have enough if you want to enable this.
> 2. Mercurial clients don't need to do anything to enable this feature, it
> only needs to be enabled on the server side.
> 3. I think on the Mercurial development side, we should try to make this
> feature more visible/known, as it could make the situation a lot better for
> people hosting Mercurial repositories.

I think it would catch 99% of the cases when mercurial would cache
the bundle from the "HEAD" (I don't know mercurial or bundles aka
whether each branch or revision would create a different bundle
than the complete repository state) and do that on-demand, so not
create a bundle for each commit.

Richard.

-- 
Richard Biener <rguenther at suse.de>
SUSE Software Solutions Germany GmbH, Frankenstrasse 146, 90461 Nuernberg,
Germany; GF: Ivo Totev, Andrew Myers, Andrew McDonald, Boudien Moerman;
HRB 36809 (AG Nuernberg)