Requests from Microsoft IP Addresses

[Mathias De Maré]

Hello,



I’d like to give a little bit of additional information about a Mercurial
feature that might provide some help here. My apologies if this is old news
and doesn't add anything useful.

(Small context: I’m a Mercurial contributor and I maintain a set of
repositories at our company that are cloned a lot at the same point in time)



By default, when you do a clone, Mercurial will take the repository,
compress it and send it across the wire to the client.

I simulated similar behaviour for the gmp repository by creating a bundle:

   - it takes about 8 seconds (so approximately 1 core occupied for < 8
   seconds, as there is a bit of overhead involved)
   - it generates a bundle of about 14 MB

That’s not much, but it of course becomes problematic when there are 700
clones at nearly the exact same moment, like you experienced.



I cannot help in reducing the bandwidth, but assuming the cpu usage is the
problem, ‘clonebundles’ would help resolve the issue. Clonebundles are a
Mercurial feature that allows the Mercurial server to reuse an existing
bundle instead of generating a new one for every user. They would basically
get rid of the cpu issue, the server simply points to the pre-generated
bundle and does not actually need to do any work. More context from Gregory
Szorc, who initially implemented this feature:
https://gregoryszorc.com/blog/2015/05/29/faster-cloning-from-hg.mozilla.org-with-server-provided-bundles/



To use clonebundles for a repository (let me know if there’s anything
missing, I have been known to make mistakes):

   1. Make sure you’re running a fairly recent Mercurial version.
   2. Put up a cronjob that generates weekly bundles of your repository (hg
   bundle –all /path/to/gmp.bundle).
   3. Make sure the bundle is placed in a location accessible to clients,
   for example: https://gmplib.org/clonebundles/gmp.bundle.
   4. In your repository, create a file .hg/clonebundles.manifest that
   refers to the bundle.
   To check what the ‘BUNDLESPEC’ parameter should be, you can run: hg
   debugbundle –spec /path/to/gmp.bundle
   Contents could for example be:
   https://gmplib.org/clonebundles/gmp.bundle BUNDLESPEC=bzip2-v2
   5. In your hgrc (either $HOME/.hgrc or /path/to/repository/.hg/hgrc),
   you will need to enable clonebundles:
   [extensions]
   clonebundles =


The next version of Mercurial (6.5) will contain a feature (implemented by
Mercurial developer Pierre-Yves David) to auto-generate clonebundles when
changes are pushed to a repository. This means you don't need the above
steps, you can purely use configuration options, as described in the help
of the clonebundles extension:
https://foss.heptapod.net/mercurial/mercurial-devel/-/blob/branch/default/hgext/clonebundles.py

Some additional notes:
1. Clonebundles reduce server load massively, in exchange for additional
disk space. Make sure you have enough if you want to enable this.
2. Mercurial clients don't need to do anything to enable this feature, it
only needs to be enabled on the server side.
3. I think on the Mercurial development side, we should try to make this
feature more visible/known, as it could make the situation a lot better for
people hosting Mercurial repositories.

Greetings,
Mathias