Launchpad itself

recipe builds fails randomly due to GIT fetch

Asked by Jasem Mutlaq

My recipes builds are failing randomly for several projects based on GIT.

It fails like 75% of the time.. sometimes when I retrigger the build, it goes through and other times it fails.

This is one example for a recipe:

# git-build-recipe format 0.4 deb-version {debupstream}~{time}
lp:~mutlaqja/+git/dummy master
nest-part src lp:libindi 3rdparty/indi-ffmv indi-ffmv
nest-part cmake lp:libindi cmake_modules indi-ffmv/cmake_modules
nest-part pack lp:libindi debian/indi-ffmv debian

Here are some logs:

https://code.launchpad.net/~mutlaqja/+archive/ubuntu/ppa/+recipebuild/2007855/+files/buildlog.txt.gz
https://code.launchpad.net/~mutlaqja/+archive/ubuntu/ppa/+recipebuild/2007853/+files/buildlog.txt.gz
https://code.launchpad.net/~mutlaqja/+archive/ubuntu/ppa/+recipebuild/2007851/+files/buildlog.txt.gz
https://code.launchpad.net/~mutlaqja/+archive/ubuntu/ppa/+recipebuild/2007876/+files/buildlog.txt.gz
https://code.launchpad.net/~mutlaqja/+archive/ubuntu/ppa/+recipebuild/2007877/+files/buildlog.txt.gz
https://code.launchpad.net/~mutlaqja/+archive/ubuntu/ppa/+recipebuild/2007880/+files/buildlog.txt.gz
https://code.launchpad.net/~mutlaqja/+archive/ubuntu/ppa/+recipebuild/2007892/+files/buildlog.txt.gz

and more....

Is there a reason why so many fail?? It appears totally random.

Some of the projects:

https://code.launchpad.net/~mutlaqja/+recipe/indi-apogee-stable
https://code.launchpad.net/~mutlaqja/+recipe/indi-nexstarevo-stable
https://code.launchpad.net/~mutlaqja/+recipe/indi-gpsnmea-stable

Question information

Language:
English Edit question
Status:
Solved
For:
Launchpad itself Edit question
Assignee:
No assignee Edit question
Solved by:
Jasem Mutlaq
Solved:
Last query:
Last reply:
Revision history for this message
Colin Watson (cjwatson) said : 		#1

This is happening because a very large number of your recipe builds are dispatched almost simultaneously, and the effect is to overload the git hosting service about once a day (not entirely due to your recipes but pretty close). Sometimes I notice and cancel the builds, and sometimes I don't and it sends the git service into a load spiral that takes some time to recover from. Either way you'll see somewhat random client-side failures.

We're working on scaling up the git service as a priority, although at this point it may not be done until the new year.

Revision history for this message
Jasem Mutlaq (mutlaqja) said : 		#2

Alright thanks, I will try to build one at a time for those that are built-on-request. However, I also have nightlies that I have no control on when they start.

Looking forward to the Git service scaling, thanks!

Revision history for this message
Colin Watson (cjwatson) said : 		#3

You should find this to be working rather better for you now. We still have more scaling work in progress, but simply doubling the RAM has helped a fair bit.