Launchpad itself

Unable to build Chromium

Asked by Jhonny Oliveira

Hi!

I'm unable to build Chromium in Launchpad: https://launchpad.net/~xtradeb/+archive/ubuntu/test/+build/23520766.

I think this might be related to lack of memory during the linking process. I used to have the same problem before I increased it. At the moment, I use 28GB of RAM and roughly the same amount of swap. Not sure exactly how much I need.

Can you please help?

Thank you!
Jhonny Oliveira

Question information

Language:
English Edit question
Status:
Solved
For:
Launchpad itself Edit question
Assignee:
Colin Watson Edit question
Solved by:
Jhonny Oliveira
Solved:
Last query:
Last reply:
Revision history for this message
Paride Legovini (paride) said : 		#1

By chance I recently tried to compile the same version of Chromium in a PPA and hit the same issue, which indeed looks like an OOM kill:

https://launchpadlibrarian.net/594681889/buildlog_ubuntu-jammy-amd64.chromium_100.0.4896.60-1_BUILDING.txt.gz

Revision history for this message
Paride Legovini (paride) said : 		#2

This may be a duplicate of: https://answers.launchpad.net/launchpad/+question/694428

Revision history for this message
Jürgen Gmach (jugmac00) said : 		#3

Thanks for your report. I think you are right - but I'll let my colleague have another look.

Revision history for this message
Colin Watson (cjwatson) said : 		#4

It may indeed be related to the last point on that list. I don't currently have any way to speed up resolution of that, though.

Revision history for this message
Launchpad Janitor (janitor) said : 		#5

This question was expired because it remained in the 'Open' state without activity for the last 15 days.

Revision history for this message
Jhonny Oliveira (jhonny-oliveira) said (last edit ): 		#6

This is still not solved.

Revision history for this message
Jhonny Oliveira (jhonny-oliveira) said : 		#7

Is there any progress on this matter?

Revision history for this message
Jhonny Oliveira (jhonny-oliveira) said : 		#8

I removed the package version associated with the reported problem. New failures can be observed here:

https://launchpad.net/~xtradeb/+archive/ubuntu/apps/+packages?field.name_filter=chromium&field.status_filter=published

Revision history for this message
Olivier Duclos (odc) said : 		#9

Bumping since this is still an issue.

How are the build servers configured for snapcraft? Chromium has no problem building there.

Also, since the Launchpad servers allocate 16GB for amd64 now, could this be a timeout issue? Chromium could need 12h easy.

Revision history for this message
Daniel Richard G. (skunk) said : 		#10

Hi Olivier, I may be able to offer some information.

The latest couple of Jhonny's Chromium builds failed at just under nine and ten hours, respectively. If a hard time limit were in play, I'd expect to see roughly the same run time for both.

I think it's just a memory issue, with the system's OOM killer striking the final link operation.

Not long ago, I ran an instrumented build of Chromium on a work system with memory in the three-digit gigabyte range. I wrapped each linker invocation with "time --format=%M" to indicate how much memory they needed. Here are the largest of those numbers from the build log:

    [54207/54831] LINK ./mksnapshot
    Max RSS: 4987640 KB

    [54815/54831] LINK ./v8_context_snapshot_generator
    Max RSS: 14971980 KB

    [54830/54831] LINK ./chrome
    Max RSS: 32880380 KB

    [54831/54831] LINK ./content_shell
    Max RSS: 24055196 KB

It seems like 16 GB is not sufficient. Given Jhonny's mention of succeeding with 28 GB, I think a request for a 32 GB builder would be reasonable for this PPA.

Revision history for this message
Colin Watson (cjwatson) said : 		#11

Olivier: The build servers for snapcraft.io builds are literally the same as the build servers used by Launchpad - snapcraft.io uses Launchpad as its backend. There is no difference there.

We do not currently have a way to provision builders with different amounts of RAM - the provisioning is the same across the entire region. We did make some builder affinity arrangements for another project recently that might make it more practical to do this in future, though we'd still need to set up the database bits for source packages (the project in question was for a different build type).

Revision history for this message
Colin Watson (cjwatson) said : 		#12

... however, if Chromium can build in snapcraft.io's build system as Olivier indicates, then it's clearly possible to do this in Launchpad somehow. You just have to work out what the snap build process is doing differently, perhaps starting from https://launchpad.net/~chromium-team/+snap/chromium-snap-from-source-stable.

Revision history for this message
Daniel Richard G. (skunk) said : 		#13

Thanks Colin, here are my observations in comparing the two builds:

* The snapcraft amd64 build completed in ~6 hours; xtradeb's was killed after nearly 9.

* The snapcraft build uses an lxc backend; xtradeb's uses chroot.

* The snapcraft build (of Chromium proper) is driven by buildsnap instead of dpkg-buildpackage.

* The snapcraft build appears to be downloading a custom build of Clang from Google (search for "chromium-browser-clang" in the log) instead of using the packaged one from Ubuntu.

I suspect that last point explains most of the difference. I'll see if I can try another test build, with Google's Clang shoved in there. But at the same time, actually using it for PPA packages would flout the need for a reproducible build.

Revision history for this message
Colin Watson (cjwatson) said : 		#14

Perhaps package chromium-browser-clang and then use it as a build-dependency?

Revision history for this message
Olivier Duclos (odc) said : 		#15

Thanks a lot for the detective work!

By the way, this PPA https://launchpad.net/~saiarcot895/+archive/ubuntu/chromium-dev managed to build the latest Chromium for amd64, apparently without using a special LLVM from what I see on his Github repo.

Revision history for this message
Jhonny Oliveira (jhonny-oliveira) said : 		#16

Hi!

From what understood, saiarcot895 uses gold instead of lld. His package structure is also bit different than mine, which is a straight forward rebuild of the Debian one. I never managed to make gold work with it. I guess the problem lies between the different patches and excluded files.

Packaging and using chromium-browser-clang could work, however, I'm afraid this package might cause conflicts with other packages. And tweaking the build recipe, makes me deviate from Debian.

I currently maintain around 80 packages and either solution seems to require considerable amount of additional learning and work. :-(

Revision history for this message
Daniel Richard G. (skunk) said : 		#17

Yes, saiarcot895's build appears to be a light tweak of the Ubuntu bionic source (which still has an un-snap-ified chromium-browser package). As general support for bionic ends this April, building the Debian package is the better long-term solution IMO.

I also share the concern about deviating from Debian. While this is likely to be unavoidable to some extent given the huge dependency footprint, any changes we make will be subject to careful review by users who (rightfully) put less trust in us as individuals than big orgs like Debian and Ubuntu. And the harder we make that review, the less suitable the resulting packages will be for widespread use.

Does anyone have an idea what is so different about Google's build of Clang? I doubt it has a big chunk of proprietary code rolled into it; it was probably just built with a different configuration/defaults. (Maybe it uses the gold linker.) If we could get a similar effect just by changing up the build flags, that would help a lot.

Revision history for this message
Jhonny Oliveira (jhonny-oliveira) said : 		#18

Hi!

I did a dirty hack and used chromium-browser-clang. Sadly, there was no improvement: https://launchpad.net/~xtradeb/+archive/ubuntu/test/+packages

You can find the log of a successful build and the time results below. From there, it seems that the build with the hack alone needed around 24GB. I believe this to be pretty much the same as the build without the hack.

https://drive.google.com/file/d/1DG1AmiKOShGLPgvQPqA3ZVpQtLhWiJ4r/view?usp=share_link
https://drive.google.com/file/d/1iRjYN1mJMQURJn41_CO85tOttuZ2VKbx/view?usp=share_link

@Colin, it would be really nice if you could pump up those resources.

Revision history for this message
Daniel Richard G. (skunk) said (last edit ): 		#19

Gentlemen, I believe I've made a breakthrough.

This past weekend, I had some spare time, and the big machine at work was free, so I ran a number of test builds. Here is an overview:

(ETA: Apologies, I don't know how to render a table properly here)

Build name Chromium source package name + version Source size
------------ ---------------------------------------------------- -----------
bullseye chromium_111.0.5563.64-1~deb11u1 654100529
bookworm chromium_110.0.5481.177-1 642331523
bionic chromium-browser_110.0.5481.100-0ubuntu0.18.04.1 1705826200
jammy-bkwm chromium_110.0.5481.177-1 642331523
jammy-saia chromium-browser_111.0.5563.8-0ubuntu1~ppa1~22.04.1 1742299375

The first three use the normal chromium(-browser) source from the respective package repos. The jammy-bkwm build uses the bookworm source, and jammy-saia uses source from the saiarcot895 PPA.

(As an aside, even though the saiarcot895 sources are similar in size to bionic's, a diff between them makes clear that they are significantly diverged)

Here are the stats from running the builds ("env time --verbose dpkg-buildpackage -b"):

Build name Max RSS (kB) Build time
------------ ------------ ----------
bullseye 2420512 51:24.49
bookworm 2347516 50:21.32
bionic 3139368 44:42.34
jammy-bkwm 32990252 4:04:48
jammy-saia 7473408 58:54.84

One of these is not like the others. Unfortunately, it happens to be the one we're trying to get through Launchpad. Most of that extra run time is in the final link step.

So I focused on comparing the bookworm and jammy-bkwm builds. Here are my findings:

* The difference between needing ~2.5 GB RAM vs. 30+ (also 1 hour vs. 4) comes down to LTO. The Debian, bionic, and jammy-saia builds are, apparently, not doing any link-time optimization at all. This makes me wonder if those Chromium builds are perceivably slower than Jhonny's.

* Using gold doesn't help. (1) LTO still requires gobs of RAM, (2) ld.gold randomly segfaults for some reason, and (3) the LLVM gold plugin isn't even installed correctly in jammy. LLVM expects to find it (LLVMgold.so) in /usr/lib/, but it lives in /usr/lib/llvm-14/lib/. Maybe I'm missing something, but I had to make a symlink ("ln -s llvm-14/lib/LLVMgold.so /usr/lib") to get it working.

* All those -ffat-lto-objects warnings in the jammy build are due to an implicit optimize=+lto setting in DEB_BUILD_MAINT_OPTIONS in debian/rules. This is because dpkg's global LTO flags are meant for GCC, not Clang (see https://wiki.ubuntu.com/ToolChain/LTO#Known_issues). The rules file should specify optimize=-lto explicitly. FYI, the actual flags live in /usr/share/perl5/Dpkg/Vendor/Debian.pm .

* The implicit optimize=+lto also puts in -flto=auto everywhere, and this is what causes the jammy-bkwm build to blow up. It doesn't happen on bookworm since, AFAICT, bookworm dpkg builds don't enable LTO by default.

* I can get the jammy build working without LTO (2.4 GB max RSS), but a lack of LTO isn't ideal. With some further tweaking, I successfully enabled Chromium's use_thin_lto option, which results in a not-unreasonable ~10.5 GB max RSS and a hopefully-faster binary. (ThinLTO appears to be a new approach to LTO that is more efficient, and allows for niceties like incremental linking.)

Jhonny, could you try this patch?

--- chromium-111.0.5563.64.orig/debian/rules 2023-03-02 01:24:58.000000000 +0000
+++ chromium-111.0.5563.64/debian/rules 2023-03-14 23:17:36.158871161 +0000
@@ -6,6 +6,9 @@
 # enable all build hardening flags
 export DEB_BUILD_MAINT_OPTIONS=hardening=+all

+# disable debian lto flags to avoid resource exhaustion
+export DEB_BUILD_MAINT_OPTIONS+=optimize=-lto
+
 # indicate that binary targets do not require root
 export DEB_RULES_REQUIRES_ROOT=no

@@ -78,7 +81,7 @@
          treat_warnings_as_errors=false \
          use_qt=false \
          is_cfi=false \
- use_thin_lto=false \
+ use_thin_lto=true \
          chrome_pgo_phase=0 \

 # enabled features
@@ -97,7 +100,7 @@
          use_system_libpng=true \
          use_system_freetype=true \
          use_system_libopenjpeg2=true \
- concurrent_links=1 \
+ $(if, concurrent_links=1) \
          proprietary_codecs=true \
          ffmpeg_branding=\"Chrome\" \
          disable_fieldtrial_testing_config=true \

(The "$(if," is just a way to "comment out" something in a makefile. use_thin_lto=true requires concurrent_links to be unset.)

Revision history for this message
Jhonny Oliveira (jhonny-oliveira) said : 		#20

Dear Daniel,

I'm amazed!!! I confirm it works. Just uploaded to the main PPA (apps).

I'm almost sure that in the beginning I tried all of these options (while still using my computer and the builds were taking ~14h), but I guess I never got them all together or I messed something else in between. Brilliant! Thank you very much!

There is only one last long standing issue I would like to take the opportunity to share.
To build successful, I'm ignoring dwz errors. The errors are below. Do you have any idea?

   dh_dwz -a
        dwz -- debian/chromium-shell/usr/lib/chromium/chromium-shell
        dwz -- debian/chromium-driver/usr/bin/chromedriver
        dwz -- debian/chromium-sandbox/usr/lib/chromium/chrome-sandbox
        dwz -- debian/chromium/usr/lib/chromium/chromium
dwz: debian/chromium-shell/usr/lib/chromium/chromium-shell: Unknown debugging section .debug_str_offsets
dwz: debian/chromium-driver/usr/bin/chromedriver: .debug_info section not present
dh_dwz: error: dwz -- debian/chromium-shell/usr/lib/chromium/chromium-shell returned exit code 1
dwz: debian/chromium-sandbox/usr/lib/chromium/chrome-sandbox: Unknown debugging section .debug_str_offsets
dwz: debian/chromium/usr/lib/chromium/chromium: Unknown debugging section .debug_str_offsets
dh_dwz: error: dwz -- debian/chromium-sandbox/usr/lib/chromium/chrome-sandbox returned exit code 1
dh_dwz: error: dwz -- debian/chromium/usr/lib/chromium/chromium returned exit code 1
        install -d debian/chromium-common/usr/lib/debug/.dwz/x86_64-linux-gnu
        dwz -mdebian/chromium-common/usr/lib/debug/.dwz/x86_64-linux-gnu/chromium-common.debug -M/usr/lib/debug/.dwz/x86_64-linux-gnu/chromium-common.debug -- debian/chromium-common/usr/lib/chromium/chrome_crashpad_handler debian/chromium-common/usr/lib/chromium/libEGL.so debian/chromium-common/usr/lib/chromium/libGLESv2.so
dwz: debian/chromium-common/usr/lib/chromium/chrome_crashpad_handler: Unknown debugging section .debug_str_offsets
dwz: debian/chromium-common/usr/lib/chromium/libEGL.so: Unknown debugging section .debug_str_offsets
dwz: debian/chromium-common/usr/lib/chromium/libGLESv2.so: Unknown debugging section .debug_str_offsets
dwz: Too few files for multifile optimization
dh_dwz: error: dwz -mdebian/chromium-common/usr/lib/debug/.dwz/x86_64-linux-gnu/chromium-common.debug -M/usr/lib/debug/.dwz/x86_64-linux-gnu/chromium-common.debug -- debian/chromium-common/usr/lib/chromium/chrome_crashpad_handler debian/chromium-common/usr/lib/chromium/libEGL.so debian/chromium-common/usr/lib/chromium/libGLESv2.so returned exit code 1
dh_dwz: error: Aborting due to earlier error
make: *** [debian/rules:120: binary] Error 25

Revision history for this message
Daniel Richard G. (skunk) said : 		#21

Fantastic :-) Glad to see this problem finally licked!

(I've submitted https://bugs.debian.org/1033015 to hopefully get the optimize=-lto tweak upstreamed, along with a fix for an incidental LDFLAGS issue)

That dh_dwz error is odd---I don't see a mention of "dwz" in *any* of the build logs I've produced these past few days. Are you getting those with a current build?

I'm happy to take a whack at it, but at present I can't even reproduce the issue. I'm not quite doing the full sbuild rigmarole, but am starting with a debootstrap'd jammy chroot, installing dpkg-dev, then the build-deps. How are you getting the error?

Revision history for this message
Jhonny Oliveira (jhonny-oliveira) said : 		#22

dh_dwz is part of the debhelper sequence. One of the last steps.

You can see that it use to work fine in Ubuntu Focal: https://launchpadlibrarian.net/539710947/buildlog_ubuntu-focal-amd64.chromium_90.0.4430.212-1~xtradeb1_BUILDING.txt.gz .

But it is not working in recent versions. You will be able to see it failing (the errors I sent) in the new build, as soon as it finishes.

Revision history for this message
Daniel Richard G. (skunk) said : 		#23

All right, I figured out what was going on with dh_dwz.

First, the reason why I wasn't seeing this is because I was building the original Debian package, which uses debhelper compat level 11. The package that's building here, however, has this bumped to 13. At that level, dh_dwz is included in the debhelper sequence. Once I likewise bumped the debhelper compat to 13, I was able to reproduce the error.

The dwz error has to do with the symbol_level build parameter, which controls debug symbols. It is set to 0 in the build, which results in -g0 being passed in (no debug symbols). The dwz utility does not like this, though I'm not clear on the exact reason why. (More on this later.)

The symbol_level parameter can also be set to 1 (minimal symbols) or 2 (full symbols). If I set it to 2, the following error results:

        dwz -mdebian/chromium-common/usr/lib/debug/.dwz/x86_64-linux-gnu/chromium-common.debug -M/usr/lib/debug/.dwz/x86_64-linux-gnu/chromium-common.debug -- debian/chromium-common/usr/lib/chromium/chrome_crashpad_handler debian/chromium-common/usr/lib/chromium/libEGL.so debian/chromium-common/usr/lib/chromium/libGLESv2.so
dwz: debian/chromium/usr/lib/chromium/chromium: Too many DIEs, not optimizing
dh_dwz: error: dwz -- debian/chromium/usr/lib/chromium/chromium returned exit code 1
dwz: debian/chromium-shell/usr/lib/chromium/chromium-shell: Too many DIEs, not optimizing
dh_dwz: error: dwz -- debian/chromium-shell/usr/lib/chromium/chromium-shell returned exit code 1
        objcopy --compress-debug-sections debian/chromium-common/usr/lib/debug/.dwz/x86_64-linux-gnu/chromium-common.debug
dh_dwz: error: Aborting due to earlier error
make: *** [debian/rules:112: binary] Error 25
dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2

If I set it to 1, however, the dh_dwz invocation (and the build as a whole) succeeds:

   dh_dwz -a
        dwz -- debian/chromium-driver/usr/bin/chromedriver
        dwz -- debian/chromium-sandbox/usr/lib/chromium/chrome-sandbox
        dwz -- debian/chromium-shell/usr/lib/chromium/chromium-shell
dwz: debian/chromium-driver/usr/bin/chromedriver: .debug_info section not present
        dwz -- debian/chromium/usr/lib/chromium/chromium
        install -d debian/chromium-common/usr/lib/debug/.dwz/x86_64-linux-gnu
        dwz -mdebian/chromium-common/usr/lib/debug/.dwz/x86_64-linux-gnu/chromium-common.debug -M/usr/lib/debug/.dwz/x86_64-linux-gnu/chromium-common.debug -- debian/chromium-common/usr/lib/chromium/chrome_crashpad_handler debian/chromium-common/usr/lib/chromium/libEGL.so debian/chromium-common/usr/lib/chromium/libGLESv2.so

(It still gives a ".debug_info section not present" message, which I had previously assumed was fatal. No idea why it's not complaining about .debug_str_offsets anymore, either.)

An alternative workaround is to add

    export DEB_BUILD_OPTIONS+=nostrip

which causes the dh_dwz invocation to be skipped altogether. However, it also results in a bazillion "Unrecognized form" warnings from objdump(1) and readelf(1).

So symbol_level=1 seems to be the way to go. It's already standard practice for dpkg builds to enable debug info, which is then split off into debug packages at the end.

Revision history for this message
Jhonny Oliveira (jhonny-oliveira) said : 		#24

You are awesome!

From the documentation I went through, symbol_level is usually set to 0 to reduce memory consumption. Since you cracked the previous problem, I don't think this is an issue anymore.

Based on your research, we have three options:
 - go back to compatibility level 11 and stay as close as possible to Debian ( I should have noticed this :-();
 - set symbol_level=1. Since now, we have a master around, I will test it and enable it next release;
 - disabling dh_dwz optimization should be a last resort measure in production binaries.

Once again, thank you very much!

Revision history for this message
Daniel Richard G. (skunk) said : 		#25

Happy to help :-)

I tried another test build with symbol_level=1 and use_thin_lto=true, and the max RSS was 17.9 GB. So symbol_level definitely affects the RAM usage. (The dbgsym *.ddeb files coming out of the build are significantly bigger too, as expected. Presumably, they will be more useful if anyone ever tries debugging this thing.)

IMO, it's not ideal to require an additional 7 GB RAM from the Launchpad builders just to avoid a minor build issue, so reducing the debhelper compat level may be a better solution. Was there a reason to bump it to 13?

Absent that, I'd say, better to stick with what Debian does. But I'll send them a heads-up about the dh_dwz issue, as they're bound to run into that sooner or later.

Revision history for this message
Jhonny Oliveira (jhonny-oliveira) said (last edit ): 		#26

There were two reason to bump the compat level to 13:
 - the greatest and the latest;
 - never had an issue with it before. :-)

I also tested it and observed the memory increase.

Better safe than sorry! Next release, I will downgrade to 11 and reset symbol_level to 0.

Thank you!
Cheers!