@odlp Awesome! Thank you so much for giving it a try, this is really good news. Look for an official universal-darwin gem in RC4!

related to #2079

See #2075 and #2079 for context. This branch attempts to generate a universal-darwin gem, following the patterns in the grpc gem.

@odlp and others running Darwin 18 or earlier: please jump over to #2079 where I've got a gem file I need tested. Thanks!

@odlp - I've created a universal-darwin gem and put it in a public storage bucket here:

https://storage.googleapis.com/nokogiri-org-concourse/test-gems/nokogiri-1.11.0.rc3-universal-darwin.gem

Can I ask that you try this on your darwin18 system by:

• downloading the file from there,
• check the md5 checksum is f8eb952cdad60740e6d411b40241aade,
• run gem install --local path/to/nokogiri-1.11.0.rc3-universal-darwin.gem
• run nokogiri -v

?

Did some poking around this morning. I noticed that the grpc gem only ships a universal-darwin gem, and that this Just Works™ on my mac running either ruby universal.x86_64-darwin19 or x86-64-darwin19. This works because of this logic in Gem::Platform.

Lesson: we don't need to ship individual gem files for each version of darwin.

Next step: let's take a look at how grpc is built. Optionally consider setting up osxcross in a linux container.

See additional context from user help request at https://gitter.im/sparklemotion/nokogiri?at=5f5a485bf969413294d84734

After migrating ruby application to centos 8, application won't start with following error : .../vendor/bundle/ruby/2.6.0/gems/nokogiri-1.10.9/lib/nokogiri.rb:32:in `require': libgcrypt.so.11: cannot open shared object file: No such file or directory - .../vendor/bundle/ruby/2.6.0/gems/nokogiri-1.10.9/lib/nokogiri/nokogiri.so (LoadError)

Libgcrypt is installed on system but libgcrypt.so.11 is not available on centos 8

To Reproduce

Install rails application with nokogiri on Centos 8.2

Environment

[root@...]# nokogiri -v
Traceback (most recent call last):
    5: from .../vendor/bundle/bin/nokogiri:29:in `<main>'
    4: from .../vendor/bundle/bin/nokogiri:29:in `load'
    3: from .../vendor/bundle/ruby/2.6.0/gems/nokogiri-1.10.9/bin/nokogiri:6:in `<top (required)>'
    2: from .../vendor/bundle/ruby/2.6.0/gems/nokogiri-1.10.9/bin/nokogiri:6:in `require'
    1: from .../vendor/bundle/ruby/2.6.0/gems/nokogiri-1.10.9/lib/nokogiri.rb:30:in `<top (required)>'
.../vendor/bundle/ruby/2.6.0/gems/nokogiri-1.10.9/lib/nokogiri.rb:30:in `require': cannot load such file -- nokogiri/2.6/nokogiri (LoadError)
    6: from .../vendor/bundle/bin/nokogiri:29:in `<main>'
    5: from .../vendor/bundle/bin/nokogiri:29:in `load'
    4: from .../vendor/bundle/ruby/2.6.0/gems/nokogiri-1.10.9/bin/nokogiri:6:in `<top (required)>'
    3: from .../vendor/bundle/ruby/2.6.0/gems/nokogiri-1.10.9/bin/nokogiri:6:in `require'
    2: from .../vendor/bundle/ruby/2.6.0/gems/nokogiri-1.10.9/lib/nokogiri.rb:28:in `<top (required)>'
    1: from .../vendor/bundle/ruby/2.6.0/gems/nokogiri-1.10.9/lib/nokogiri.rb:32:in `rescue in <top (required)>'
.../vendor/bundle/ruby/2.6.0/gems/nokogiri-1.10.9/lib/nokogiri.rb:32:in `require': libgcrypt.so.11: cannot open shared object file: No such file or directory - .../vendor/bundle/ruby/2.6.0/gems/nokogiri-1.10.9/lib/nokogiri/nokogiri.so (LoadError)

Closing because OP hasn't responded. Please feel free to comment if you'd like to re-open the discussion!

Aaaand a more complete Dockerfile using osxcross at https://github.com/appPlant/docker-images/blob/mruby-cli/Dockerfile

tagging @twk3 so he can follow along as the original reporter

@twk3 I've created #2081 to investigate the Centos6 issue you reported above. Thanks again!

(Originally mentioned at https://github.com/sparklemotion/nokogiri/issues/2075#issuecomment-689063142)

On Centos6, running a version of Ruby compiled like this, we see the following error message when loading the nokogiri.so shared library:

/lib64/libc.so.6: version `GLIBC_2.14' not found (required by /usr/local/lib/ruby/gems/2.6.0/gems/nokogiri-1.11.0.rc3-x86_64-linux/lib/nokogiri/2.6/nokogiri.so)

I'm not deeply familiar with our glibc requirements, but if we go back to the early days of Linux native gems, in #1571, @lars mentions that we need GLIBC 2.17 or later (and we check for this here).

It looks like CentOS/6.10 is based on glibc 2.12 which might explain why this isn't working -- but the error message is a little confusing if this is the case. @larskanis can you take a look and confirm my understanding?

To Reproduce

host $ docker pull registry.gitlab.com/gitlab-org/gitlab-omnibus-builder/centos_6:0.0.72
...

host $ docker run -t registry.gitlab.com/gitlab-org/gitlab-omnibus-builder/centos_6:0.0.72 /bin/bash

guest # gem install --prerelease nokogiri
Fetching: nokogiri-1.11.0.rc3-x86_64-linux.gem (100%)
Nokogiri is built with the packaged libraries: libxml2-2.9.10, libxslt-1.1.34, zlib-1.2.11, libiconv-1.15.
Successfully installed nokogiri-1.11.0.rc3-x86_64-linux
Parsing documentation for nokogiri-1.11.0.rc3-x86_64-linux
Installing ri documentation for nokogiri-1.11.0.rc3-x86_64-linux
Done installing documentation for nokogiri after 2 seconds
1 gem installed

guest # nokogiri -v
# nokogiri -v
Traceback (most recent call last):
	7: from /usr/local/bin/nokogiri:23:in `<main>'
	6: from /usr/local/bin/nokogiri:23:in `load'
	5: from /usr/local/lib/ruby/gems/2.6.0/gems/nokogiri-1.11.0.rc3-x86_64-linux/bin/nokogiri:6:in `<top (required)>'
	4: from /usr/local/lib/ruby/site_ruby/2.6.0/rubygems/core_ext/kernel_require.rb:55:in `require'
	3: from /usr/local/lib/ruby/site_ruby/2.6.0/rubygems/core_ext/kernel_require.rb:55:in `require'
	2: from /usr/local/lib/ruby/gems/2.6.0/gems/nokogiri-1.11.0.rc3-x86_64-linux/lib/nokogiri.rb:13:in `<top (required)>'
	1: from /usr/local/lib/ruby/site_ruby/2.6.0/rubygems/core_ext/kernel_require.rb:55:in `require'
/usr/local/lib/ruby/site_ruby/2.6.0/rubygems/core_ext/kernel_require.rb:55:in `require': /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /usr/local/lib/ruby/gems/2.6.0/gems/nokogiri-1.11.0.rc3-x86_64-linux/lib/nokogiri/2.6/nokogiri.so) - /usr/local/lib/ruby/gems/2.6.0/gems/nokogiri-1.11.0.rc3-x86_64-linux/lib/nokogiri/2.6/nokogiri.so (LoadError)
	8: from /usr/local/bin/nokogiri:23:in `<main>'
	7: from /usr/local/bin/nokogiri:23:in `load'
	6: from /usr/local/lib/ruby/gems/2.6.0/gems/nokogiri-1.11.0.rc3-x86_64-linux/bin/nokogiri:6:in `<top (required)>'
	5: from /usr/local/lib/ruby/site_ruby/2.6.0/rubygems/core_ext/kernel_require.rb:55:in `require'
	4: from /usr/local/lib/ruby/site_ruby/2.6.0/rubygems/core_ext/kernel_require.rb:55:in `require'
	3: from /usr/local/lib/ruby/gems/2.6.0/gems/nokogiri-1.11.0.rc3-x86_64-linux/lib/nokogiri.rb:11:in `<top (required)>'
	2: from /usr/local/lib/ruby/gems/2.6.0/gems/nokogiri-1.11.0.rc3-x86_64-linux/lib/nokogiri.rb:15:in `rescue in <top (required)>'
	1: from /usr/local/lib/ruby/site_ruby/2.6.0/rubygems/core_ext/kernel_require.rb:55:in `require'
/usr/local/lib/ruby/site_ruby/2.6.0/rubygems/core_ext/kernel_require.rb:55:in `require': libruby.so.2.5: cannot open shared object file: No such file or directory - /usr/local/lib/ruby/gems/2.6.0/gems/nokogiri-1.11.0.rc3-x86_64-linux/lib/nokogiri/nokogiri.so (LoadError)

Example of how this was used in the past in a docker container: https://github.com/hone/mruby-cli-docker/blob/master/Dockerfile which is used by the https://github.com/hone/mruby-cli project.

@luislavena I'm going to use osxcross to experiment with creating a rake-compiler-dock-compatible image for OSX!

It looks like this might be helpful, too: https://github.com/tpoechtrager/osxcross which is a cross-compiler toolchain for OSX.

OK - I ran a git bisect and ba16682aae2cbd42c196bd8afdfcfe8a5d82fbdb introduced this regression. Tagging @jvshahid for his awareness :two_hearts:

nokogiri $ git bisect good
ba16682aae2cbd42c196bd8afdfcfe8a5d82fbdb is the first bad commit
commit ba16682aae2cbd42c196bd8afdfcfe8a5d82fbdb
Author: John Shahid <[email protected]>
Date:   Thu Apr 18 13:58:18 2019 -0400

    Split setInputSource into setIOInputSource and setStringInputSource
    
    This commit also consolidates some of the encoding handling logic that was
    repeated in multiple places.

 ext/java/nokogiri/HtmlDocument.java              |  25 +++---
 ext/java/nokogiri/HtmlSaxParserContext.java      | 100 ++++++-----------------
 ext/java/nokogiri/XmlDocument.java               |  33 +++-----
 ext/java/nokogiri/XmlSaxParserContext.java       |  14 ++--
 ext/java/nokogiri/internals/NokogiriHelpers.java |  12 +++
 ext/java/nokogiri/internals/ParserContext.java   |  71 ++++++----------
 6 files changed, 94 insertions(+), 161 deletions(-)

@twk3 How did you compile/build/install that version of Ruby? Looks like yum install ruby on centos6 installs Ruby 1.8.7 (!). I'd like to reproduce what you're seeing.

@twk3 Interesting! Thank you for reporting this, I'll take a look.

Thank you for submitting this! Fantastic executable bug report. I appreciate it.

@thbar Thanks for testing this. If you need to, please open a new GH issue for the jruby problems -- I don't think they're going to be related to the native gem work for Linux and OSX. I'd like to know more about what you're seeing, though!

Note that @drbrain says: "Travis has older Xcode for linking on older Darwin, I use it for building work stuff for MacOS" at https://twitter.com/drbrain/status/1303369620099751936

Nokogiri v1.11.0.rc3 only supports OSX x86_64-darwin19 platform.

At least one commented at #2075 has asked about x86_64-darwin18. If that's a common platform we should consider shipping that, as well.

I'll be honest and say that my inability to virtualize OSX is a real challenge here. I have a work laptop that's darwin19, and I have an 2009 mac mini that might have darwin18 on it (which looks like it's x86_64).

When we ship precompiled native gems, we no longer need mini_portile2 as an install-time dependency.

Previously @larskanis and I did some work on this, resulting in:

• https://github.com/rake-compiler/rake-compiler/pull/171
• cebf400 (workaround)
• 33e5b42 (workaround)
• 94ecf99 (workaround)

The workarounds are brittle; this didn't seem to work for the Darwin native gem. Let's figure out a way to do this reliably.

Related to #2075.

Because the gem is precompiled, there's no need to also ship the C source and header files.

Package the gem without the C code.

Related to #2075.

When building a native Linux or OSX gem, we package a fifth nokogiri.so or nokogiri.bundle that appears to be generated by the "host" ruby (not "cross-compiled").

This additional shared library file should not be present.

Related to #2075

@odlp - Thanks for giving it a try! I'm not building darwin18 gems at this point, because I don't have a system available to do so. Only darwin19 is supported right now.

Hello! If you're arriving here to give feedback on the v1.11.0 Release Candidates, please make sure to:

• tell us which linux distro you're using!
• (If you can) a docker image in which we can reproduce the failure you're seeing!

Thanks so much for using Nokogiri and for giving us feedback on this experiment!

See #1571 for the PR from @larskanis that enables building precompiled libraries for linux.

This issue is for discussion on how to ship those precompiled libraries in a gem for linux users in a way that won't break the installation experience for people.

Some things that we'll need to figure out were mentioned in #1571 by @larskanis:

1. Bundler isn't clever at selection of unsupported ruby versions. See https://github.com/bundler/bundler/pull/6247 . This requires users to explicit opt-out of binary gems, when they want to run nokogiri on ruby-trunk.

2. There is a known issue on Alpine Linux: https://github.com/rake-compiler/rake-compiler-dock/issues/20 . I did a manual test on the alpine docker image and it reproducible fails at Nokogiri::XML::Reader#namespaces somewhere within the tests: https://gist.github.com/larskanis/94920901e6466c9383360ee1346f9f34 . I didn't analyze the reason for the segfault so far, but I think we should before the release.

And he later mentioned:

Issue 1 in the above comment will probably finally be fixed by the next bundler release. See rubygems/bundler#7522 . This way bundler will select the platform=ruby gem automatically, if fat binary gem doesn't fit to the ruby release.

I think we'll probably also be able to fix / work around issue 2, so that I hope we'll be able to ship binary releases some day.

Finally, I'd like for VersionInfo to contain some indication that the user is using precompiled libraries. A patch that worked at one point on the #1571 branch (but now needs to be rewritten) is:

From 20f69908732b50258be0ee18e1d78b841d462dc4 Mon Sep 17 00:00:00 2001
From: Mike Dalessio <[email protected]>
Date: Sun, 13 Jan 2019 01:15:06 -0500
Subject: [PATCH] `nokogiri -v` emits info about precompiled binaries

It may be useful information to be able to tell if the libraries were
compiled on installation, or whether they were precompiled in a fat
binary native gem.
---
 Rakefile                | 3 +++
 ext/nokogiri/nokogiri.c | 6 ++++++
 lib/nokogiri/version.rb | 6 ++++++
 3 files changed, 15 insertions(+)

diff --git a/Rakefile b/Rakefile
index 5ef937d..72edb7e 100644
--- a/Rakefile
+++ b/Rakefile
@@ -383,6 +383,9 @@ task :cross do
     raise "rake-compiler has not installed any cross rubies. Use rake-compiler-dock or 'rake gem:windows' for building binary windows gems."
   end
 
+  ENV['CFLAGS'] ||= ""
+  ENV['CFLAGS'] += " -DNOKOGIRI_USE_PRECOMPILED_NATIVE"
+
   CROSS_RUBIES.each do |cross_ruby|
     task "tmp/#{cross_ruby.platform}/nokogiri/#{cross_ruby.ver}/nokogiri.so" do |t|
       # To reduce the gem file size strip mingw32 dlls before packaging
diff --git a/ext/nokogiri/nokogiri.c b/ext/nokogiri/nokogiri.c
index 740b30d..5b32500 100644
--- a/ext/nokogiri/nokogiri.c
+++ b/ext/nokogiri/nokogiri.c
@@ -104,6 +104,12 @@ void Init_nokogiri()
   rb_const_set(mNokogiri, rb_intern("NOKOGIRI_LIBXSLT_PATCHES"), Qnil);
 #endif
 
+#ifdef NOKOGIRI_USE_PRECOMPILED_NATIVE
+  rb_const_set(mNokogiri, rb_intern("NOKOGIRI_USE_PRECOMPILED_NATIVE"), Qtrue);
+#else
+  rb_const_set(mNokogiri, rb_intern("NOKOGIRI_USE_PRECOMPILED_NATIVE"), Qfalse);
+#endif
+
 #ifdef LIBXML_ICONV_ENABLED
   rb_const_set(mNokogiri, rb_intern("LIBXML_ICONV_ENABLED"), Qtrue);
 #else
diff --git a/lib/nokogiri/version.rb b/lib/nokogiri/version.rb
index 44c076a..c05e1a0 100644
--- a/lib/nokogiri/version.rb
+++ b/lib/nokogiri/version.rb
@@ -46,6 +46,10 @@ module Nokogiri
       NOKOGIRI_USE_PACKAGED_LIBRARIES
     end
 
+    def libxml2_using_precompiled_native?
+      NOKOGIRI_USE_PRECOMPILED_NATIVE
+    end
+
     def warnings
       warnings = []
 
@@ -78,6 +82,7 @@ module Nokogiri
           vi["libxml"] = {}.tap do |libxml|
             if libxml2_using_packaged?
               libxml["source"] = "packaged"
+              libxml["precompiled"] = libxml2_using_precompiled_native?
               libxml["patches"] = NOKOGIRI_LIBXML2_PATCHES
             else
               libxml["source"] = "system"
@@ -89,6 +94,7 @@ module Nokogiri
           vi["libxslt"] = {}.tap do |libxslt|
             if libxml2_using_packaged?
               libxslt["source"] = "packaged"
+              libxslt["precompiled"] = libxml2_using_precompiled_native?
               libxslt["patches"] = NOKOGIRI_LIBXSLT_PATCHES
             else
               libxslt["source"] = "system"
-- 
2.17.1

And tagging @tenderlove for visibility because he had some thoughts on the previous #1571 thread.

Closing this issue. RC3 should be out shortly, and I'm pointing people to https://github.com/sparklemotion/nokogiri/issues/2075 to provide feedback.

This is a placeholder for a future conversation (with myself?) about precompiling Darwin libraries, similarly to how we've shipped precompiled libraries for Windows since ~v1.4 and how we will hopefully ship for Linux in v1.11.0.

cc @larskanis @tenderlove tell me if you think I'm crazy for thinking we can do this.

Here's what I've learned so far

(I'll keep this section up to date as I go along)

Architecture names relevant to us

• RUBY_PLATFORM
  • used to identify the ruby architecture
  • which on my linux system is "x86_64-linux"
  • using OSX system Ruby is "universal.x86_64-darwin19"
  • using OSX homebrew- or rvm-installed Ruby is "x86_64-darwin19"
• Gem::Platform.new(RUBY_PLATFORM)
  • used by rake-compiler to name native tasks
  • on my linux system is "x86_64-linux"
  • using OSX system Ruby is "universal.x86_64-darwin-19" (note the addition of a "-" before the "19")
  • using OSX homebrew- or rvm-installed Ruby is "x86_64-darwin-19" (note the "-" here also)
• Gem::Platform.local
  • used by rubygems to search for an appropriate native platform gem
  • on my linux system is "x86_64-linux"
  • using OSX system Ruby is "universal-darwin-19"
  • using OSX homebrew- or rvm-installed Ruby is "x86_64-darwin-19"

Needed changes to Nokogiri to support OSX

Invoke the properly-named gem task:

diff --git a/tasks/cross-ruby.rb b/tasks/cross-ruby.rb
index 35c098f..1d5c2fd 100644
--- a/tasks/cross-ruby.rb
+++ b/tasks/cross-ruby.rb
@@ -179,7 +181,7 @@ namespace "gem" do
       task "guest" do
         # use Task#invoke because the pkg/*gem task is defined at runtime
         Rake::Task["native:#{plat}"].invoke
-        Rake::Task["pkg/#{HOE.spec.full_name}-#{plat}.gem"].invoke
+        Rake::Task["pkg/#{HOE.spec.full_name}-#{Gem::Platform.new(plat).to_s}.gem"].invoke
       end
     end
   end

Compatibility

If Ruby is configured with --enable-shared, the shared object libraries generated will contain references to libruby (and possibly other libraries like libgmp, libcrypt, and librt).

This means that a shared object library built by a ruby that was configured with --enabled-shared can't be run by a ruby that was configured with --disabled-shared. However, a shared object library built by a ruby that was configured with --disable-shared seems to run fine in a ruby configured with --enable-shared.

It's possible to hack the shared object files to get around this. See here for how rake-compiler-dock gets around this:

• https://github.com/rake-compiler/rake-compiler-dock/blob/master/Dockerfile.mri.erb#L103-L110
• https://github.com/rake-compiler/rake-compiler-dock/issues/13

And it's possible on a Mac to modify the .bundle file path to contain @rpath and then make sure that DYLD_LIBRARY_PATH is set appropriately. Some resources on this topic:

• https://medium.com/@donblas/fun-with-rpath-otool-and-install-name-tool-e3e41ae86172
• https://developer.apple.com/library/archive/documentation/DeveloperTools/Conceptual/DynamicLibraries/100-Articles/OverviewOfDynamicLibraries.html#//apple_ref/doc/uid/TP40001873-SW1
• https://wincent.com/wiki/@executable_path,@load_path_and@rpath

But these seem like unnatural workarounds, and so I'm going to attempt to use a Ruby built with --disable-shared to generate the precompiled native gem for darwin.

• [x] Publish the recipe for building that version of Ruby so that it's reproducible.

Universal architecture

The OSX system ruby has an architecture of "universal.x86_64" which ... I'm not sure what that is. I started googling and found:

• https://docs.elementscompiler.com/Platforms/Cocoa/CpuArchitectures/

which implies that Universal is a binary that would work for both x86_64 and arm 64.

Closing. Will cut an rc3 with darwin native gems, and will point people to https://github.com/sparklemotion/nokogiri/issues/2075 to provide feedback.

This issue exists to capture feedback and action items about the precompiled native gems for Linux and OSX that we're shipping in the Nokogiri v1.11 release candidates (prereleases).

(note: at the time of opening this issue, rc3 is not yet out with OSX native gems. i opened this so I could reference it in the changelog for rc3. shouldn't be more than an hour or so, hang tight.)

How do I try this out?

We're only shipping native gems for the following platforms (run bundle platform or ruby -e 'puts Gem::Platform.local.to_s' to identify your system):

• New platforms as of v1.11 release candidates:
  • Linux: x86-linux and x86_64-linux
  • OSX/Darwin: x86_64-darwin19
• Previously existing native gems (note these have already been available for many years):
  • Windows: x86-mingw32 and x64-mingw32
  • JRuby: java (note this native gem has already been available for many years)

If you're on Linux or OSX, try updating your Gemfile with:

gem "nokogiri", "~> 1.11.0.rc3"`

And let us know how it works for you in the comments!

Providing Feedback

If you love this idea in a vague general sense, please give this comment a :heart:.

If you tried native gems on Linux or OSX and they worked just fine, please give this comment a :+1:.

If you're having problems installing or using Nokogiri's precompiled native gems, please comment on this issue with the following information:

• output from ruby -v
• output from gem -v
• output from gem env
• if you're using bundler,
  • output from bundle version
  • output from bundle config (take care to redact any personal info or credentials)
• output from gem install or bundle install, including the command you ran

Why is a precompiled native gem useful?

Installing a precompiled native gem, installation takes about 1 second, and avoids the challenging bits of how to set up the compiler toolchain on your development system. This will reduce the support burden for Team Nokogiri and allow us to move a bit faster on bugfixes and features.

For comparison, compiling and installing the gem (and its vendored dependencies) from source, when only using a single core, my laptop takes 52 seconds. When using all 8 cores, this is cut down to 32 seconds.

Why would I not want to use a precompiled native gem?

I can imagine some folks might have trust issues; if this is you, please let us know in a comment here (or in #2013) what we could do to increase that trust (e.g., I can imagine providing a chain of custody including public build logs with cryptographic hashes of artifacts).

If you have other reasons for not wanting to use a precompiled native gem, let us know in the comments.

How can I avoid using a precompiled native gem?

If you want to avoid using a precompiled gem in your project, do one of the following:

• If you're not using bundler, then run gem install nokogiri -v 1.11.0.rc3 --platform=ruby
• If you are using bundler, then run bundle config set force_ruby_platform true and you'll never use a precompiled native gem

Shout-outs

Thanks to @larskanis for all the time he spends maintaining the gems rake-compiler and rake-compiler-dock, and for being the one who wrote the first PR for Linux native gem support.

Thanks to the GRPC team who already ships native gems (see https://github.com/grpc/grpc). They blazed the trail and provided inspiration and a how-to manual for some edge cases.

Thanks to @tenderlove who paired with me on the nasty OSX bits that I was struggling with.

Resources

• #1571 is the original PR from @larskanis for Linux native gems
• #1983 is an issue to polish Linux native gem support
• #2063 is an issue researching OSX/Darwin native gem support
• #2073 is a PR introducing OSX/Darwin native gem support
• #2013 is an issue to increase trust in published gems

Action Items

These are the remaining action items we need before a final release includes precompiled native gems for Linux and OSX:

• [ ] ship rc3
  • [ ] update CHANGELOG language to point to this issue
  • [ ] email ruby-talk and nokogiri-talk to ask for feedback
  • [ ] tweet to ask for feedback
• [ ] update LICENSE-DEPENDENCIES.txt to reflect reality for all these different combinations of releases
• [ ] eliminate the "extra" dll being shipped as lib/nokogiri/nokogiri.{bundle,so}
• [ ] make sure Nokogumbo still installs cleanly (see https://github.com/rubys/nokogumbo/pull/148 and https://github.com/sparklemotion/nokogiri/pull/1788)
• [ ] note in VersionInfo that Nokogiri is precompiled
• [ ] finish most of the issues under the v1.11.0 milestone

What problem is this PR intended to solve?

Proposed work for #2063 -- precompiled native gems for OSX/Darwin.

Have you included adequate test coverage?

Needs:

• [x] implement verify_dll for the darwin .bundle

Does this change affect the C or the Java implementations?

No functional changes -- only developer changes in how we build gems.

I'm going to merge this so I can ship an RC and get feedback. Would still love to hear anybody's feedback about this!

Thank you for contributing to Nokogiri! To help us prioritize, please take care to answer the questions below when you submit this pull request.

The Nokogiri core team work off of master, so please submit all PRs based on the master branch. We generally will cherry-pick relevant bug fixes onto the current release branch.

What problem is this PR intended to solve?

If there is an existing issue that describes this, feel free to simply link to that issue.

Otherwise, please provide enough context for the Nokogiri maintainers to understand your intent.

Have you included adequate test coverage?

We have a thorough test suite that allows us to create releases confidently and prevent accidental regressions. Any proposed change in behavior must be accompanied by tests.

If possible, please try to write the tests so that they communicate intent.

Does this change affect the C or the Java implementations?

If you're proposing a change to the C implementation, has the behavior change been made to the Java code as well? And vice versa?

If not, that may be OK, just please note it here.

Marked as spam, user blocked.

@larskanis @tenderlove I could use a sanity check on these remaining references. This seems reasonable to me. I'll probably just ship an RC and see if we get feedback, but wanted to highlight that I made a decision with incomplete information here.

Update:

• pull request at #2073 seems to work for x86_64-darwin for ruby 2.4, 2.5, 2.6, 2.7.
  • which includes a recipe for installing a set of rubies for building the native gems at scripts/setup-osx-native-builders
• I won't ship a separate universal-darwin-19 gem unless and until people ask (and I'll note that the x86_64-darwin-19 gem works for the system ruby that self-identifies as universal-darwin-19, even if the platform strings don't match)

What problem is this PR intended to solve?

Proposed work for #2063 -- precompiled native gems for OSX/Darwin.

Have you included adequate test coverage?

TBD: testing these gems similar to how we're testing the linux gems

Does this change affect the C or the Java implementations?

No functional changes -- only developer changes in how we build gems.

@DougEdey I couldn't reproduce this using the same toolchain and setup. So I'd like to ask some exploratory questions:

1. what's the output from bundle info mini_portile2? Or, if bundler won't tell you because it didn't finish bundling, what's the output from gem info mini_portile2?

I see:

shipify $ bundle info mini_portile2
  * mini_portile2 (2.4.0)
	Summary: Simplistic port-like solution for developers
	Homepage: http://github.com/flavorjones/mini_portile
	Path: /Users/flavorjones/.gem/ruby/2.6.6/gems/mini_portile2-2.4.0

shipify $ gem info mini_portile2

*** LOCAL GEMS ***

mini_portile2 (2.5.0, 2.4.0)
    Authors: Luis Lavena, Mike Dalessio, Lars Kanis
    Homepage: http://github.com/flavorjones/mini_portile
    License: MIT
    Installed at (2.5.0): /Users/flavorjones/.gem/ruby/2.6.6
                 (2.4.0): /Users/flavorjones/.gem/ruby/2.6.6

    Simplistic port-like solution for developers

2. Is the mini_portile2 installation OK? Are the files we expect there? Here's what I see:

shipify $ find /Users/flavorjones/.gem/ruby/2.6.6/gems/mini_portile2-2.4.0/lib
/Users/flavorjones/.gem/ruby/2.6.6/gems/mini_portile2-2.4.0/lib
/Users/flavorjones/.gem/ruby/2.6.6/gems/mini_portile2-2.4.0/lib/mini_portile2
/Users/flavorjones/.gem/ruby/2.6.6/gems/mini_portile2-2.4.0/lib/mini_portile2/mini_portile.rb
/Users/flavorjones/.gem/ruby/2.6.6/gems/mini_portile2-2.4.0/lib/mini_portile2/mini_portile_cmake.rb
/Users/flavorjones/.gem/ruby/2.6.6/gems/mini_portile2-2.4.0/lib/mini_portile2/version.rb
/Users/flavorjones/.gem/ruby/2.6.6/gems/mini_portile2-2.4.0/lib/mini_portile2.rb

3. What is your bundle config output? I'd be interested in anything related to paths. Here's what I see:

shipify $ bundle config
Settings are listed in order of priority. The top value will be used.
packages.shopify.io
Set for the current user (/Users/flavorjones/.bundle/config): "<redacted>"

jobs
Set for your local app (/Users/flavorjones/src/github.com/Shopify/shipify/.bundle/config): 4

(please redact your packages credential!)

4. What's your gem env? Here's mine (with shell paths redacted):

shipify $ gem env
RubyGems Environment:
  - RUBYGEMS VERSION: 3.0.3
  - RUBY VERSION: 2.6.6 (2020-03-31 patchlevel 146) [x86_64-darwin19]
  - INSTALLATION DIRECTORY: /Users/flavorjones/.gem/ruby/2.6.6
  - USER INSTALLATION DIRECTORY: /Users/flavorjones/.gem/ruby/2.6.0
  - RUBY EXECUTABLE: /opt/rubies/2.6.6/bin/ruby
  - GIT EXECUTABLE: /Users/flavorjones/.nix-profile/bin/git
  - EXECUTABLE DIRECTORY: /Users/flavorjones/.gem/ruby/2.6.6/bin
  - SPEC CACHE DIRECTORY: /Users/flavorjones/.gem/specs
  - SYSTEM CONFIGURATION DIRECTORY: /opt/rubies/2.6.6/etc
  - RUBYGEMS PLATFORMS:
    - ruby
    - x86_64-darwin-19
  - GEM PATHS:
     - /Users/flavorjones/.gem/ruby/2.6.6
     - /opt/rubies/2.6.6/lib/ruby/gems/2.6.0
  - GEM CONFIGURATION:
     - :update_sources => true
     - :verbose => true
     - :backtrace => false
     - :bulk_threshold => 1000
     - "gem" => "--no-ri --no-rdoc --no-document"
  - REMOTE SOURCES:
     - https://rubygems.org/
  - SHELL PATH:
... <snip> ...

I'm fishing for differences in our configurations, or issues that might be specific to your gem installation.

@DougEdey Sorry you're having problems, I'll try to help.

(Coordinating with Doug in our company Slack, will close the loop here.)

Confirmed, setting --enable-shared or --disable-shared affects whether libruby.so is linked-to from nokogiri.so.

Ruby build with --enable-shared:

juno ruby-2.7.0 go1.14.6 (master *)
nokogiri $ ruby -e 'puts RbConfig::CONFIG["configure_args"]'
 '--prefix=/home/flavorjones/.rvm/rubies/ruby-2.7.0' '--disable-install-doc' '--enable-shared'

juno ruby-2.7.0 go1.14.6 (master *)
nokogiri $ be rake lib/nokogiri/nokogiri.so
install -c tmp/x86_64-linux/nokogiri/2.7.0/nokogiri.so lib/nokogiri/nokogiri.so
cp tmp/x86_64-linux/nokogiri/2.7.0/nokogiri.so tmp/x86_64-linux/stage/lib/nokogiri/nokogiri.so

juno ruby-2.7.0 go1.14.6 (master *)
nokogiri $ ldd lib/nokogiri/nokogiri.so 
	linux-vdso.so.1 (0x00007ffc2b3e8000)
	libruby.so.2.7 => /home/flavorjones/.rvm/rubies/ruby-2.7.0/lib/libruby.so.2.7 (0x00007fd440168000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd43fff9000)
	liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007fd43ffd0000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fd43ffb4000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd43fdc0000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd43fd9d000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fd43fd92000)
	libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007fd43fd0e000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd43fd08000)
	libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007fd43fccd000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fd440974000)

juno ruby-2.7.0 go1.14.6 (master *)
nokogiri $ ls -l lib/nokogiri/nokogiri.so 
-rwxrwxr-x 1 flavorjones flavorjones 2876352 2020-09-04 10:37 lib/nokogiri/nokogiri.so*

juno ruby-2.7.0 go1.14.6 (master *)
nokogiri $ nm lib/nokogiri/nokogiri.so | fgrep rb_gc_mark
                 U rb_gc_mark

Ruby built with --disable-shared:

juno ruby-2.7.1 go1.14.6 (master *)
nokogiri $ ruby -e 'puts RbConfig::CONFIG["configure_args"]'
 '--prefix=/home/flavorjones/.rvm/rubies/ruby-2.7.1' '--disable-shared' '--disable-install-doc'

juno ruby-2.7.1 go1.14.6 (master *)
nokogiri $ be rake lib/nokogiri/nokogiri.so
install -c tmp/x86_64-linux/nokogiri/2.7.1/nokogiri.so lib/nokogiri/nokogiri.so
cp tmp/x86_64-linux/nokogiri/2.7.1/nokogiri.so tmp/x86_64-linux/stage/lib/nokogiri/nokogiri.so

juno ruby-2.7.1 go1.14.6 (master *)
nokogiri $ ldd lib/nokogiri/nokogiri.so
	linux-vdso.so.1 (0x00007ffdd8f58000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f8e5f900000)
	liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f8e5f8d7000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f8e5f8bb000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f8e5f6c9000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f8e5fcaf000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f8e5f6a4000)

juno ruby-2.7.1 go1.14.6 (master *)
nokogiri $ ls -l lib/nokogiri/nokogiri.so
-rwxrwxr-x 1 flavorjones flavorjones 2876240 2020-09-04 10:38 lib/nokogiri/nokogiri.so*

juno ruby-2.7.1 go1.14.6 (master *)
nokogiri $ nm lib/nokogiri/nokogiri.so | fgrep rb_gc_mark
                 U rb_gc_mark

@larskanis I noticed that last night (something I had not seen before)!

It looks like when ruby is configured without --enable-shared that it might not link to libruby (based only on looking at the .bundle generated by rubies configured with and without that option). I need to test more rigorously that this is the case, but either way I understand more deeply what's going on now and can probably hack something. I'll update this issue when I learn more!

@rubys - I agree that leveraging Nokogumbo is the easiest next step (and potentially the only step, at least for another 10 years :laughing: :sob:).

To be clear: a consequence of this decision is for Loofah to start using Nokogumbo for HTML parsing, meaning that Nokogumbo will become a dependency of Rails, and almost certainly will have an increased support burden as a result.

None of us is paid for our work. How can we preserve maintainer happiness on both projects while maintaining an acceptable level of support for the Rails community?

Proposed next steps

I want to be very respectful of boundaries here. The Nokogumbo contributors have filled this functionality gap very well and I don't want to presume that consolidation or assimilation of either project is necessary or desirable.

If it would be easier to shift this discussion into a realtime face-to-face conversation, I will gladly do that.

1. 2020:

Status quo for a few more months. Ship Nokogiri v1.11.0 towards the end of 2020 with precompiled native gems for windows, linux, and osx/darwin.

Ensure that Nokogumbo installation doesn't break (see https://github.com/rubys/nokogumbo/pull/148 and https://github.com/sparklemotion/nokogiri/pull/1788).

2. Decision point:

After the precompiled native gem release has baked for a bit and we've learned more about whether it was a good decision or not and how hard it is to maintain this approach -- maybe in early 2021? -- make a decision, involving the key stakeholders from both projects, about whether to merge the projects.

3.a. No, don't merge:

If we decide not to merge, then we firm up the contract between the two gems -- both at installation and at runtime -- and invest in integration testing. [1]

3.b. Yes, please merge:

If we decide to merge, then I'd like to suggest Nokogumbo merge into Nokogiri, rather than both merging into a net-new gem; but I am open to a conversation about other approaches. Please let me know if this is a contentious idea, and I'm happy to jump on a realtime call to discuss it.

The core contributors for the merged gem would be the set-union of the Nokogumbo and Nokogiri core contributors, and the commit history for both projects would be preserved. We would make a big deal out of it, and graciously name and thank the people who've made the projects a success. Yay us!

Immediate work would be to functionally integrate the code, and then get precompilation (native gems) working and tested for Nokogiri::HTML5 and Gumbo for a Nokogiri v1.12.0 release -- or potentially a v2.0 release? -- in 2021.

Brief thoughts on our options

My mental model is such that I think it makes most sense to merge (3.b.), but would really like the key Nokogumbo creators and contributors to talk about their own mental models, opinions, fears, etc.

I suspect that keeping the gems separate is going to lead to unhappiness among the maintainers. The Nokogumbo maintainers will inevitably shift focus from "making HTML5 parsing work in Ruby" to "making this C codebase compile and install on an infinite variety of ill-configured systems" as the Rails community comes to depend on it. (Ask me how I know! :sob:) The Nokogiri and Loofah maintainers will experience burnout when their sphere of responsibility doesn't match their sphere of control -- specifically when users loudly complain about something not working perfectly across the integration boundaries.

One reason to not merge might be the size and complexity of the codebase increases for both teams in a merger. However, I think we'd more than offset or mitigate that complexity when we plan to reduce the total support burden by: a) improving the installation experience, b) improving the developer experience, c) removing a vector (libxml2's HTML parser) for security vulnerabilities, d) replacing the informal brittle install-time contract with a robust compile-time contract, e) having a unified team large enough for folks to still specialize within the codebase.

[1]: Nokogumbo's installation process is still pretty tightly-coupled to Nokogiri's installation process, and there's an implied contract between the two that involves the structure of Nokogiri::VersionInfo and the presence of header files on disk. So we'd need to tighten up that contract, and ensure we're integration-testing across the two projects.

Loofah is updated to use Nokogumbo (only on CRuby), making Nokogumbo a dependency of Rails. Nokogumbo support burden would likely increase as a thundering herd of Rails developers are now installing Nokogumbo.

I expect some demand for a faster/better installation experience for Nokogumbo, similar to what Nokogiri has seen over the past few years. Work to keep the support burden low by investing in a single installer toolchain, and a unified memory testing approach (today Nokogiri uses valgrind, but I'm looking into ASan). We'd likely spend some time re-implementing the toolchain and keeping it in sync across the two projects. This doesn't sound like fun to me, but we could do it.

Hi there!

I googled but didn't find a way to search for a partial XPath in an XML file.

I need to find everything ending with Status word, for example: <diskSizeStatus>1</diskSizeStatus>.

I am using Nokogiri::XML and tried xml.xpath('//@*search', search: 'Status') but was unsuccessful.

Anyone could help me with this, please?

Hi, thanks for asking this question, I'll try to help.

I'd like to note that this is a general XPath question, and not a Nokogiri-specific question. A bit of googling led me to https://stackoverflow.com/questions/40934644/xpath-for-element-whose-attribute-value-ends-with-a-specific-string

Nokogiri's version of libxml2 implements XPath 1.0 (and not XPath 2.0) so you'll have to use the more complicated query:

#! /usr/bin/env ruby

require 'nokogiri'

xml = <<EOF
<root>
  <TagFoo>asdf</TagFoo>
  <TagBar>qwer</TagBar>
  <TagBarBaz>zxcv</TagBarBaz>
</root>
EOF

doc = Nokogiri::XML.parse(xml)

results = doc.xpath("//*[substring(name(), string-length(name()) - string-length('Bar')+1) = 'Bar']")
puts results.to_xml
# => <TagBar>qwer</TagBar>

I'll note that there are a few other ways you could have done this, depending on how sensitive you are to performance.

I hope this helps.

Hi, I'm sorry, I'm not sure how to help you. I've tried to reproduce what you're seeing and cannot -- and demonstrated that it's possible to run an unsupported version of Nokogiri on an unsupported version of Ubuntu.

You may want to investigate a bit more yourself, these error messages indicate that you're having trouble compiling libxml2 on your system:

nanohttp.c:1190:35: error: dereferencing pointer to incomplete type
      } else if (have_ipv6 () && (h->h_addrtype == AF_INET6)) {
                                   ^
nanohttp.c:1192:23: error: dereferencing pointer to incomplete type
   if ((unsigned int) h->h_length > sizeof(ia6)) {
                       ^
nanohttp.c:1196:18: error: dereferencing pointer to incomplete type
   memcpy (&ia6, h->h_addr_list[i], h->h_length);
                  ^
nanohttp.c:1196:37: error: dereferencing pointer to incomplete type
   memcpy (&ia6, h->h_addr_list[i], h->h_length);
                                     ^
nanohttp.c:1197:26: error: dereferencing pointer to incomplete type
   sockin6.sin6_family = h->h_addrtype;
                          ^
make[2]: *** [nanohttp.lo] Error 1
make[2]: Leaving directory

My advice would be to either a) investigate how to get libxml2 to build on your system, or b) try installating nokogiri against system libraries that you can install on Ubuntu Trusty.

You can read more about option (b) here: https://nokogiri.org/tutorials/installing_nokogiri.html#install-with-system-libraries, you'll need to install the libxml2-dev and libxslt-dev packages (they may be named slightly differently on such an old system? I honestly can't remember).

Good luck.

Hi, I fired up a docker image of trusty:

root@9df393c99ac8:/# cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.6 LTS"

and installed Ruby 2.2.10 through RVM:

`` root@9df393c99ac8:/# ruby -v ruby 2.2.10p489 (2018-03-28 revision 63023) [x86_64-linux]

and installed Nokogiri v1.9.1 without problems:

root@9df393c99ac8:/# nokogiri -v

Nokogiri (1.9.1)

---
warnings: []
nokogiri: 1.9.1
ruby:
  version: 2.2.10
  platform: x86_64-linux
  description: ruby 2.2.10p489 (2018-03-28 revision 63023) [x86_64-linux]
  engine: ruby
libxml:
  binding: extension
  source: packaged
  libxml2_path: "/usr/local/rvm/gems/ruby-2.2.10/gems/nokogiri-1.9.1/ports/x86_64-pc-linux-gnu/libxml2/2.9.8"
  libxslt_path: "/usr/local/rvm/gems/ruby-2.2.10/gems/nokogiri-1.9.1/ports/x86_64-pc-linux-gnu/libxslt/1.1.32"
  libxml2_patches:
  - 0001-Revert-Do-not-URI-escape-in-server-side-includes.patch
  - 0002-Fix-nullptr-deref-with-XPath-logic-ops.patch
  - 0003-Fix-infinite-loop-in-LZMA-decompression.patch
  libxslt_patches: []
  compiled: 2.9.8
  loaded: 2.9.8

Can you tell me more about how your system is configured?

Updates the requirements on google-api-client to permit the latest version. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/googleapis/google-api-ruby-client/blob/master/CHANGELOG.md">google-api-client's changelog</a>.</em></p> <blockquote> <h1>0.44.0</h1> <ul> <li>Breaking changes: <ul> <li>Deleted <code>dns_v2beta1</code></li> <li>Deleted <code>oauth2_v2</code></li> <li>Deleted <code>storage_v1beta2</code></li> </ul> </li> <li>Backwards compatible changes: <ul> <li>Added <code>fitness_v1</code></li> <li>Added <code>groupsmigration_v1</code></li> <li>Added <code>localservices_v1</code></li> <li>Updated <code>accessapproval_v1</code></li> <li>Updated <code>accesscontextmanager_v1</code></li> <li>Updated <code>accesscontextmanager_v1beta</code></li> <li>Updated <code>adexchangebuyer2_v2beta1</code></li> <li>Updated <code>adexchangebuyer_v1_4</code></li> <li>Updated <code>adexperiencereport_v1</code></li> <li>Updated <code>admin_datatransfer_v1</code></li> <li>Updated <code>admin_directory_v1</code></li> <li>Updated <code>admin_reports_v1</code></li> <li>Updated <code>admob_v1</code></li> <li>Updated <code>alertcenter_v1beta1</code></li> <li>Updated <code>androidenterprise_v1</code></li> <li>Updated <code>androidmanagement_v1</code></li> <li>Updated <code>androidpublisher_v3</code></li> <li>Updated <code>apigee_v1</code></li> <li>Updated <code>appengine_v1</code></li> <li>Updated <code>appengine_v1alpha</code></li> <li>Updated <code>appengine_v1beta</code></li> <li>Updated <code>appsmarket_v2</code></li> <li>Updated <code>artifactregistry_v1beta1</code></li> <li>Updated <code>bigquery_v2</code></li> <li>Updated <code>bigqueryconnection_v1beta1</code></li> <li>Updated <code>bigquerydatatransfer_v1</code></li> <li>Updated <code>bigqueryreservation_v1</code></li> <li>Updated <code>bigqueryreservation_v1alpha2</code></li> <li>Updated <code>bigqueryreservation_v1beta1</code></li> <li>Updated <code>bigtableadmin_v1</code></li> <li>Updated <code>bigtableadmin_v2</code></li> <li>Updated <code>billingbudgets_v1beta1</code></li> <li>Updated <code>binaryauthorization_v1</code></li> <li>Updated <code>binaryauthorization_v1beta1</code></li> <li>Updated <code>calendar_v3</code></li> <li>Updated <code>chat_v1</code></li> <li>Updated <code>civicinfo_v2</code></li> <li>Updated <code>cloudasset_v1</code></li> <li>Updated <code>cloudasset_v1beta1</code></li> <li>Updated <code>cloudasset_v1p1beta1</code></li> <li>Updated <code>cloudasset_v1p4beta1</code></li> <li>Updated <code>cloudasset_v1p5beta1</code></li> <li>Updated <code>cloudbilling_v1</code></li> <li>Updated <code>cloudbuild_v1</code></li> </ul> </li> </ul> <!-- raw HTML omitted --> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/googleapis/google-api-ruby-client/commit/bedfa959508ca78405998f98a439ccfa51fbe6db"><code>bedfa95</code></a> 0.44.0</li> <li><a href="https://github.com/googleapis/google-api-ruby-client/commit/76a32f11cf615c1f7bbd041f1b72e12f0449d41e"><code>76a32f1</code></a> Autogenerated update (2020-08-29)</li> <li><a href="https://github.com/googleapis/google-api-ruby-client/commit/32fac21291a60b0f09546c27f8e547f8b138f297"><code>32fac21</code></a> Autogenerated update (2020-08-28)</li> <li><a href="https://github.com/googleapis/google-api-ruby-client/commit/5874aa5aa4188ff7e5e7e86def4111a24d1362a9"><code>5874aa5</code></a> Autogenerated update (2020-08-27)</li> <li><a href="https://github.com/googleapis/google-api-ruby-client/commit/f06309c6ba3235e27eda94a56138f6e5e21f98f1"><code>f06309c</code></a> Autogenerated update (2020-08-26)</li> <li><a href="https://github.com/googleapis/google-api-ruby-client/commit/8b8f32d9ca34a37db69dc147574f6339178e947c"><code>8b8f32d</code></a> Autogenerated update (2020-08-25)</li> <li><a href="https://github.com/googleapis/google-api-ruby-client/commit/d0d22a2baa3c733d6def93ef2342d4045e6d10db"><code>d0d22a2</code></a> Autogenerated update (2020-08-22)</li> <li><a href="https://github.com/googleapis/google-api-ruby-client/commit/e08f0d3bef470cf05ec89b5eeca8d8d806b317e0"><code>e08f0d3</code></a> Autogenerated update (2020-08-21)</li> <li><a href="https://github.com/googleapis/google-api-ruby-client/commit/ccf85d8cdabdb5f5616a7d2dbad2b3f2560c8901"><code>ccf85d8</code></a> Autogenerated update (2020-08-20)</li> <li><a href="https://github.com/googleapis/google-api-ruby-client/commit/e5cce9ef8df2fc5a255d5426a3aa164dc0cfc747"><code>e5cce9e</code></a> Autogenerated update (2020-08-19)</li> <li>Additional commits viewable in <a href="https://github.com/google/google-api-ruby-client/compare/0.24.0...0.44.0">compare view</a></li> </ul> </details> <br />

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

<details> <summary>Dependabot commands and options</summary> <br />

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot merge will merge this PR after your CI passes on it
• @dependabot squash and merge will squash and merge this PR after your CI passes on it
• @dependabot cancel merge will cancel a previously requested merge and block automerging
• @dependabot reopen will reopen this PR if it is closed
• @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
• @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
• @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
• @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
• @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language
• @dependabot badge me will comment on this PR with code to add a "Dependabot enabled" badge to your readme

Additionally, you can set the following in your Dependabot dashboard:

• Update frequency (including time of day and day of week)
• Pull request limits (per update run and/or open at any time)
• Automerge options (never/patch/minor, and dev/runtime dependencies)
• Out-of-range updates (receive only lockfile updates, if desired)
• Security updates (receive only security updates, if desired)

</details>

This blog post has some good info on linker paths on OSX, along with links to deeper reference material: https://medium.com/@donblas/fun-with-rpath-otool-and-install-name-tool-e3e41ae86172

On linux, dynamic libraries are names searched for in a runtime path (LD_LIBRARY_PATH):

$ ldd lib/nokogiri/nokogiri.so 
	linux-vdso.so.1 (0x00007ffcdff91000)
	libruby.so.2.5 => not found
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fabb8487000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fabb846b000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fabb8448000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fabb8254000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fabb8a2a000)

On OSX this is what I see:

nokogiri $ otool -L lib/nokogiri/nokogiri.bundle
lib/nokogiri/nokogiri.bundle:
	/Users/flavorjones/.rvm/rubies/ruby-2.7.0/lib/libruby.2.7.dylib (compatibility version 2.7.0, current version 2.7.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.100.1)
	/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
	/usr/local/opt/xz/lib/liblzma.5.dylib (compatibility version 8.0.0, current version 8.5.0)
	/usr/local/opt/zlib/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)

Maybe worth noting: if I build the binary gem using homebrew-installed ruby 2.7, but install it into an rvm-installed ruby 2.7, it installs but I get this error at runtime:

$ nokogiri -v
Traceback (most recent call last):
	9: from /Users/flavorjones/.rvm/gems/ruby-2.7.0/bin/ruby_executable_hooks:24:in `<main>'
	8: from /Users/flavorjones/.rvm/gems/ruby-2.7.0/bin/ruby_executable_hooks:24:in `eval'
	7: from /Users/flavorjones/.rvm/gems/ruby-2.7.0/bin/nokogiri:23:in `<main>'
	6: from /Users/flavorjones/.rvm/gems/ruby-2.7.0/bin/nokogiri:23:in `load'
	5: from /Users/flavorjones/.rvm/gems/ruby-2.7.0/gems/nokogiri-1.11.0.rc2-x86_64-darwin-19/bin/nokogiri:6:in `<top (required)>'
	4: from /Users/flavorjones/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require'
	3: from /Users/flavorjones/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require'
	2: from /Users/flavorjones/.rvm/gems/ruby-2.7.0/gems/nokogiri-1.11.0.rc2-x86_64-darwin-19/lib/nokogiri.rb:13:in `<top (required)>'
	1: from /Users/flavorjones/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require'
/Users/flavorjones/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require': cannot load such file -- nokogiri/2.7/nokogiri (LoadError)
	10: from /Users/flavorjones/.rvm/gems/ruby-2.7.0/bin/ruby_executable_hooks:24:in `<main>'
	 9: from /Users/flavorjones/.rvm/gems/ruby-2.7.0/bin/ruby_executable_hooks:24:in `eval'
	 8: from /Users/flavorjones/.rvm/gems/ruby-2.7.0/bin/nokogiri:23:in `<main>'
	 7: from /Users/flavorjones/.rvm/gems/ruby-2.7.0/bin/nokogiri:23:in `load'
	 6: from /Users/flavorjones/.rvm/gems/ruby-2.7.0/gems/nokogiri-1.11.0.rc2-x86_64-darwin-19/bin/nokogiri:6:in `<top (required)>'
	 5: from /Users/flavorjones/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require'
	 4: from /Users/flavorjones/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require'
	 3: from /Users/flavorjones/.rvm/gems/ruby-2.7.0/gems/nokogiri-1.11.0.rc2-x86_64-darwin-19/lib/nokogiri.rb:11:in `<top (required)>'
	 2: from /Users/flavorjones/.rvm/gems/ruby-2.7.0/gems/nokogiri-1.11.0.rc2-x86_64-darwin-19/lib/nokogiri.rb:15:in `rescue in <top (required)>'
	 1: from /Users/flavorjones/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require'
/Users/flavorjones/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require': dlopen(/Users/flavorjones/.rvm/gems/ruby-2.7.0/gems/nokogiri-1.11.0.rc2-x86_64-darwin-19/lib/nokogiri/nokogiri.bundle, 9): Library not loaded: /usr/local/opt/ruby/lib/libruby.2.7.dylib (LoadError)
  Referenced from: /Users/flavorjones/.rvm/gems/ruby-2.7.0/gems/nokogiri-1.11.0.rc2-x86_64-darwin-19/lib/nokogiri/nokogiri.bundle
  Reason: image not found - /Users/flavorjones/.rvm/gems/ruby-2.7.0/gems/nokogiri-1.11.0.rc2-x86_64-darwin-19/lib/nokogiri/nokogiri.bundle

But then when I re-build the gem with rvm-installed ruby, and then install the gem, it works fine:

# Nokogiri (1.11.0.rc2)
    ---
    warnings: []
    nokogiri: 1.11.0.rc2
    ruby:
      version: 2.7.0
      platform: x86_64-darwin19
      description: ruby 2.7.0p0 (2019-12-25 revision 647ee6f091) [x86_64-darwin19]
      engine: ruby
    libxml:
      source: packaged
      patches:
      - 0001-Revert-Do-not-URI-escape-in-server-side-includes.patch
      - 0002-Remove-script-macro-support.patch
      - 0003-Update-entities-to-remove-handling-of-ssi.patch
      - 0004-libxml2.la-is-in-top_builddir.patch
      - 0005-Fix-infinite-loop-in-xmlStringLenDecodeEntities.patch
      compiled: 2.9.10
      loaded: 2.9.10
    libxslt:
      source: packaged
      patches: []
      compiled: 1.1.34
      loaded: 1.1.34

The problem is that both of those rubies (homebrew-installed and rvm-installed) have the same RUBY_PLATFORM and gem env platform. :thinking:

Voila:

huygens (master *)
nokogiri $ time gem install pkg/nokogiri-1.11.0.rc2-x86_64-darwin-19.gem
Successfully installed nokogiri-1.11.0.rc2-x86_64-darwin-19
1 gem installed

real	0m0.764s
user	0m0.355s
sys	0m0.187s

huygens (master *)
nokogiri $ ruby -S nokogiri -v
# Nokogiri (1.11.0.rc2)
    ---
    warnings: []
    nokogiri: 1.11.0.rc2
    ruby:
      version: 2.7.1
      platform: x86_64-darwin19
      description: ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x86_64-darwin19]
      engine: ruby
    libxml:
      source: packaged
      patches:
      - 0001-Revert-Do-not-URI-escape-in-server-side-includes.patch
      - 0002-Remove-script-macro-support.patch
      - 0003-Update-entities-to-remove-handling-of-ssi.patch
      - 0004-libxml2.la-is-in-top_builddir.patch
      - 0005-Fix-infinite-loop-in-xmlStringLenDecodeEntities.patch
      compiled: 2.9.10
      loaded: 2.9.10
    libxslt:
      source: packaged
      patches: []
      compiled: 1.1.34
      loaded: 1.1.34

Voila:

Ah, this is actually better and addresses the root problem: that we're trying to invoke the wrongly-named rake task:

diff --git a/.cross_rubies b/.cross_rubies
index 57c8fb0..e884b16 100644
--- a/.cross_rubies
+++ b/.cross_rubies
@@ -2,6 +2,7 @@
 2.7.0:x86_64-w64-mingw32
 2.7.0:i686-linux-gnu
 2.7.0:x86_64-linux-gnu
+2.7.0:x86_64-darwin19
 2.6.0:i686-w64-mingw32
 2.6.0:x86_64-w64-mingw32
 2.6.0:i686-linux-gnu
diff --git a/tasks/cross-ruby.rb b/tasks/cross-ruby.rb
index 35c098f..1d5c2fd 100644
--- a/tasks/cross-ruby.rb
+++ b/tasks/cross-ruby.rb
@@ -26,6 +26,8 @@ CrossRuby = Struct.new(:version, :host) do
         "x86_64-linux"
       when /\Ai[3-6]86.*linux/
         "x86-linux"
+      when /\Ax86_64-darwin19/
+        "x86_64-darwin19"
       else
         raise "unsupported host: #{host}"
       end
@@ -179,7 +181,7 @@ namespace "gem" do
       task "guest" do
         # use Task#invoke because the pkg/*gem task is defined at runtime
         Rake::Task["native:#{plat}"].invoke
-        Rake::Task["pkg/#{HOE.spec.full_name}-#{plat}.gem"].invoke
+        Rake::Task["pkg/#{HOE.spec.full_name}-#{Gem::Platform.new(plat).to_s}.gem"].invoke
       end
     end
   end

And the output file is now nokogiri-1.11.0.rc2-x86_64-darwin-19.gem which is TOTALLY GOING TO WORK because gem env will be looking for the same platform name.

OK, I managed to build a working native darwin gem by applying the following patch!

diff --git a/.cross_rubies b/.cross_rubies
index 57c8fb0..e3f5f1e 100644
--- a/.cross_rubies
+++ b/.cross_rubies
@@ -2,6 +2,7 @@
 2.7.0:x86_64-w64-mingw32
 2.7.0:i686-linux-gnu
 2.7.0:x86_64-linux-gnu
+2.7.0:universal-darwin-19
 2.6.0:i686-w64-mingw32
 2.6.0:x86_64-w64-mingw32
 2.6.0:i686-linux-gnu
diff --git a/Rakefile b/Rakefile
index f5f6f47..16be862 100644
--- a/Rakefile
+++ b/Rakefile
@@ -118,6 +118,7 @@ else
   end
 
   Rake::ExtensionTask.new("nokogiri", HOE.spec) do |ext|
+    ext.platform = "universal-darwin-19"
     ext.lib_dir = File.join(*['lib', 'nokogiri', ENV['FAT_DIR']].compact)
     ext.config_options << ENV['EXTOPTS']
     ext.cross_compile  = true
diff --git a/tasks/cross-ruby.rb b/tasks/cross-ruby.rb
index 35c098f..d4e84bd 100644
--- a/tasks/cross-ruby.rb
+++ b/tasks/cross-ruby.rb
@@ -26,6 +26,8 @@ CrossRuby = Struct.new(:version, :host) do
         "x86_64-linux"
       when /\Ai[3-6]86.*linux/
         "x86-linux"
+      when /\Auniversal-darwin-19/
+        "universal-darwin-19"
       else
         raise "unsupported host: #{host}"
       end

and then running rake gem:universal-darwin-19:guest -- because obviously we don't have a guest container.

$ time gem install pkg/nokogiri-1.11.0.rc2-universal-darwin-19.gem 
Successfully installed nokogiri-1.11.0.rc2-universal-darwin-19
1 gem installed

real	0m0.795s
user	0m0.350s
sys	0m0.185s

$ ruby -S nokogiri -v
# Nokogiri (1.11.0.rc2)
    ---
    warnings: []
    nokogiri: 1.11.0.rc2
    ruby:
      version: 2.7.1
      platform: x86_64-darwin19
      description: ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x86_64-darwin19]
      engine: ruby
    libxml:
      source: packaged
      patches:
      - 0001-Revert-Do-not-URI-escape-in-server-side-includes.patch
      - 0002-Remove-script-macro-support.patch
      - 0003-Update-entities-to-remove-handling-of-ssi.patch
      - 0004-libxml2.la-is-in-top_builddir.patch
      - 0005-Fix-infinite-loop-in-xmlStringLenDecodeEntities.patch
      compiled: 2.9.10
      loaded: 2.9.10
    libxslt:
      source: packaged
      patches: []
      compiled: 1.1.34
      loaded: 1.1.34

@larskanis I'm also happy to send a PR to rake-compiler if that's what's needed to sort this out -- I'm only asking for your advice on what solution to pursue! I'll keep playing around.

@larskanis I'm hoping you can help clarify some things for me. There seems to be a few different platform name values on OSX and I'm not sure how to configure the cross-ruby tasks to do a proof-of-concept native darwin build.

TL;DR:

On Linux, all three of these values are the same:

• RUBY_PLATFORM # => "x86_64-linux"
• Gem::Platform.new(RUBY_PLATFORM).to_s # => "x86_64-linux"
• gem env reports x86_64-linux as one of the "RUBYGEMS PLATFORMS"

On OSX, all three of these values are different:

• RUBY_PLATFORM # => "universal.x86_64-darwin19"
• Gem::Platform.new(RUBY_PLATFORM).to_s # => "universal.x86_64-darwin-19"
• gem env reports universal-darwin-19 as one of the "RUBYGEMS PLATFORMS"

Longer explanation

First, the native tasks are defined in rake-compiler's Rake::ExtensionTask#define_native_tasks. Right now this method is being called from #define with no arguments, and so platf is set to platform which returns RUBY_PLATFORM when @platform isn't set -- and so this value on a mac is universal.x86_64-darwin19.

This matches what ruby -v tells me: ruby 2.6.3p62 (2019-04-16 revision 67580) [universal.x86_64-darwin19].

gem env, however, says the platform is universal-darwin-19.

Finally, though, if I add universal.x86_64-darwin19 to both .cross_rubies and tasks/cross-ruby.rb:

  def platform
    @platform ||= case host
      when /\Ax86_64.*mingw32/
        "x64-mingw32"
      when /\Ai[3-6]86.*mingw32/
        "x86-mingw32"
      when /\Ax86_64.*linux/
        "x86_64-linux"
      when /\Ai[3-6]86.*linux/
        "x86-linux"
      when /\Auniversal.x86_64-darwin19/
        "universal.x86_64-darwin19"
      else
        raise "unsupported host: #{host}"
      end
  end

then the error I see is:

Don't know how to build task 'pkg/nokogiri-1.11.0.rc2-universal.x86_64-darwin19.gem' (See the list of available tasks with `rake --tasks`)
Did you mean?  pkg/nokogiri-1.11.0.rc2-universal.x86_64-darwin-19.gem

which implies the existence of a THIRD platform name which inserts a space between "darwin" and "19" -- and I discovered that this is the spec platform which is set to Gem::Platform.new(platf) inside define_native_tasks:

Gem::Platform.new("universal.x86_64-darwin19").to_s
=> "universal.x86_64-darwin-19"

I can't quite put all the pieces together and make this work, but it feels like this might be pointing towards an inconsistency in how these names are handled in rake-compiler?

Additionally, I see:

conftest.c:5:2: error: #error libxml2 is older than 2.9.3

which tells me you're compiling against system libraries, and libxml2 2.9.3 is too old for Nokogiri to support. Please use the vendored libraries if this continues to be a blocker for you.

I am getting below error when deploying on Ubuntu 14 LTS

(Backtrace restricted to imported tasks) cap aborted! SSHKit::Runner::ExecuteError: Exception while executing as [email protected]: bundle exit status: 5 bundle stdout: An error occurred while installing nokogiri (1.9.1), and Bundler cannot continue. Make sure that gem install nokogiri -v '1.9.1' succeeds before bundling. bundle stderr: /home/deploy/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/bundler-1.10.4/lib/bundler.rb:315: warning: Insecure world writable dir /home/deploy/.rbenv/versions/2.2.2/bin in PATH, mode 040777

Caused by: SSHKit::Command::Failed: bundle exit status: 5 bundle stdout: An error occurred while installing nokogiri (1.9.1), and Bundler cannot continue. Make sure that gem install nokogiri -v '1.9.1' succeeds before bundling. bundle stderr: /home/deploy/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/bundler-1.10.4/lib/bundler.rb:315: warning: Insecure world writable dir /home/deploy/.rbenv/versions/2.2.2/bin in PATH, mode 040777

Tasks: TOP => deploy:updated => bundler:install (See full trace by running task with --trace) The deploy has failed with an error: Exception while executing as [email protected]: bundle exit status: 5 bundle stdout: An error occurred while installing nokogiri (1.9.1), and Bundler cannot continue. Make sure that gem install nokogiri -v '1.9.1' succeeds before bundling. bundle stderr: /home/deploy/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/bundler-1.10.4/lib/bundler.rb:315: warning: Insecure world writable dir /home/deploy/.rbenv/versions/2.2.2/bin in PATH, mode 040777

The problem, buried in this log, is:

Cannot create temporary file in /tmp/: No space left on device

I'm going to close this for now, but let us know if you have more trouble after resolving the diskspace issue.

Please don't merge this. See 0ebf963 for information on why simplecov is pinned to v0.17.x.

@khanimdad Hi, sorry you're having trouble. I'll try to help.

When you submitted this issue, you ignored the template that asks questions relevant to installation problems. In order to best help you, I have to ask you to answer those questions. Here they are:

Have you read and followed the installation tutorial at http://www.nokogiri.org/tutorials/installing_nokogiri.html?

What is the complete output of gem install?

Please make sure to escape the file contents with triple-backticks.

What are the complete contents of the mkmf.log file generated during the failed installation?

Please make sure to escape the file contents with triple-backticks.

What operating system are you using?

Are you using Linux, MacOS, Windows, a BSD variant, or something else entirely?

If MacOS, please note the version of XCode you have installed.

If Linux, please include the output of uname -a and the contents of /etc/lsb-release.

If Windows, please note whether you're installing the precompiled gems, or compiling yourself with DevKit. Also please note the version of RubyInstaller you've installed.