profile
viewpoint
Mike Dalessio flavorjones Pivotal / Cloud Foundry New York City / New Jersey http://mike.daless.io/ Ummmmm

eschulte/rinari 413

Rinari Is Not A Rails IDE (it is an Emacs minor mode for Rails)

flavorjones/chromedriver-helper 316

Deprecated in favor of the `webdrivers` gem.

brynary/dryopteris 44

HTML sanitization using Nokogiri

flavorjones/arel 9

A Relational Algebra

flavorjones/calendar-assistant 9

Command-line tool to manage your Google Calendar

flavorjones/aintablog 4

Open source tumblog/aggregator.

flavorjones/active_hash 2

A readonly ActiveRecord-esque base class that lets you use a hash, a Yaml file or a custom file as the datasource

flavorjones/acts_as_fu 1

When you need a dash of ActiveRecord and nothing more

issue commentsparklemotion/nokogiri

Native gems: create releases for additional darwin platforms

@odlp Awesome! Thank you so much for giving it a try, this is really good news. Look for an official universal-darwin gem in RC4!

flavorjones

comment created time in 4 days

PR opened sparklemotion/nokogiri

gem: attempt to build a universal-darwin gem

related to #2079

See #2075 and #2079 for context. This branch attempts to generate a universal-darwin gem, following the patterns in the grpc gem.

+16 -15

0 comment

3 changed files

pr created time in 6 days

create barnchsparklemotion/nokogiri

branch : 2079-flavorjones-universal-darwin-gems

created branch time in 6 days

issue commentsparklemotion/nokogiri

Feedback and final to-do items for precompiled native gems on Linux and OSX

@odlp and others running Darwin 18 or earlier: please jump over to #2079 where I've got a gem file I need tested. Thanks!

flavorjones

comment created time in 6 days

issue commentsparklemotion/nokogiri

Native gems: create releases for additional darwin platforms

@odlp - I've created a universal-darwin gem and put it in a public storage bucket here:

https://storage.googleapis.com/nokogiri-org-concourse/test-gems/nokogiri-1.11.0.rc3-universal-darwin.gem

Can I ask that you try this on your darwin18 system by:

  • downloading the file from there,
  • check the md5 checksum is f8eb952cdad60740e6d411b40241aade,
  • run gem install --local path/to/nokogiri-1.11.0.rc3-universal-darwin.gem
  • run nokogiri -v

?

flavorjones

comment created time in 6 days

issue commentsparklemotion/nokogiri

Native gems: create releases for additional darwin platforms

Did some poking around this morning. I noticed that the grpc gem only ships a universal-darwin gem, and that this Just Works™ on my mac running either ruby universal.x86_64-darwin19 or x86-64-darwin19. This works because of this logic in Gem::Platform.

Lesson: we don't need to ship individual gem files for each version of darwin.

Next step: let's take a look at how grpc is built. Optionally consider setting up osxcross in a linux container.

flavorjones

comment created time in 6 days

pull request commentsparklemotion/nokogiri

Nokogiri.parse accepts Pathname object as argument, but only work with small files then.

See additional context from user help request at https://gitter.im/sparklemotion/nokogiri?at=5f5a485bf969413294d84734

phokz

comment created time in 9 days

issue closedsparklemotion/nokogiri

libgcrypt.so.11 required by nokogiri 1.10.9 not found for centos 8

After migrating ruby application to centos 8, application won't start with following error : .../vendor/bundle/ruby/2.6.0/gems/nokogiri-1.10.9/lib/nokogiri.rb:32:in `require': libgcrypt.so.11: cannot open shared object file: No such file or directory - .../vendor/bundle/ruby/2.6.0/gems/nokogiri-1.10.9/lib/nokogiri/nokogiri.so (LoadError)

Libgcrypt is installed on system but libgcrypt.so.11 is not available on centos 8

To Reproduce

Install rails application with nokogiri on Centos 8.2

Environment

[root@...]# nokogiri -v
Traceback (most recent call last):
    5: from .../vendor/bundle/bin/nokogiri:29:in `<main>'
    4: from .../vendor/bundle/bin/nokogiri:29:in `load'
    3: from .../vendor/bundle/ruby/2.6.0/gems/nokogiri-1.10.9/bin/nokogiri:6:in `<top (required)>'
    2: from .../vendor/bundle/ruby/2.6.0/gems/nokogiri-1.10.9/bin/nokogiri:6:in `require'
    1: from .../vendor/bundle/ruby/2.6.0/gems/nokogiri-1.10.9/lib/nokogiri.rb:30:in `<top (required)>'
.../vendor/bundle/ruby/2.6.0/gems/nokogiri-1.10.9/lib/nokogiri.rb:30:in `require': cannot load such file -- nokogiri/2.6/nokogiri (LoadError)
    6: from .../vendor/bundle/bin/nokogiri:29:in `<main>'
    5: from .../vendor/bundle/bin/nokogiri:29:in `load'
    4: from .../vendor/bundle/ruby/2.6.0/gems/nokogiri-1.10.9/bin/nokogiri:6:in `<top (required)>'
    3: from .../vendor/bundle/ruby/2.6.0/gems/nokogiri-1.10.9/bin/nokogiri:6:in `require'
    2: from .../vendor/bundle/ruby/2.6.0/gems/nokogiri-1.10.9/lib/nokogiri.rb:28:in `<top (required)>'
    1: from .../vendor/bundle/ruby/2.6.0/gems/nokogiri-1.10.9/lib/nokogiri.rb:32:in `rescue in <top (required)>'
.../vendor/bundle/ruby/2.6.0/gems/nokogiri-1.10.9/lib/nokogiri.rb:32:in `require': libgcrypt.so.11: cannot open shared object file: No such file or directory - .../vendor/bundle/ruby/2.6.0/gems/nokogiri-1.10.9/lib/nokogiri/nokogiri.so (LoadError)

closed time in 10 days

fmonvoisin

issue commentsparklemotion/nokogiri

libgcrypt.so.11 required by nokogiri 1.10.9 not found for centos 8

Closing because OP hasn't responded. Please feel free to comment if you'd like to re-open the discussion!

fmonvoisin

comment created time in 10 days

issue commentsparklemotion/nokogiri

Native gems: create releases for additional darwin platforms

Aaaand a more complete Dockerfile using osxcross at https://github.com/appPlant/docker-images/blob/mruby-cli/Dockerfile

flavorjones

comment created time in 10 days

issue commentsparklemotion/nokogiri

Native gems: centos6 support

tagging @twk3 so he can follow along as the original reporter

flavorjones

comment created time in 10 days

issue commentsparklemotion/nokogiri

Feedback and final to-do items for precompiled native gems on Linux and OSX

@twk3 I've created #2081 to investigate the Centos6 issue you reported above. Thanks again!

flavorjones

comment created time in 10 days

issue openedsparklemotion/nokogiri

Native gems: centos6 support

(Originally mentioned at https://github.com/sparklemotion/nokogiri/issues/2075#issuecomment-689063142)

On Centos6, running a version of Ruby compiled like this, we see the following error message when loading the nokogiri.so shared library:

/lib64/libc.so.6: version `GLIBC_2.14' not found (required by /usr/local/lib/ruby/gems/2.6.0/gems/nokogiri-1.11.0.rc3-x86_64-linux/lib/nokogiri/2.6/nokogiri.so)

I'm not deeply familiar with our glibc requirements, but if we go back to the early days of Linux native gems, in #1571, @lars mentions that we need GLIBC 2.17 or later (and we check for this here).

It looks like CentOS/6.10 is based on glibc 2.12 which might explain why this isn't working -- but the error message is a little confusing if this is the case. @larskanis can you take a look and confirm my understanding?

To Reproduce

host $ docker pull registry.gitlab.com/gitlab-org/gitlab-omnibus-builder/centos_6:0.0.72
...

host $ docker run -t registry.gitlab.com/gitlab-org/gitlab-omnibus-builder/centos_6:0.0.72 /bin/bash

guest # gem install --prerelease nokogiri
Fetching: nokogiri-1.11.0.rc3-x86_64-linux.gem (100%)
Nokogiri is built with the packaged libraries: libxml2-2.9.10, libxslt-1.1.34, zlib-1.2.11, libiconv-1.15.
Successfully installed nokogiri-1.11.0.rc3-x86_64-linux
Parsing documentation for nokogiri-1.11.0.rc3-x86_64-linux
Installing ri documentation for nokogiri-1.11.0.rc3-x86_64-linux
Done installing documentation for nokogiri after 2 seconds
1 gem installed

guest # nokogiri -v
# nokogiri -v
Traceback (most recent call last):
	7: from /usr/local/bin/nokogiri:23:in `<main>'
	6: from /usr/local/bin/nokogiri:23:in `load'
	5: from /usr/local/lib/ruby/gems/2.6.0/gems/nokogiri-1.11.0.rc3-x86_64-linux/bin/nokogiri:6:in `<top (required)>'
	4: from /usr/local/lib/ruby/site_ruby/2.6.0/rubygems/core_ext/kernel_require.rb:55:in `require'
	3: from /usr/local/lib/ruby/site_ruby/2.6.0/rubygems/core_ext/kernel_require.rb:55:in `require'
	2: from /usr/local/lib/ruby/gems/2.6.0/gems/nokogiri-1.11.0.rc3-x86_64-linux/lib/nokogiri.rb:13:in `<top (required)>'
	1: from /usr/local/lib/ruby/site_ruby/2.6.0/rubygems/core_ext/kernel_require.rb:55:in `require'
/usr/local/lib/ruby/site_ruby/2.6.0/rubygems/core_ext/kernel_require.rb:55:in `require': /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /usr/local/lib/ruby/gems/2.6.0/gems/nokogiri-1.11.0.rc3-x86_64-linux/lib/nokogiri/2.6/nokogiri.so) - /usr/local/lib/ruby/gems/2.6.0/gems/nokogiri-1.11.0.rc3-x86_64-linux/lib/nokogiri/2.6/nokogiri.so (LoadError)
	8: from /usr/local/bin/nokogiri:23:in `<main>'
	7: from /usr/local/bin/nokogiri:23:in `load'
	6: from /usr/local/lib/ruby/gems/2.6.0/gems/nokogiri-1.11.0.rc3-x86_64-linux/bin/nokogiri:6:in `<top (required)>'
	5: from /usr/local/lib/ruby/site_ruby/2.6.0/rubygems/core_ext/kernel_require.rb:55:in `require'
	4: from /usr/local/lib/ruby/site_ruby/2.6.0/rubygems/core_ext/kernel_require.rb:55:in `require'
	3: from /usr/local/lib/ruby/gems/2.6.0/gems/nokogiri-1.11.0.rc3-x86_64-linux/lib/nokogiri.rb:11:in `<top (required)>'
	2: from /usr/local/lib/ruby/gems/2.6.0/gems/nokogiri-1.11.0.rc3-x86_64-linux/lib/nokogiri.rb:15:in `rescue in <top (required)>'
	1: from /usr/local/lib/ruby/site_ruby/2.6.0/rubygems/core_ext/kernel_require.rb:55:in `require'
/usr/local/lib/ruby/site_ruby/2.6.0/rubygems/core_ext/kernel_require.rb:55:in `require': libruby.so.2.5: cannot open shared object file: No such file or directory - /usr/local/lib/ruby/gems/2.6.0/gems/nokogiri-1.11.0.rc3-x86_64-linux/lib/nokogiri/nokogiri.so (LoadError)

created time in 10 days

issue commentsparklemotion/nokogiri

Native gems: create releases for additional darwin platforms

Example of how this was used in the past in a docker container: https://github.com/hone/mruby-cli-docker/blob/master/Dockerfile which is used by the https://github.com/hone/mruby-cli project.

flavorjones

comment created time in 10 days

issue commentsparklemotion/nokogiri

Native gems: create releases for additional darwin platforms

@luislavena I'm going to use osxcross to experiment with creating a rake-compiler-dock-compatible image for OSX!

flavorjones

comment created time in 10 days

issue commentsparklemotion/nokogiri

Native gems: create releases for additional darwin platforms

It looks like this might be helpful, too: https://github.com/tpoechtrager/osxcross which is a cross-compiler toolchain for OSX.

flavorjones

comment created time in 10 days

pull request commentsparklemotion/nokogiri

Issue reproduction for ISO-8859-1 parsing on JRuby

OK - I ran a git bisect and ba16682aae2cbd42c196bd8afdfcfe8a5d82fbdb introduced this regression. Tagging @jvshahid for his awareness :two_hearts:

nokogiri $ git bisect good
ba16682aae2cbd42c196bd8afdfcfe8a5d82fbdb is the first bad commit
commit ba16682aae2cbd42c196bd8afdfcfe8a5d82fbdb
Author: John Shahid <jvshahid@gmail.com>
Date:   Thu Apr 18 13:58:18 2019 -0400

    Split setInputSource into setIOInputSource and setStringInputSource
    
    This commit also consolidates some of the encoding handling logic that was
    repeated in multiple places.

 ext/java/nokogiri/HtmlDocument.java              |  25 +++---
 ext/java/nokogiri/HtmlSaxParserContext.java      | 100 ++++++-----------------
 ext/java/nokogiri/XmlDocument.java               |  33 +++-----
 ext/java/nokogiri/XmlSaxParserContext.java       |  14 ++--
 ext/java/nokogiri/internals/NokogiriHelpers.java |  12 +++
 ext/java/nokogiri/internals/ParserContext.java   |  71 ++++++----------
 6 files changed, 94 insertions(+), 161 deletions(-)

thbar

comment created time in 11 days

issue commentsparklemotion/nokogiri

Feedback and final to-do items for precompiled native gems on Linux and OSX

@twk3 How did you compile/build/install that version of Ruby? Looks like yum install ruby on centos6 installs Ruby 1.8.7 (!). I'd like to reproduce what you're seeing.

flavorjones

comment created time in 11 days

issue commentsparklemotion/nokogiri

Feedback and final to-do items for precompiled native gems on Linux and OSX

@twk3 Interesting! Thank you for reporting this, I'll take a look.

flavorjones

comment created time in 11 days

pull request commentsparklemotion/nokogiri

Issue reproduction for ISO-8859-1 parsing on JRuby

Thank you for submitting this! Fantastic executable bug report. I appreciate it.

thbar

comment created time in 11 days

issue commentsparklemotion/nokogiri

Feedback and final to-do items for precompiled native gems on Linux and OSX

@thbar Thanks for testing this. If you need to, please open a new GH issue for the jruby problems -- I don't think they're going to be related to the native gem work for Linux and OSX. I'd like to know more about what you're seeing, though!

flavorjones

comment created time in 11 days

issue commentsparklemotion/nokogiri

Native gems: create releases for additional darwin platforms

Note that @drbrain says: "Travis has older Xcode for linking on older Darwin, I use it for building work stuff for MacOS" at https://twitter.com/drbrain/status/1303369620099751936

flavorjones

comment created time in 11 days

issue openedsparklemotion/nokogiri

Native gems: create releases for additional darwin platforms

Nokogiri v1.11.0.rc3 only supports OSX x86_64-darwin19 platform.

At least one commented at #2075 has asked about x86_64-darwin18. If that's a common platform we should consider shipping that, as well.

I'll be honest and say that my inability to virtualize OSX is a real challenge here. I have a work laptop that's darwin19, and I have an 2009 mac mini that might have darwin18 on it (which looks like it's x86_64).

created time in 11 days

issue openedsparklemotion/nokogiri

Native gems: ensure that mini_portile2 is not a dependency

When we ship precompiled native gems, we no longer need mini_portile2 as an install-time dependency.

Previously @larskanis and I did some work on this, resulting in:

  • https://github.com/rake-compiler/rake-compiler/pull/171
  • cebf400 (workaround)
  • 33e5b42 (workaround)
  • 94ecf99 (workaround)

The workarounds are brittle; this didn't seem to work for the Darwin native gem. Let's figure out a way to do this reliably.

Related to #2075.

created time in 11 days

issue openedsparklemotion/nokogiri

Native gems: remove the C source code from the precompiled gem

Because the gem is precompiled, there's no need to also ship the C source and header files.

Package the gem without the C code.

Related to #2075.

created time in 11 days

issue openedsparklemotion/nokogiri

Native gems: eliminate the "extra" dll

When building a native Linux or OSX gem, we package a fifth nokogiri.so or nokogiri.bundle that appears to be generated by the "host" ruby (not "cross-compiled").

This additional shared library file should not be present.

Related to #2075

created time in 11 days

issue commentsparklemotion/nokogiri

Feedback and final to-do items for precompiled native gems on Linux and OSX

@odlp - Thanks for giving it a try! I'm not building darwin18 gems at this point, because I don't have a system available to do so. Only darwin19 is supported right now.

flavorjones

comment created time in 11 days

push eventsparklemotion/nokogiri.org

Mike Dalessio

commit sha 904f09acd33395dd3975c60ee3e0eac7f387a914

Deployed 2cffa78 with MkDocs version: 1.1.2

view details

push time in 11 days

push eventsparklemotion/nokogiri

Mike Dalessio

commit sha f384738b3849f13e7753aa59a7ebd95246ee36ad

CHANGELOG: markdown links

view details

push time in 11 days

release sparklemotion/nokogiri

v1.11.0.rc3

released time in 11 days

created tagsparklemotion/nokogiri

tagv1.11.0.rc3

Nokogiri (鋸) is a Rubygem providing HTML, XML, SAX, and Reader parsers with XPath and CSS selector support.

created time in 11 days

push eventsparklemotion/nokogiri

Mike Dalessio

commit sha 7b0c056ea7b94edbddce1906601aed03a6819619

dev: update .hoerc to ignore issue- and sorbet-related temp dirs

view details

Mike Dalessio

commit sha 959db1d8a6d6afaaa5e2cef554492e367f1791ff

version bump to v1.11.0.rc3

view details

push time in 11 days

issue closedsparklemotion/nokogiri

Ship precompiled gems for linux

Hello! If you're arriving here to give feedback on the v1.11.0 Release Candidates, please make sure to:

  • tell us which linux distro you're using!
  • (If you can) a docker image in which we can reproduce the failure you're seeing!

Thanks so much for using Nokogiri and for giving us feedback on this experiment!


See #1571 for the PR from @larskanis that enables building precompiled libraries for linux.

This issue is for discussion on how to ship those precompiled libraries in a gem for linux users in a way that won't break the installation experience for people.

Some things that we'll need to figure out were mentioned in #1571 by @larskanis:

  1. Bundler isn't clever at selection of unsupported ruby versions. See https://github.com/bundler/bundler/pull/6247 . This requires users to explicit opt-out of binary gems, when they want to run nokogiri on ruby-trunk.

  2. There is a known issue on Alpine Linux: https://github.com/rake-compiler/rake-compiler-dock/issues/20 . I did a manual test on the alpine docker image and it reproducible fails at Nokogiri::XML::Reader#namespaces somewhere within the tests: https://gist.github.com/larskanis/94920901e6466c9383360ee1346f9f34 . I didn't analyze the reason for the segfault so far, but I think we should before the release.

And he later mentioned:

Issue 1 in the above comment will probably finally be fixed by the next bundler release. See rubygems/bundler#7522 . This way bundler will select the platform=ruby gem automatically, if fat binary gem doesn't fit to the ruby release.

I think we'll probably also be able to fix / work around issue 2, so that I hope we'll be able to ship binary releases some day.

Finally, I'd like for VersionInfo to contain some indication that the user is using precompiled libraries. A patch that worked at one point on the #1571 branch (but now needs to be rewritten) is:

From 20f69908732b50258be0ee18e1d78b841d462dc4 Mon Sep 17 00:00:00 2001
From: Mike Dalessio <mike.dalessio@gmail.com>
Date: Sun, 13 Jan 2019 01:15:06 -0500
Subject: [PATCH] `nokogiri -v` emits info about precompiled binaries

It may be useful information to be able to tell if the libraries were
compiled on installation, or whether they were precompiled in a fat
binary native gem.
---
 Rakefile                | 3 +++
 ext/nokogiri/nokogiri.c | 6 ++++++
 lib/nokogiri/version.rb | 6 ++++++
 3 files changed, 15 insertions(+)

diff --git a/Rakefile b/Rakefile
index 5ef937d..72edb7e 100644
--- a/Rakefile
+++ b/Rakefile
@@ -383,6 +383,9 @@ task :cross do
     raise "rake-compiler has not installed any cross rubies. Use rake-compiler-dock or 'rake gem:windows' for building binary windows gems."
   end
 
+  ENV['CFLAGS'] ||= ""
+  ENV['CFLAGS'] += " -DNOKOGIRI_USE_PRECOMPILED_NATIVE"
+
   CROSS_RUBIES.each do |cross_ruby|
     task "tmp/#{cross_ruby.platform}/nokogiri/#{cross_ruby.ver}/nokogiri.so" do |t|
       # To reduce the gem file size strip mingw32 dlls before packaging
diff --git a/ext/nokogiri/nokogiri.c b/ext/nokogiri/nokogiri.c
index 740b30d..5b32500 100644
--- a/ext/nokogiri/nokogiri.c
+++ b/ext/nokogiri/nokogiri.c
@@ -104,6 +104,12 @@ void Init_nokogiri()
   rb_const_set(mNokogiri, rb_intern("NOKOGIRI_LIBXSLT_PATCHES"), Qnil);
 #endif
 
+#ifdef NOKOGIRI_USE_PRECOMPILED_NATIVE
+  rb_const_set(mNokogiri, rb_intern("NOKOGIRI_USE_PRECOMPILED_NATIVE"), Qtrue);
+#else
+  rb_const_set(mNokogiri, rb_intern("NOKOGIRI_USE_PRECOMPILED_NATIVE"), Qfalse);
+#endif
+
 #ifdef LIBXML_ICONV_ENABLED
   rb_const_set(mNokogiri, rb_intern("LIBXML_ICONV_ENABLED"), Qtrue);
 #else
diff --git a/lib/nokogiri/version.rb b/lib/nokogiri/version.rb
index 44c076a..c05e1a0 100644
--- a/lib/nokogiri/version.rb
+++ b/lib/nokogiri/version.rb
@@ -46,6 +46,10 @@ module Nokogiri
       NOKOGIRI_USE_PACKAGED_LIBRARIES
     end
 
+    def libxml2_using_precompiled_native?
+      NOKOGIRI_USE_PRECOMPILED_NATIVE
+    end
+
     def warnings
       warnings = []
 
@@ -78,6 +82,7 @@ module Nokogiri
           vi["libxml"] = {}.tap do |libxml|
             if libxml2_using_packaged?
               libxml["source"] = "packaged"
+              libxml["precompiled"] = libxml2_using_precompiled_native?
               libxml["patches"] = NOKOGIRI_LIBXML2_PATCHES
             else
               libxml["source"] = "system"
@@ -89,6 +94,7 @@ module Nokogiri
           vi["libxslt"] = {}.tap do |libxslt|
             if libxml2_using_packaged?
               libxslt["source"] = "packaged"
+              libxslt["precompiled"] = libxml2_using_precompiled_native?
               libxslt["patches"] = NOKOGIRI_LIBXSLT_PATCHES
             else
               libxslt["source"] = "system"
-- 
2.17.1

And tagging @tenderlove for visibility because he had some thoughts on the previous #1571 thread.

closed time in 11 days

flavorjones

issue commentsparklemotion/nokogiri

Ship precompiled gems for linux

Closing this issue. RC3 should be out shortly, and I'm pointing people to https://github.com/sparklemotion/nokogiri/issues/2075 to provide feedback.

flavorjones

comment created time in 11 days

issue closedsparklemotion/nokogiri

Explore precompiled binaries for Darwin (OSX)

This is a placeholder for a future conversation (with myself?) about precompiling Darwin libraries, similarly to how we've shipped precompiled libraries for Windows since ~v1.4 and how we will hopefully ship for Linux in v1.11.0.

cc @larskanis @tenderlove tell me if you think I'm crazy for thinking we can do this.


Here's what I've learned so far

(I'll keep this section up to date as I go along)

Architecture names relevant to us

  • RUBY_PLATFORM
    • used to identify the ruby architecture
    • which on my linux system is "x86_64-linux"
    • using OSX system Ruby is "universal.x86_64-darwin19"
    • using OSX homebrew- or rvm-installed Ruby is "x86_64-darwin19"
  • Gem::Platform.new(RUBY_PLATFORM)
    • used by rake-compiler to name native tasks
    • on my linux system is "x86_64-linux"
    • using OSX system Ruby is "universal.x86_64-darwin-19" (note the addition of a "-" before the "19")
    • using OSX homebrew- or rvm-installed Ruby is "x86_64-darwin-19" (note the "-" here also)
  • Gem::Platform.local
    • used by rubygems to search for an appropriate native platform gem
    • on my linux system is "x86_64-linux"
    • using OSX system Ruby is "universal-darwin-19"
    • using OSX homebrew- or rvm-installed Ruby is "x86_64-darwin-19"

Needed changes to Nokogiri to support OSX

Invoke the properly-named gem task:

diff --git a/tasks/cross-ruby.rb b/tasks/cross-ruby.rb
index 35c098f..1d5c2fd 100644
--- a/tasks/cross-ruby.rb
+++ b/tasks/cross-ruby.rb
@@ -179,7 +181,7 @@ namespace "gem" do
       task "guest" do
         # use Task#invoke because the pkg/*gem task is defined at runtime
         Rake::Task["native:#{plat}"].invoke
-        Rake::Task["pkg/#{HOE.spec.full_name}-#{plat}.gem"].invoke
+        Rake::Task["pkg/#{HOE.spec.full_name}-#{Gem::Platform.new(plat).to_s}.gem"].invoke
       end
     end
   end

Compatibility

If Ruby is configured with --enable-shared, the shared object libraries generated will contain references to libruby (and possibly other libraries like libgmp, libcrypt, and librt).

This means that a shared object library built by a ruby that was configured with --enabled-shared can't be run by a ruby that was configured with --disabled-shared. However, a shared object library built by a ruby that was configured with --disable-shared seems to run fine in a ruby configured with --enable-shared.

It's possible to hack the shared object files to get around this. See here for how rake-compiler-dock gets around this:

  • https://github.com/rake-compiler/rake-compiler-dock/blob/master/Dockerfile.mri.erb#L103-L110
  • https://github.com/rake-compiler/rake-compiler-dock/issues/13

And it's possible on a Mac to modify the .bundle file path to contain @rpath and then make sure that DYLD_LIBRARY_PATH is set appropriately. Some resources on this topic:

  • https://medium.com/@donblas/fun-with-rpath-otool-and-install-name-tool-e3e41ae86172
  • https://developer.apple.com/library/archive/documentation/DeveloperTools/Conceptual/DynamicLibraries/100-Articles/OverviewOfDynamicLibraries.html#//apple_ref/doc/uid/TP40001873-SW1
  • https://wincent.com/wiki/@executable_path,@load_path_and@rpath

But these seem like unnatural workarounds, and so I'm going to attempt to use a Ruby built with --disable-shared to generate the precompiled native gem for darwin.

  • [x] Publish the recipe for building that version of Ruby so that it's reproducible.

Universal architecture

The OSX system ruby has an architecture of "universal.x86_64" which ... I'm not sure what that is. I started googling and found:

  • https://docs.elementscompiler.com/Platforms/Cocoa/CpuArchitectures/

which implies that Universal is a binary that would work for both x86_64 and arm 64.

closed time in 11 days

flavorjones

issue commentsparklemotion/nokogiri

Explore precompiled binaries for Darwin (OSX)

Closing. Will cut an rc3 with darwin native gems, and will point people to https://github.com/sparklemotion/nokogiri/issues/2075 to provide feedback.

flavorjones

comment created time in 11 days

issue openedsparklemotion/nokogiri

Feedback and final to-do items for precompiled native gems on Linux and OSX

This issue exists to capture feedback and action items about the precompiled native gems for Linux and OSX that we're shipping in the Nokogiri v1.11 release candidates (prereleases).

(note: at the time of opening this issue, rc3 is not yet out with OSX native gems. i opened this so I could reference it in the changelog for rc3. shouldn't be more than an hour or so, hang tight.)

How do I try this out?

We're only shipping native gems for the following platforms (run bundle platform or ruby -e 'puts Gem::Platform.local.to_s' to identify your system):

  • New platforms as of v1.11 release candidates:
    • Linux: x86-linux and x86_64-linux
    • OSX/Darwin: x86_64-darwin19
  • Previously existing native gems (note these have already been available for many years):
    • Windows: x86-mingw32 and x64-mingw32
    • JRuby: java (note this native gem has already been available for many years)

If you're on Linux or OSX, try updating your Gemfile with:

gem "nokogiri", "~> 1.11.0.rc3"`

And let us know how it works for you in the comments!

Providing Feedback

If you love this idea in a vague general sense, please give this comment a :heart:.

If you tried native gems on Linux or OSX and they worked just fine, please give this comment a :+1:.

If you're having problems installing or using Nokogiri's precompiled native gems, please comment on this issue with the following information:

  • output from ruby -v
  • output from gem -v
  • output from gem env
  • if you're using bundler,
    • output from bundle version
    • output from bundle config (take care to redact any personal info or credentials)
  • output from gem install or bundle install, including the command you ran

Why is a precompiled native gem useful?

Installing a precompiled native gem, installation takes about 1 second, and avoids the challenging bits of how to set up the compiler toolchain on your development system. This will reduce the support burden for Team Nokogiri and allow us to move a bit faster on bugfixes and features.

For comparison, compiling and installing the gem (and its vendored dependencies) from source, when only using a single core, my laptop takes 52 seconds. When using all 8 cores, this is cut down to 32 seconds.

Why would I not want to use a precompiled native gem?

I can imagine some folks might have trust issues; if this is you, please let us know in a comment here (or in #2013) what we could do to increase that trust (e.g., I can imagine providing a chain of custody including public build logs with cryptographic hashes of artifacts).

If you have other reasons for not wanting to use a precompiled native gem, let us know in the comments.

How can I avoid using a precompiled native gem?

If you want to avoid using a precompiled gem in your project, do one of the following:

  • If you're not using bundler, then run gem install nokogiri -v 1.11.0.rc3 --platform=ruby
  • If you are using bundler, then run bundle config set force_ruby_platform true and you'll never use a precompiled native gem

Shout-outs

Thanks to @larskanis for all the time he spends maintaining the gems rake-compiler and rake-compiler-dock, and for being the one who wrote the first PR for Linux native gem support.

Thanks to the GRPC team who already ships native gems (see https://github.com/grpc/grpc). They blazed the trail and provided inspiration and a how-to manual for some edge cases.

Thanks to @tenderlove who paired with me on the nasty OSX bits that I was struggling with.

Resources

  • #1571 is the original PR from @larskanis for Linux native gems
  • #1983 is an issue to polish Linux native gem support
  • #2063 is an issue researching OSX/Darwin native gem support
  • #2073 is a PR introducing OSX/Darwin native gem support
  • #2013 is an issue to increase trust in published gems

Action Items

These are the remaining action items we need before a final release includes precompiled native gems for Linux and OSX:

  • [ ] ship rc3
    • [ ] update CHANGELOG language to point to this issue
    • [ ] email ruby-talk and nokogiri-talk to ask for feedback
    • [ ] tweet to ask for feedback
  • [ ] update LICENSE-DEPENDENCIES.txt to reflect reality for all these different combinations of releases
  • [ ] eliminate the "extra" dll being shipped as lib/nokogiri/nokogiri.{bundle,so}
  • [ ] make sure Nokogumbo still installs cleanly (see https://github.com/rubys/nokogumbo/pull/148 and https://github.com/sparklemotion/nokogiri/pull/1788)
  • [ ] note in VersionInfo that Nokogiri is precompiled
  • [ ] finish most of the issues under the v1.11.0 milestone

created time in 11 days

CommitCommentEvent

delete branch sparklemotion/nokogiri

delete branch : 2063-darwin-native-gem

delete time in 11 days

push eventsparklemotion/nokogiri

Mike Dalessio

commit sha 8e5fa3313277e08c15e4c1bb74b31b5c80ba311c

gem: get ready for darwin where the gem platform name is different from RUBY_PLATFORM. - call the gem task correctly - have `nokogiri -v` display Gem::Platform.local

view details

Mike Dalessio

commit sha 08f6a4cb34c084155082157bfeb64effafc2b351

gem: move RakeExtension tasks from Rakefile into tasks/cross-ruby.rb so all our cross-compilation and native gem logic is in one place

view details

Mike Dalessio

commit sha 69ec3ec0f6d9a53417dd4a9f0143d74acd12c656

gem: add darwin gems to .cross_rubies - update CrossRuby to recognize it - improve the parsing code for .cross_rubies - only create rake-compiler-docker rake tasks for windows and linux

view details

Mike Dalessio

commit sha 2202c479684d4346b7d03af629acb80693c8b1e8

gem: placeholder for darwin .bundle checks

view details

Mike Dalessio

commit sha ac1c1fe426d80008aa773731121df91609329501

dev: move build_all into scripts/

view details

Mike Dalessio

commit sha d0cf1412c61541a5766a58e0b328c79ec1d4453c

gem: setup-osx-native-builders to set up rubies for native gem

view details

Mike Dalessio

commit sha 94ecf99ea2c113648c5650be77dd82657c224598

gem: rake tasks for native darwin gem and make sure we gitignore the .bundle files

view details

Mike Dalessio

commit sha 1e6e972248b4b0171659ba7f77ec4dbc3ee5e42a

dev: clean up verify_dll tests of native gem shared libraries

view details

Mike Dalessio

commit sha 7d9b7ca2c7b540ffad44fa68d5d2a5a6b7143025

dev: move dll staging path logic into CrossRuby

view details

Mike Dalessio

commit sha d0befd1bb4b78a657550aede3f82b4b314cfc154

dev: clean up CrossRuby and prepare for darwin verify_dll

view details

Mike Dalessio

commit sha 96de09fb0e13eaf4503044878d42c935b90004d5

gem: improve verify_dll so linux does same checks as windows

view details

Mike Dalessio

commit sha 0a495a477fd4d82e522014799b61694a81cc209a

gem: verify_dll for darwin native gem libraries

view details

Mike Dalessio

commit sha ade7ec17b510c57832d8de13c65fdd7884c6e57a

gem: verify_dll now checks for allowed references and we need to optionally allow the objc libraries

view details

Mike Dalessio

commit sha 5c0fbfa1829f1b3f11cc1c92e3e93b56280685a3

Merge pull request #2073 from sparklemotion/2063-darwin-native-gem Darwin native gems

view details

push time in 11 days

PR merged sparklemotion/nokogiri

Reviewers
Darwin native gems

What problem is this PR intended to solve?

Proposed work for #2063 -- precompiled native gems for OSX/Darwin.

Have you included adequate test coverage?

Needs:

  • [x] implement verify_dll for the darwin .bundle

Does this change affect the C or the Java implementations?

No functional changes -- only developer changes in how we build gems.

+257 -118

4 comments

9 changed files

flavorjones

pr closed time in 11 days

pull request commentsparklemotion/nokogiri

Darwin native gems

I'm going to merge this so I can ship an RC and get feedback. Would still love to hear anybody's feedback about this!

flavorjones

comment created time in 11 days

PR closed sparklemotion/nokogiri

Create gh pr meta/spam

Thank you for contributing to Nokogiri! To help us prioritize, please take care to answer the questions below when you submit this pull request.

The Nokogiri core team work off of master, so please submit all PRs based on the master branch. We generally will cherry-pick relevant bug fixes onto the current release branch.

What problem is this PR intended to solve?

If there is an existing issue that describes this, feel free to simply link to that issue.

Otherwise, please provide enough context for the Nokogiri maintainers to understand your intent.

Have you included adequate test coverage?

We have a thorough test suite that allows us to create releases confidently and prevent accidental regressions. Any proposed change in behavior must be accompanied by tests.

If possible, please try to write the tests so that they communicate intent.

Does this change affect the C or the Java implementations?

If you're proposing a change to the C implementation, has the behavior change been made to the Java code as well? And vice versa?

If not, that may be OK, just please note it here.

+1 -0

3 comments

1 changed file

ifuseandprint

pr closed time in 11 days

pull request commentsparklemotion/nokogiri

Create gh pr

Marked as spam, user blocked.

ifuseandprint

comment created time in 11 days

Pull request review commentsparklemotion/nokogiri

WIP: Darwin native gems

 def dlls         end),         "libc.so.6",       ]+    when DARWIN_PLATFORM_REGEX+      [+        "/usr/lib/libSystem.B.dylib",+        "/usr/lib/liblzma.5.dylib",+        "/usr/lib/libobjc.A.dylib",

@larskanis @tenderlove I could use a sanity check on these remaining references. This seems reasonable to me. I'll probably just ship an RC and see if we get feedback, but wanted to highlight that I made a decision with incomplete information here.

flavorjones

comment created time in 11 days

PullRequestReviewEvent

push eventsparklemotion/nokogiri

Mike Dalessio

commit sha d0cf1412c61541a5766a58e0b328c79ec1d4453c

gem: setup-osx-native-builders to set up rubies for native gem

view details

Mike Dalessio

commit sha 94ecf99ea2c113648c5650be77dd82657c224598

gem: rake tasks for native darwin gem and make sure we gitignore the .bundle files

view details

Mike Dalessio

commit sha 1e6e972248b4b0171659ba7f77ec4dbc3ee5e42a

dev: clean up verify_dll tests of native gem shared libraries

view details

Mike Dalessio

commit sha 7d9b7ca2c7b540ffad44fa68d5d2a5a6b7143025

dev: move dll staging path logic into CrossRuby

view details

Mike Dalessio

commit sha d0befd1bb4b78a657550aede3f82b4b314cfc154

dev: clean up CrossRuby and prepare for darwin verify_dll

view details

Mike Dalessio

commit sha 96de09fb0e13eaf4503044878d42c935b90004d5

gem: improve verify_dll so linux does same checks as windows

view details

Mike Dalessio

commit sha 0a495a477fd4d82e522014799b61694a81cc209a

gem: verify_dll for darwin native gem libraries

view details

Mike Dalessio

commit sha ade7ec17b510c57832d8de13c65fdd7884c6e57a

gem: verify_dll now checks for allowed references and we need to optionally allow the objc libraries

view details

push time in 11 days

push eventsparklemotion/nokogiri

Mike Dalessio

commit sha 9f19c7095692197dc6b0501d49aa4ddf642e6357

gem: improve verify_dll so linux does same checks as windows

view details

push time in 12 days

push eventsparklemotion/nokogiri

Mike Dalessio

commit sha d88c368cf512d0e646f07828493837a1236d13c9

dev: clean up CrossRuby and prepare for darwin verify_dll

view details

push time in 12 days

created tagflavorjones/concourse-gem

tagv0.36.0

Utility to ease management of Concourse pipelines. See https://concourse-ci.org/ to learn about Concourse.

created time in 12 days

push eventflavorjones/concourse-gem

Mike Dalessio

commit sha a9416526de402007f90f630c1b83df4ecdc04407

remove generated files from CLOBBER rake variable

view details

Mike Dalessio

commit sha 980289f12fd4678351990403919d41102a9475c7

version bump to v0.36.0

view details

push time in 12 days

push eventsparklemotion/nokogiri

Mike Dalessio

commit sha 34efda88f5c7dab9ed97403e0cca13352cde4c46

dev: clean up verify_dll tests of native gem shared libraries

view details

Mike Dalessio

commit sha dd9705cbb6c44ec8945e4f5363e2bc0d16a40a1c

dev: move dll staging path logic into CrossRuby

view details

push time in 12 days

push eventsparklemotion/nokogiri

Mike Dalessio

commit sha 2202c479684d4346b7d03af629acb80693c8b1e8

gem: placeholder for darwin .bundle checks

view details

Mike Dalessio

commit sha ac1c1fe426d80008aa773731121df91609329501

dev: move build_all into scripts/

view details

Mike Dalessio

commit sha 79ff218170206c8f8f99c70433860250755541ad

gem: setup-osx-native-builders to set up rubies for native gem

view details

Mike Dalessio

commit sha fc72c85053832fadd0714e725872f4f6daf0f79e

gem: rake tasks for native darwin gem and make sure we gitignore the .bundle files

view details

push time in 12 days

issue commentsparklemotion/nokogiri

Explore precompiled binaries for Darwin (OSX)

Update:

  • pull request at #2073 seems to work for x86_64-darwin for ruby 2.4, 2.5, 2.6, 2.7.
    • which includes a recipe for installing a set of rubies for building the native gems at scripts/setup-osx-native-builders
  • I won't ship a separate universal-darwin-19 gem unless and until people ask (and I'll note that the x86_64-darwin-19 gem works for the system ruby that self-identifies as universal-darwin-19, even if the platform strings don't match)
flavorjones

comment created time in 12 days

created tagflavorjones/concourse-gem

tagv0.35.0

Utility to ease management of Concourse pipelines. See https://concourse-ci.org/ to learn about Concourse.

created time in 12 days

push eventflavorjones/concourse-gem

Mike Dalessio

commit sha 21bfb8ea1c9e86fc567f92c1c860394370f8f9fa

ensure tests pass with new fly argument handling

view details

Mike Dalessio

commit sha 45eb181007ae9e2ef490a601c53c8036a225fe0b

remove take task "clean" this should have been done in v0.26.0

view details

Mike Dalessio

commit sha 04d3ffd7fdfd694c1d3bce96db6a7e76979c556c

version bump to v0.35.0

view details

push time in 12 days

push eventsparklemotion/nokogiri

Mike Dalessio

commit sha f8f1afdd8a0918c0a49f14907e4d9a0c2d20ea82

gem: setup-osx-native-builders to set up rubies for native gem

view details

Mike Dalessio

commit sha bbe0132b8b4c1493f54695d19e4fe1fd7efaf39c

gem: rake tasks for native darwin gem and make sure we gitignore the .bundle files

view details

push time in 12 days

push eventsparklemotion/nokogiri

Mike Dalessio

commit sha 8e5fa3313277e08c15e4c1bb74b31b5c80ba311c

gem: get ready for darwin where the gem platform name is different from RUBY_PLATFORM. - call the gem task correctly - have `nokogiri -v` display Gem::Platform.local

view details

Mike Dalessio

commit sha 08f6a4cb34c084155082157bfeb64effafc2b351

gem: move RakeExtension tasks from Rakefile into tasks/cross-ruby.rb so all our cross-compilation and native gem logic is in one place

view details

Mike Dalessio

commit sha 69ec3ec0f6d9a53417dd4a9f0143d74acd12c656

gem: add darwin gems to .cross_rubies - update CrossRuby to recognize it - improve the parsing code for .cross_rubies - only create rake-compiler-docker rake tasks for windows and linux

view details

Mike Dalessio

commit sha 1fc55451cbae048e165c02abfe624e1caf5b9308

gem: placeholder for darwin .bundle checks

view details

Mike Dalessio

commit sha 95748e3f60232ca13f8c7723cc58110e8b833874

dev: move build_all into scripts/

view details

Mike Dalessio

commit sha a232b9291f85d0d2729fb6188218694ad498bb26

gem: OSX script to set up rubies for native gem compilation

view details

Mike Dalessio

commit sha b81d9a62ecd956585662ffcf12e27136582238f7

gem: rake tasks for native darwin gem and make sure we gitignore the .bundle files

view details

push time in 12 days

PR opened sparklemotion/nokogiri

Reviewers
WIP: Darwin native gems

What problem is this PR intended to solve?

Proposed work for #2063 -- precompiled native gems for OSX/Darwin.

Have you included adequate test coverage?

TBD: testing these gems similar to how we're testing the linux gems

Does this change affect the C or the Java implementations?

No functional changes -- only developer changes in how we build gems.

+155 -85

0 comment

8 changed files

pr created time in 12 days

push eventsparklemotion/nokogiri

Mike Dalessio

commit sha 20f868e16ef0628ea2f49e2b9c141df603a26793

gem: add darwin gems to .cross_rubies - update CrossRuby to recognize it - improve the parsing code for .cross_rubies - only create rake-compiler-docker rake tasks for windows and linux

view details

Mike Dalessio

commit sha 2919f6de81fca0eaf64a66727d70314bb395f157

gem: placeholder for darwin .bundle checks

view details

Mike Dalessio

commit sha 2eb776273763d8888fe13daf333e6b165803b3c9

dev: move build_all into scripts/

view details

Mike Dalessio

commit sha 79598e402f64916529cf2e0c02b63afa8a3b2517

gem: OSX script to set up rubies for native gem compilation

view details

Mike Dalessio

commit sha 108fa2fe0fc483f7e0249c15f394c16ec38922cc

gem: rake tasks for native darwin gem and make sure we gitignore the .bundle files

view details

push time in 12 days

push eventsparklemotion/nokogiri

Mike Dalessio

commit sha 6511a110f16497e7b8db856cf8fdddb3e81a95a1

gem: placeholder for darwin .bundle checks

view details

push time in 13 days

push eventsparklemotion/nokogiri

Mike Dalessio

commit sha 6046edcb18d8ca28561934c815352f96d6ce2481

gem: add darwin gems to .cross_rubies - update CrossRuby to recognize it - and improve the parsing code for .cross_rubies

view details

push time in 13 days

create barnchsparklemotion/nokogiri

branch : 2063-darwin-native-gem

created branch time in 13 days

push eventsparklemotion/nokogiri

Mike Dalessio

commit sha b2754219a1f7b4b75760704e4700f3fbcca24eaa

github issue templates: tweaking things

view details

push time in 14 days

push eventsparklemotion/nokogiri

Mike Dalessio

commit sha 9eda2bf4ad37d4035abe39c053193897b20763d3

issue template: improve the "installation difficulties" template to reflect the questions I ask when problems get really nutty

view details

push time in 14 days

issue commentsparklemotion/nokogiri

Installation difficulties- `cannot load such file -- mini_portile2`

@DougEdey I couldn't reproduce this using the same toolchain and setup. So I'd like to ask some exploratory questions:

  1. what's the output from bundle info mini_portile2? Or, if bundler won't tell you because it didn't finish bundling, what's the output from gem info mini_portile2?

I see:

shipify $ bundle info mini_portile2
  * mini_portile2 (2.4.0)
	Summary: Simplistic port-like solution for developers
	Homepage: http://github.com/flavorjones/mini_portile
	Path: /Users/flavorjones/.gem/ruby/2.6.6/gems/mini_portile2-2.4.0

shipify $ gem info mini_portile2

*** LOCAL GEMS ***

mini_portile2 (2.5.0, 2.4.0)
    Authors: Luis Lavena, Mike Dalessio, Lars Kanis
    Homepage: http://github.com/flavorjones/mini_portile
    License: MIT
    Installed at (2.5.0): /Users/flavorjones/.gem/ruby/2.6.6
                 (2.4.0): /Users/flavorjones/.gem/ruby/2.6.6

    Simplistic port-like solution for developers

  1. Is the mini_portile2 installation OK? Are the files we expect there? Here's what I see:
shipify $ find /Users/flavorjones/.gem/ruby/2.6.6/gems/mini_portile2-2.4.0/lib
/Users/flavorjones/.gem/ruby/2.6.6/gems/mini_portile2-2.4.0/lib
/Users/flavorjones/.gem/ruby/2.6.6/gems/mini_portile2-2.4.0/lib/mini_portile2
/Users/flavorjones/.gem/ruby/2.6.6/gems/mini_portile2-2.4.0/lib/mini_portile2/mini_portile.rb
/Users/flavorjones/.gem/ruby/2.6.6/gems/mini_portile2-2.4.0/lib/mini_portile2/mini_portile_cmake.rb
/Users/flavorjones/.gem/ruby/2.6.6/gems/mini_portile2-2.4.0/lib/mini_portile2/version.rb
/Users/flavorjones/.gem/ruby/2.6.6/gems/mini_portile2-2.4.0/lib/mini_portile2.rb
  1. What is your bundle config output? I'd be interested in anything related to paths. Here's what I see:
shipify $ bundle config
Settings are listed in order of priority. The top value will be used.
packages.shopify.io
Set for the current user (/Users/flavorjones/.bundle/config): "<redacted>"

jobs
Set for your local app (/Users/flavorjones/src/github.com/Shopify/shipify/.bundle/config): 4

(please redact your packages credential!)

  1. What's your gem env? Here's mine (with shell paths redacted):
shipify $ gem env
RubyGems Environment:
  - RUBYGEMS VERSION: 3.0.3
  - RUBY VERSION: 2.6.6 (2020-03-31 patchlevel 146) [x86_64-darwin19]
  - INSTALLATION DIRECTORY: /Users/flavorjones/.gem/ruby/2.6.6
  - USER INSTALLATION DIRECTORY: /Users/flavorjones/.gem/ruby/2.6.0
  - RUBY EXECUTABLE: /opt/rubies/2.6.6/bin/ruby
  - GIT EXECUTABLE: /Users/flavorjones/.nix-profile/bin/git
  - EXECUTABLE DIRECTORY: /Users/flavorjones/.gem/ruby/2.6.6/bin
  - SPEC CACHE DIRECTORY: /Users/flavorjones/.gem/specs
  - SYSTEM CONFIGURATION DIRECTORY: /opt/rubies/2.6.6/etc
  - RUBYGEMS PLATFORMS:
    - ruby
    - x86_64-darwin-19
  - GEM PATHS:
     - /Users/flavorjones/.gem/ruby/2.6.6
     - /opt/rubies/2.6.6/lib/ruby/gems/2.6.0
  - GEM CONFIGURATION:
     - :update_sources => true
     - :verbose => true
     - :backtrace => false
     - :bulk_threshold => 1000
     - "gem" => "--no-ri --no-rdoc --no-document"
  - REMOTE SOURCES:
     - https://rubygems.org/
  - SHELL PATH:
... <snip> ...

I'm fishing for differences in our configurations, or issues that might be specific to your gem installation.

DougEdey

comment created time in 14 days

issue commentsparklemotion/nokogiri

Installation difficulties- `cannot load such file -- mini_portile2`

@DougEdey Sorry you're having problems, I'll try to help.

(Coordinating with Doug in our company Slack, will close the loop here.)

DougEdey

comment created time in 14 days

issue commentsparklemotion/nokogiri

Explore precompiled binaries for Darwin (OSX)

Confirmed, setting --enable-shared or --disable-shared affects whether libruby.so is linked-to from nokogiri.so.

Ruby build with --enable-shared:

juno ruby-2.7.0 go1.14.6 (master *)
nokogiri $ ruby -e 'puts RbConfig::CONFIG["configure_args"]'
 '--prefix=/home/flavorjones/.rvm/rubies/ruby-2.7.0' '--disable-install-doc' '--enable-shared'

juno ruby-2.7.0 go1.14.6 (master *)
nokogiri $ be rake lib/nokogiri/nokogiri.so
install -c tmp/x86_64-linux/nokogiri/2.7.0/nokogiri.so lib/nokogiri/nokogiri.so
cp tmp/x86_64-linux/nokogiri/2.7.0/nokogiri.so tmp/x86_64-linux/stage/lib/nokogiri/nokogiri.so

juno ruby-2.7.0 go1.14.6 (master *)
nokogiri $ ldd lib/nokogiri/nokogiri.so 
	linux-vdso.so.1 (0x00007ffc2b3e8000)
	libruby.so.2.7 => /home/flavorjones/.rvm/rubies/ruby-2.7.0/lib/libruby.so.2.7 (0x00007fd440168000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd43fff9000)
	liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007fd43ffd0000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fd43ffb4000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd43fdc0000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd43fd9d000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fd43fd92000)
	libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007fd43fd0e000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd43fd08000)
	libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007fd43fccd000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fd440974000)

juno ruby-2.7.0 go1.14.6 (master *)
nokogiri $ ls -l lib/nokogiri/nokogiri.so 
-rwxrwxr-x 1 flavorjones flavorjones 2876352 2020-09-04 10:37 lib/nokogiri/nokogiri.so*

juno ruby-2.7.0 go1.14.6 (master *)
nokogiri $ nm lib/nokogiri/nokogiri.so | fgrep rb_gc_mark
                 U rb_gc_mark

Ruby built with --disable-shared:

juno ruby-2.7.1 go1.14.6 (master *)
nokogiri $ ruby -e 'puts RbConfig::CONFIG["configure_args"]'
 '--prefix=/home/flavorjones/.rvm/rubies/ruby-2.7.1' '--disable-shared' '--disable-install-doc'

juno ruby-2.7.1 go1.14.6 (master *)
nokogiri $ be rake lib/nokogiri/nokogiri.so
install -c tmp/x86_64-linux/nokogiri/2.7.1/nokogiri.so lib/nokogiri/nokogiri.so
cp tmp/x86_64-linux/nokogiri/2.7.1/nokogiri.so tmp/x86_64-linux/stage/lib/nokogiri/nokogiri.so

juno ruby-2.7.1 go1.14.6 (master *)
nokogiri $ ldd lib/nokogiri/nokogiri.so
	linux-vdso.so.1 (0x00007ffdd8f58000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f8e5f900000)
	liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f8e5f8d7000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f8e5f8bb000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f8e5f6c9000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f8e5fcaf000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f8e5f6a4000)

juno ruby-2.7.1 go1.14.6 (master *)
nokogiri $ ls -l lib/nokogiri/nokogiri.so
-rwxrwxr-x 1 flavorjones flavorjones 2876240 2020-09-04 10:38 lib/nokogiri/nokogiri.so*

juno ruby-2.7.1 go1.14.6 (master *)
nokogiri $ nm lib/nokogiri/nokogiri.so | fgrep rb_gc_mark
                 U rb_gc_mark
flavorjones

comment created time in 15 days

issue commentsparklemotion/nokogiri

Explore precompiled binaries for Darwin (OSX)

@larskanis I noticed that last night (something I had not seen before)!

It looks like when ruby is configured without --enable-shared that it might not link to libruby (based only on looking at the .bundle generated by rubies configured with and without that option). I need to test more rigorously that this is the case, but either way I understand more deeply what's going on now and can probably hack something. I'll update this issue when I learn more!

flavorjones

comment created time in 16 days

startedcolinbendell/google-meet-grid-view

started time in 18 days

issue commentsparklemotion/nokogiri

RFC: Explore alternatives to libxml2 for HTML parsing

@rubys - I agree that leveraging Nokogumbo is the easiest next step (and potentially the only step, at least for another 10 years :laughing: :sob:).

To be clear: a consequence of this decision is for Loofah to start using Nokogumbo for HTML parsing, meaning that Nokogumbo will become a dependency of Rails, and almost certainly will have an increased support burden as a result.

None of us is paid for our work. How can we preserve maintainer happiness on both projects while maintaining an acceptable level of support for the Rails community?

Proposed next steps

I want to be very respectful of boundaries here. The Nokogumbo contributors have filled this functionality gap very well and I don't want to presume that consolidation or assimilation of either project is necessary or desirable.

If it would be easier to shift this discussion into a realtime face-to-face conversation, I will gladly do that.

1. 2020:

Status quo for a few more months. Ship Nokogiri v1.11.0 towards the end of 2020 with precompiled native gems for windows, linux, and osx/darwin.

Ensure that Nokogumbo installation doesn't break (see https://github.com/rubys/nokogumbo/pull/148 and https://github.com/sparklemotion/nokogiri/pull/1788).

2. Decision point:

After the precompiled native gem release has baked for a bit and we've learned more about whether it was a good decision or not and how hard it is to maintain this approach -- maybe in early 2021? -- make a decision, involving the key stakeholders from both projects, about whether to merge the projects.

3.a. No, don't merge:

If we decide not to merge, then we firm up the contract between the two gems -- both at installation and at runtime -- and invest in integration testing. [1]

3.b. Yes, please merge:

If we decide to merge, then I'd like to suggest Nokogumbo merge into Nokogiri, rather than both merging into a net-new gem; but I am open to a conversation about other approaches. Please let me know if this is a contentious idea, and I'm happy to jump on a realtime call to discuss it.

The core contributors for the merged gem would be the set-union of the Nokogumbo and Nokogiri core contributors, and the commit history for both projects would be preserved. We would make a big deal out of it, and graciously name and thank the people who've made the projects a success. Yay us!

Immediate work would be to functionally integrate the code, and then get precompilation (native gems) working and tested for Nokogiri::HTML5 and Gumbo for a Nokogiri v1.12.0 release -- or potentially a v2.0 release? -- in 2021.

Brief thoughts on our options

My mental model is such that I think it makes most sense to merge (3.b.), but would really like the key Nokogumbo creators and contributors to talk about their own mental models, opinions, fears, etc.

I suspect that keeping the gems separate is going to lead to unhappiness among the maintainers. The Nokogumbo maintainers will inevitably shift focus from "making HTML5 parsing work in Ruby" to "making this C codebase compile and install on an infinite variety of ill-configured systems" as the Rails community comes to depend on it. (Ask me how I know! :sob:) The Nokogiri and Loofah maintainers will experience burnout when their sphere of responsibility doesn't match their sphere of control -- specifically when users loudly complain about something not working perfectly across the integration boundaries.

One reason to not merge might be the size and complexity of the codebase increases for both teams in a merger. However, I think we'd more than offset or mitigate that complexity when we plan to reduce the total support burden by: a) improving the installation experience, b) improving the developer experience, c) removing a vector (libxml2's HTML parser) for security vulnerabilities, d) replacing the informal brittle install-time contract with a robust compile-time contract, e) having a unified team large enough for folks to still specialize within the codebase.


[1]: Nokogumbo's installation process is still pretty tightly-coupled to Nokogiri's installation process, and there's an implied contract between the two that involves the structure of Nokogiri::VersionInfo and the presence of header files on disk. So we'd need to tighten up that contract, and ensure we're integration-testing across the two projects.

Loofah is updated to use Nokogumbo (only on CRuby), making Nokogumbo a dependency of Rails. Nokogumbo support burden would likely increase as a thundering herd of Rails developers are now installing Nokogumbo.

I expect some demand for a faster/better installation experience for Nokogumbo, similar to what Nokogiri has seen over the past few years. Work to keep the support burden low by investing in a single installer toolchain, and a unified memory testing approach (today Nokogiri uses valgrind, but I'm looking into ASan). We'd likely spend some time re-implementing the toolchain and keeping it in sync across the two projects. This doesn't sound like fun to me, but we could do it.

flavorjones

comment created time in 18 days

issue closedsparklemotion/nokogiri

HELP | Partial Search XML by XPath

Hi there!

I googled but didn't find a way to search for a partial XPath in an XML file.

I need to find everything ending with Status word, for example: <diskSizeStatus>1</diskSizeStatus>.

I am using Nokogiri::XML and tried xml.xpath('//@*search', search: 'Status') but was unsuccessful.

Anyone could help me with this, please?

closed time in 18 days

lflucasferreira

issue commentsparklemotion/nokogiri

HELP | Partial Search XML by XPath

Hi, thanks for asking this question, I'll try to help.

I'd like to note that this is a general XPath question, and not a Nokogiri-specific question. A bit of googling led me to https://stackoverflow.com/questions/40934644/xpath-for-element-whose-attribute-value-ends-with-a-specific-string

Nokogiri's version of libxml2 implements XPath 1.0 (and not XPath 2.0) so you'll have to use the more complicated query:

#! /usr/bin/env ruby

require 'nokogiri'

xml = <<EOF
<root>
  <TagFoo>asdf</TagFoo>
  <TagBar>qwer</TagBar>
  <TagBarBaz>zxcv</TagBarBaz>
</root>
EOF

doc = Nokogiri::XML.parse(xml)

results = doc.xpath("//*[substring(name(), string-length(name()) - string-length('Bar')+1) = 'Bar']")
puts results.to_xml
# => <TagBar>qwer</TagBar>

I'll note that there are a few other ways you could have done this, depending on how sensitive you are to performance.

I hope this helps.

lflucasferreira

comment created time in 18 days

issue commentsparklemotion/nokogiri

An error occurred while installing nokogiri (1.9.1), and Bundler cannot continue.

Hi, I'm sorry, I'm not sure how to help you. I've tried to reproduce what you're seeing and cannot -- and demonstrated that it's possible to run an unsupported version of Nokogiri on an unsupported version of Ubuntu.

You may want to investigate a bit more yourself, these error messages indicate that you're having trouble compiling libxml2 on your system:

nanohttp.c:1190:35: error: dereferencing pointer to incomplete type
      } else if (have_ipv6 () && (h->h_addrtype == AF_INET6)) {
                                   ^
nanohttp.c:1192:23: error: dereferencing pointer to incomplete type
   if ((unsigned int) h->h_length > sizeof(ia6)) {
                       ^
nanohttp.c:1196:18: error: dereferencing pointer to incomplete type
   memcpy (&ia6, h->h_addr_list[i], h->h_length);
                  ^
nanohttp.c:1196:37: error: dereferencing pointer to incomplete type
   memcpy (&ia6, h->h_addr_list[i], h->h_length);
                                     ^
nanohttp.c:1197:26: error: dereferencing pointer to incomplete type
   sockin6.sin6_family = h->h_addrtype;
                          ^
make[2]: *** [nanohttp.lo] Error 1
make[2]: Leaving directory

My advice would be to either a) investigate how to get libxml2 to build on your system, or b) try installating nokogiri against system libraries that you can install on Ubuntu Trusty.

You can read more about option (b) here: https://nokogiri.org/tutorials/installing_nokogiri.html#install-with-system-libraries, you'll need to install the libxml2-dev and libxslt-dev packages (they may be named slightly differently on such an old system? I honestly can't remember).

Good luck.

khanimdad

comment created time in 18 days

issue commentsparklemotion/nokogiri

An error occurred while installing nokogiri (1.9.1), and Bundler cannot continue.

Hi, I fired up a docker image of trusty:

root@9df393c99ac8:/# cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.6 LTS"

and installed Ruby 2.2.10 through RVM:

`` root@9df393c99ac8:/# ruby -v ruby 2.2.10p489 (2018-03-28 revision 63023) [x86_64-linux]


and installed Nokogiri v1.9.1 without problems:

root@9df393c99ac8:/# nokogiri -v

Nokogiri (1.9.1)

---
warnings: []
nokogiri: 1.9.1
ruby:
  version: 2.2.10
  platform: x86_64-linux
  description: ruby 2.2.10p489 (2018-03-28 revision 63023) [x86_64-linux]
  engine: ruby
libxml:
  binding: extension
  source: packaged
  libxml2_path: "/usr/local/rvm/gems/ruby-2.2.10/gems/nokogiri-1.9.1/ports/x86_64-pc-linux-gnu/libxml2/2.9.8"
  libxslt_path: "/usr/local/rvm/gems/ruby-2.2.10/gems/nokogiri-1.9.1/ports/x86_64-pc-linux-gnu/libxslt/1.1.32"
  libxml2_patches:
  - 0001-Revert-Do-not-URI-escape-in-server-side-includes.patch
  - 0002-Fix-nullptr-deref-with-XPath-logic-ops.patch
  - 0003-Fix-infinite-loop-in-LZMA-decompression.patch
  libxslt_patches: []
  compiled: 2.9.8
  loaded: 2.9.8

Can you tell me more about how your system is configured?
khanimdad

comment created time in 18 days

delete branch flavorjones/calendar-assistant

delete branch : dependabot/bundler/google-api-client-gte-0.24-and-lt-0.45

delete time in 18 days

push eventflavorjones/calendar-assistant

dependabot-preview[bot]

commit sha 177c16c2624a78b76cd9b6b17e260527dac75234

build(deps): update google-api-client requirement from >= 0.24, < 0.44 to >= 0.24, < 0.45 Updates the requirements on [google-api-client](https://github.com/google/google-api-ruby-client) to permit the latest version. - [Release notes](https://github.com/google/google-api-ruby-client/releases) - [Changelog](https://github.com/googleapis/google-api-ruby-client/blob/master/CHANGELOG.md) - [Commits](https://github.com/google/google-api-ruby-client/compare/0.24.0...0.44.0) Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

view details

Mike Dalessio

commit sha f2c8b8549db38f489ee8016ecf3aa2f7ffa7fc65

Merge pull request #169 from flavorjones/dependabot/bundler/google-api-client-gte-0.24-and-lt-0.45 build(deps): update google-api-client requirement from >= 0.24, < 0.44 to >= 0.24, < 0.45

view details

push time in 18 days

PR merged flavorjones/calendar-assistant

build(deps): update google-api-client requirement from >= 0.24, < 0.44 to >= 0.24, < 0.45 dependencies

Updates the requirements on google-api-client to permit the latest version. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/googleapis/google-api-ruby-client/blob/master/CHANGELOG.md">google-api-client's changelog</a>.</em></p> <blockquote> <h1>0.44.0</h1> <ul> <li>Breaking changes: <ul> <li>Deleted <code>dns_v2beta1</code></li> <li>Deleted <code>oauth2_v2</code></li> <li>Deleted <code>storage_v1beta2</code></li> </ul> </li> <li>Backwards compatible changes: <ul> <li>Added <code>fitness_v1</code></li> <li>Added <code>groupsmigration_v1</code></li> <li>Added <code>localservices_v1</code></li> <li>Updated <code>accessapproval_v1</code></li> <li>Updated <code>accesscontextmanager_v1</code></li> <li>Updated <code>accesscontextmanager_v1beta</code></li> <li>Updated <code>adexchangebuyer2_v2beta1</code></li> <li>Updated <code>adexchangebuyer_v1_4</code></li> <li>Updated <code>adexperiencereport_v1</code></li> <li>Updated <code>admin_datatransfer_v1</code></li> <li>Updated <code>admin_directory_v1</code></li> <li>Updated <code>admin_reports_v1</code></li> <li>Updated <code>admob_v1</code></li> <li>Updated <code>alertcenter_v1beta1</code></li> <li>Updated <code>androidenterprise_v1</code></li> <li>Updated <code>androidmanagement_v1</code></li> <li>Updated <code>androidpublisher_v3</code></li> <li>Updated <code>apigee_v1</code></li> <li>Updated <code>appengine_v1</code></li> <li>Updated <code>appengine_v1alpha</code></li> <li>Updated <code>appengine_v1beta</code></li> <li>Updated <code>appsmarket_v2</code></li> <li>Updated <code>artifactregistry_v1beta1</code></li> <li>Updated <code>bigquery_v2</code></li> <li>Updated <code>bigqueryconnection_v1beta1</code></li> <li>Updated <code>bigquerydatatransfer_v1</code></li> <li>Updated <code>bigqueryreservation_v1</code></li> <li>Updated <code>bigqueryreservation_v1alpha2</code></li> <li>Updated <code>bigqueryreservation_v1beta1</code></li> <li>Updated <code>bigtableadmin_v1</code></li> <li>Updated <code>bigtableadmin_v2</code></li> <li>Updated <code>billingbudgets_v1beta1</code></li> <li>Updated <code>binaryauthorization_v1</code></li> <li>Updated <code>binaryauthorization_v1beta1</code></li> <li>Updated <code>calendar_v3</code></li> <li>Updated <code>chat_v1</code></li> <li>Updated <code>civicinfo_v2</code></li> <li>Updated <code>cloudasset_v1</code></li> <li>Updated <code>cloudasset_v1beta1</code></li> <li>Updated <code>cloudasset_v1p1beta1</code></li> <li>Updated <code>cloudasset_v1p4beta1</code></li> <li>Updated <code>cloudasset_v1p5beta1</code></li> <li>Updated <code>cloudbilling_v1</code></li> <li>Updated <code>cloudbuild_v1</code></li> </ul> </li> </ul> <!-- raw HTML omitted --> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/googleapis/google-api-ruby-client/commit/bedfa959508ca78405998f98a439ccfa51fbe6db"><code>bedfa95</code></a> 0.44.0</li> <li><a href="https://github.com/googleapis/google-api-ruby-client/commit/76a32f11cf615c1f7bbd041f1b72e12f0449d41e"><code>76a32f1</code></a> Autogenerated update (2020-08-29)</li> <li><a href="https://github.com/googleapis/google-api-ruby-client/commit/32fac21291a60b0f09546c27f8e547f8b138f297"><code>32fac21</code></a> Autogenerated update (2020-08-28)</li> <li><a href="https://github.com/googleapis/google-api-ruby-client/commit/5874aa5aa4188ff7e5e7e86def4111a24d1362a9"><code>5874aa5</code></a> Autogenerated update (2020-08-27)</li> <li><a href="https://github.com/googleapis/google-api-ruby-client/commit/f06309c6ba3235e27eda94a56138f6e5e21f98f1"><code>f06309c</code></a> Autogenerated update (2020-08-26)</li> <li><a href="https://github.com/googleapis/google-api-ruby-client/commit/8b8f32d9ca34a37db69dc147574f6339178e947c"><code>8b8f32d</code></a> Autogenerated update (2020-08-25)</li> <li><a href="https://github.com/googleapis/google-api-ruby-client/commit/d0d22a2baa3c733d6def93ef2342d4045e6d10db"><code>d0d22a2</code></a> Autogenerated update (2020-08-22)</li> <li><a href="https://github.com/googleapis/google-api-ruby-client/commit/e08f0d3bef470cf05ec89b5eeca8d8d806b317e0"><code>e08f0d3</code></a> Autogenerated update (2020-08-21)</li> <li><a href="https://github.com/googleapis/google-api-ruby-client/commit/ccf85d8cdabdb5f5616a7d2dbad2b3f2560c8901"><code>ccf85d8</code></a> Autogenerated update (2020-08-20)</li> <li><a href="https://github.com/googleapis/google-api-ruby-client/commit/e5cce9ef8df2fc5a255d5426a3aa164dc0cfc747"><code>e5cce9e</code></a> Autogenerated update (2020-08-19)</li> <li>Additional commits viewable in <a href="https://github.com/google/google-api-ruby-client/compare/0.24.0...0.44.0">compare view</a></li> </ul> </details> <br />

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


<details> <summary>Dependabot commands and options</summary> <br />

You can trigger Dependabot actions by commenting on this PR:

  • @dependabot rebase will rebase this PR
  • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
  • @dependabot merge will merge this PR after your CI passes on it
  • @dependabot squash and merge will squash and merge this PR after your CI passes on it
  • @dependabot cancel merge will cancel a previously requested merge and block automerging
  • @dependabot reopen will reopen this PR if it is closed
  • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
  • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
  • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
  • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language
  • @dependabot badge me will comment on this PR with code to add a "Dependabot enabled" badge to your readme

Additionally, you can set the following in your Dependabot dashboard:

  • Update frequency (including time of day and day of week)
  • Pull request limits (per update run and/or open at any time)
  • Automerge options (never/patch/minor, and dev/runtime dependencies)
  • Out-of-range updates (receive only lockfile updates, if desired)
  • Security updates (receive only security updates, if desired)

</details>

+1 -1

1 comment

1 changed file

dependabot-preview[bot]

pr closed time in 18 days

issue commentsparklemotion/nokogiri

Explore precompiled binaries for Darwin (OSX)

This blog post has some good info on linker paths on OSX, along with links to deeper reference material: https://medium.com/@donblas/fun-with-rpath-otool-and-install-name-tool-e3e41ae86172

flavorjones

comment created time in 18 days

issue commentsparklemotion/nokogiri

Explore precompiled binaries for Darwin (OSX)

On linux, dynamic libraries are names searched for in a runtime path (LD_LIBRARY_PATH):

$ ldd lib/nokogiri/nokogiri.so 
	linux-vdso.so.1 (0x00007ffcdff91000)
	libruby.so.2.5 => not found
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fabb8487000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fabb846b000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fabb8448000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fabb8254000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fabb8a2a000)

On OSX this is what I see:

nokogiri $ otool -L lib/nokogiri/nokogiri.bundle
lib/nokogiri/nokogiri.bundle:
	/Users/flavorjones/.rvm/rubies/ruby-2.7.0/lib/libruby.2.7.dylib (compatibility version 2.7.0, current version 2.7.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.100.1)
	/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
	/usr/local/opt/xz/lib/liblzma.5.dylib (compatibility version 8.0.0, current version 8.5.0)
	/usr/local/opt/zlib/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
flavorjones

comment created time in 18 days

issue commentsparklemotion/nokogiri

Explore precompiled binaries for Darwin (OSX)

Maybe worth noting: if I build the binary gem using homebrew-installed ruby 2.7, but install it into an rvm-installed ruby 2.7, it installs but I get this error at runtime:

$ nokogiri -v
Traceback (most recent call last):
	9: from /Users/flavorjones/.rvm/gems/ruby-2.7.0/bin/ruby_executable_hooks:24:in `<main>'
	8: from /Users/flavorjones/.rvm/gems/ruby-2.7.0/bin/ruby_executable_hooks:24:in `eval'
	7: from /Users/flavorjones/.rvm/gems/ruby-2.7.0/bin/nokogiri:23:in `<main>'
	6: from /Users/flavorjones/.rvm/gems/ruby-2.7.0/bin/nokogiri:23:in `load'
	5: from /Users/flavorjones/.rvm/gems/ruby-2.7.0/gems/nokogiri-1.11.0.rc2-x86_64-darwin-19/bin/nokogiri:6:in `<top (required)>'
	4: from /Users/flavorjones/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require'
	3: from /Users/flavorjones/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require'
	2: from /Users/flavorjones/.rvm/gems/ruby-2.7.0/gems/nokogiri-1.11.0.rc2-x86_64-darwin-19/lib/nokogiri.rb:13:in `<top (required)>'
	1: from /Users/flavorjones/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require'
/Users/flavorjones/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require': cannot load such file -- nokogiri/2.7/nokogiri (LoadError)
	10: from /Users/flavorjones/.rvm/gems/ruby-2.7.0/bin/ruby_executable_hooks:24:in `<main>'
	 9: from /Users/flavorjones/.rvm/gems/ruby-2.7.0/bin/ruby_executable_hooks:24:in `eval'
	 8: from /Users/flavorjones/.rvm/gems/ruby-2.7.0/bin/nokogiri:23:in `<main>'
	 7: from /Users/flavorjones/.rvm/gems/ruby-2.7.0/bin/nokogiri:23:in `load'
	 6: from /Users/flavorjones/.rvm/gems/ruby-2.7.0/gems/nokogiri-1.11.0.rc2-x86_64-darwin-19/bin/nokogiri:6:in `<top (required)>'
	 5: from /Users/flavorjones/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require'
	 4: from /Users/flavorjones/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require'
	 3: from /Users/flavorjones/.rvm/gems/ruby-2.7.0/gems/nokogiri-1.11.0.rc2-x86_64-darwin-19/lib/nokogiri.rb:11:in `<top (required)>'
	 2: from /Users/flavorjones/.rvm/gems/ruby-2.7.0/gems/nokogiri-1.11.0.rc2-x86_64-darwin-19/lib/nokogiri.rb:15:in `rescue in <top (required)>'
	 1: from /Users/flavorjones/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require'
/Users/flavorjones/.rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require': dlopen(/Users/flavorjones/.rvm/gems/ruby-2.7.0/gems/nokogiri-1.11.0.rc2-x86_64-darwin-19/lib/nokogiri/nokogiri.bundle, 9): Library not loaded: /usr/local/opt/ruby/lib/libruby.2.7.dylib (LoadError)
  Referenced from: /Users/flavorjones/.rvm/gems/ruby-2.7.0/gems/nokogiri-1.11.0.rc2-x86_64-darwin-19/lib/nokogiri/nokogiri.bundle
  Reason: image not found - /Users/flavorjones/.rvm/gems/ruby-2.7.0/gems/nokogiri-1.11.0.rc2-x86_64-darwin-19/lib/nokogiri/nokogiri.bundle

But then when I re-build the gem with rvm-installed ruby, and then install the gem, it works fine:

# Nokogiri (1.11.0.rc2)
    ---
    warnings: []
    nokogiri: 1.11.0.rc2
    ruby:
      version: 2.7.0
      platform: x86_64-darwin19
      description: ruby 2.7.0p0 (2019-12-25 revision 647ee6f091) [x86_64-darwin19]
      engine: ruby
    libxml:
      source: packaged
      patches:
      - 0001-Revert-Do-not-URI-escape-in-server-side-includes.patch
      - 0002-Remove-script-macro-support.patch
      - 0003-Update-entities-to-remove-handling-of-ssi.patch
      - 0004-libxml2.la-is-in-top_builddir.patch
      - 0005-Fix-infinite-loop-in-xmlStringLenDecodeEntities.patch
      compiled: 2.9.10
      loaded: 2.9.10
    libxslt:
      source: packaged
      patches: []
      compiled: 1.1.34
      loaded: 1.1.34

The problem is that both of those rubies (homebrew-installed and rvm-installed) have the same RUBY_PLATFORM and gem env platform. :thinking:

flavorjones

comment created time in 18 days

issue commentsparklemotion/nokogiri

Explore precompiled binaries for Darwin (OSX)

Voila:

huygens (master *)
nokogiri $ time gem install pkg/nokogiri-1.11.0.rc2-x86_64-darwin-19.gem
Successfully installed nokogiri-1.11.0.rc2-x86_64-darwin-19
1 gem installed

real	0m0.764s
user	0m0.355s
sys	0m0.187s

huygens (master *)
nokogiri $ ruby -S nokogiri -v
# Nokogiri (1.11.0.rc2)
    ---
    warnings: []
    nokogiri: 1.11.0.rc2
    ruby:
      version: 2.7.1
      platform: x86_64-darwin19
      description: ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x86_64-darwin19]
      engine: ruby
    libxml:
      source: packaged
      patches:
      - 0001-Revert-Do-not-URI-escape-in-server-side-includes.patch
      - 0002-Remove-script-macro-support.patch
      - 0003-Update-entities-to-remove-handling-of-ssi.patch
      - 0004-libxml2.la-is-in-top_builddir.patch
      - 0005-Fix-infinite-loop-in-xmlStringLenDecodeEntities.patch
      compiled: 2.9.10
      loaded: 2.9.10
    libxslt:
      source: packaged
      patches: []
      compiled: 1.1.34
      loaded: 1.1.34


flavorjones

comment created time in 18 days

issue commentsparklemotion/nokogiri

Explore precompiled binaries for Darwin (OSX)

Voila:

flavorjones

comment created time in 18 days

issue commentsparklemotion/nokogiri

Explore precompiled binaries for Darwin (OSX)

Ah, this is actually better and addresses the root problem: that we're trying to invoke the wrongly-named rake task:

diff --git a/.cross_rubies b/.cross_rubies
index 57c8fb0..e884b16 100644
--- a/.cross_rubies
+++ b/.cross_rubies
@@ -2,6 +2,7 @@
 2.7.0:x86_64-w64-mingw32
 2.7.0:i686-linux-gnu
 2.7.0:x86_64-linux-gnu
+2.7.0:x86_64-darwin19
 2.6.0:i686-w64-mingw32
 2.6.0:x86_64-w64-mingw32
 2.6.0:i686-linux-gnu
diff --git a/tasks/cross-ruby.rb b/tasks/cross-ruby.rb
index 35c098f..1d5c2fd 100644
--- a/tasks/cross-ruby.rb
+++ b/tasks/cross-ruby.rb
@@ -26,6 +26,8 @@ CrossRuby = Struct.new(:version, :host) do
         "x86_64-linux"
       when /\Ai[3-6]86.*linux/
         "x86-linux"
+      when /\Ax86_64-darwin19/
+        "x86_64-darwin19"
       else
         raise "unsupported host: #{host}"
       end
@@ -179,7 +181,7 @@ namespace "gem" do
       task "guest" do
         # use Task#invoke because the pkg/*gem task is defined at runtime
         Rake::Task["native:#{plat}"].invoke
-        Rake::Task["pkg/#{HOE.spec.full_name}-#{plat}.gem"].invoke
+        Rake::Task["pkg/#{HOE.spec.full_name}-#{Gem::Platform.new(plat).to_s}.gem"].invoke
       end
     end
   end

And the output file is now nokogiri-1.11.0.rc2-x86_64-darwin-19.gem which is TOTALLY GOING TO WORK because gem env will be looking for the same platform name.

flavorjones

comment created time in 18 days

issue commentsparklemotion/nokogiri

Explore precompiled binaries for Darwin (OSX)

OK, I managed to build a working native darwin gem by applying the following patch!

diff --git a/.cross_rubies b/.cross_rubies
index 57c8fb0..e3f5f1e 100644
--- a/.cross_rubies
+++ b/.cross_rubies
@@ -2,6 +2,7 @@
 2.7.0:x86_64-w64-mingw32
 2.7.0:i686-linux-gnu
 2.7.0:x86_64-linux-gnu
+2.7.0:universal-darwin-19
 2.6.0:i686-w64-mingw32
 2.6.0:x86_64-w64-mingw32
 2.6.0:i686-linux-gnu
diff --git a/Rakefile b/Rakefile
index f5f6f47..16be862 100644
--- a/Rakefile
+++ b/Rakefile
@@ -118,6 +118,7 @@ else
   end
 
   Rake::ExtensionTask.new("nokogiri", HOE.spec) do |ext|
+    ext.platform = "universal-darwin-19"
     ext.lib_dir = File.join(*['lib', 'nokogiri', ENV['FAT_DIR']].compact)
     ext.config_options << ENV['EXTOPTS']
     ext.cross_compile  = true
diff --git a/tasks/cross-ruby.rb b/tasks/cross-ruby.rb
index 35c098f..d4e84bd 100644
--- a/tasks/cross-ruby.rb
+++ b/tasks/cross-ruby.rb
@@ -26,6 +26,8 @@ CrossRuby = Struct.new(:version, :host) do
         "x86_64-linux"
       when /\Ai[3-6]86.*linux/
         "x86-linux"
+      when /\Auniversal-darwin-19/
+        "universal-darwin-19"
       else
         raise "unsupported host: #{host}"
       end

and then running rake gem:universal-darwin-19:guest -- because obviously we don't have a guest container.

$ time gem install pkg/nokogiri-1.11.0.rc2-universal-darwin-19.gem 
Successfully installed nokogiri-1.11.0.rc2-universal-darwin-19
1 gem installed

real	0m0.795s
user	0m0.350s
sys	0m0.185s

$ ruby -S nokogiri -v
# Nokogiri (1.11.0.rc2)
    ---
    warnings: []
    nokogiri: 1.11.0.rc2
    ruby:
      version: 2.7.1
      platform: x86_64-darwin19
      description: ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x86_64-darwin19]
      engine: ruby
    libxml:
      source: packaged
      patches:
      - 0001-Revert-Do-not-URI-escape-in-server-side-includes.patch
      - 0002-Remove-script-macro-support.patch
      - 0003-Update-entities-to-remove-handling-of-ssi.patch
      - 0004-libxml2.la-is-in-top_builddir.patch
      - 0005-Fix-infinite-loop-in-xmlStringLenDecodeEntities.patch
      compiled: 2.9.10
      loaded: 2.9.10
    libxslt:
      source: packaged
      patches: []
      compiled: 1.1.34
      loaded: 1.1.34
flavorjones

comment created time in 18 days

issue commentsparklemotion/nokogiri

Explore precompiled binaries for Darwin (OSX)

@larskanis I'm also happy to send a PR to rake-compiler if that's what's needed to sort this out -- I'm only asking for your advice on what solution to pursue! I'll keep playing around.

flavorjones

comment created time in 19 days

issue commentsparklemotion/nokogiri

Explore precompiled binaries for Darwin (OSX)

@larskanis I'm hoping you can help clarify some things for me. There seems to be a few different platform name values on OSX and I'm not sure how to configure the cross-ruby tasks to do a proof-of-concept native darwin build.

TL;DR:

On Linux, all three of these values are the same:

  • RUBY_PLATFORM # => "x86_64-linux"
  • Gem::Platform.new(RUBY_PLATFORM).to_s # => "x86_64-linux"
  • gem env reports x86_64-linux as one of the "RUBYGEMS PLATFORMS"

On OSX, all three of these values are different:

  • RUBY_PLATFORM # => "universal.x86_64-darwin19"
  • Gem::Platform.new(RUBY_PLATFORM).to_s # => "universal.x86_64-darwin-19"
  • gem env reports universal-darwin-19 as one of the "RUBYGEMS PLATFORMS"

Longer explanation

First, the native tasks are defined in rake-compiler's Rake::ExtensionTask#define_native_tasks. Right now this method is being called from #define with no arguments, and so platf is set to platform which returns RUBY_PLATFORM when @platform isn't set -- and so this value on a mac is universal.x86_64-darwin19.

This matches what ruby -v tells me: ruby 2.6.3p62 (2019-04-16 revision 67580) [universal.x86_64-darwin19].

gem env, however, says the platform is universal-darwin-19.

Finally, though, if I add universal.x86_64-darwin19 to both .cross_rubies and tasks/cross-ruby.rb:

  def platform
    @platform ||= case host
      when /\Ax86_64.*mingw32/
        "x64-mingw32"
      when /\Ai[3-6]86.*mingw32/
        "x86-mingw32"
      when /\Ax86_64.*linux/
        "x86_64-linux"
      when /\Ai[3-6]86.*linux/
        "x86-linux"
      when /\Auniversal.x86_64-darwin19/
        "universal.x86_64-darwin19"
      else
        raise "unsupported host: #{host}"
      end
  end

then the error I see is:

Don't know how to build task 'pkg/nokogiri-1.11.0.rc2-universal.x86_64-darwin19.gem' (See the list of available tasks with `rake --tasks`)
Did you mean?  pkg/nokogiri-1.11.0.rc2-universal.x86_64-darwin-19.gem

which implies the existence of a THIRD platform name which inserts a space between "darwin" and "19" -- and I discovered that this is the spec platform which is set to Gem::Platform.new(platf) inside define_native_tasks:

Gem::Platform.new("universal.x86_64-darwin19").to_s
=> "universal.x86_64-darwin-19"

I can't quite put all the pieces together and make this work, but it feels like this might be pointing towards an inconsistency in how these names are handled in rake-compiler?

flavorjones

comment created time in 19 days

push eventsparklemotion/nokogiri

Mike Dalessio

commit sha 1ae26a68c35328aab92bfcb55194407e8bbe1565

ci: looks like we don't need to set BUNDLER_GEMFILE anymore previously we had to because of these bugs: bundler: - https://github.com/rubygems/bundler/issues/6162 - https://github.com/rubygems/bundler/issues/6154 cruby docker: - https://github.com/docker-library/ruby/pull/209 - https://github.com/docker-library/ruby/pull/306 jruby docker: - https://github.com/cpuguy83/docker-jruby/issues/65 - https://github.com/cpuguy83/docker-jruby/pull/67/files

view details

push time in 19 days

delete branch flavorjones/concourse

delete branch : flavorjones-fly-validate-pipeline-enable-across-step

delete time in 19 days

delete branch sparklemotion/nokogiri

delete branch : 1855-package-gem-version-with-tests

delete time in 19 days

delete branch sparklemotion/nokogiri

delete branch : 1992-backport-cve-2020-7595

delete time in 19 days

push eventsparklemotion/nokogiri

Mike Dalessio

commit sha 41ca71a9f1d0ac8709a6597a6be86cd386ac88ba

ci: remove docker-compose file accidentally committed

view details

Mike Dalessio

commit sha 04d91ceb8c4e062a8e4c6b78d29fcf2ccc98f617

ci: use an intermediate env var TEST_WITH_SYSTEM_LIBRARIES Now "t" is on and anything else is off for TEST_WITH_VALGRIND and TEST_WITH_SYSTEM_LIBRARIES. This better allows for on/off setting from pipeline configs, because NOKOGIRI_USE_SYSTEM_LIBRARIES is considered set even if it's present-but-blank.

view details

Mike Dalessio

commit sha b1cdf6e6ae6a36b6d9b3c932cec3c216d66aa37b

dep: update dev dep concourse to v0.34.0

view details

Mike Dalessio

commit sha 23c78c486d34383a624964468941d1c37586b5dc

Merge branch 'flavorjones-concourse-across-matrix'

view details

push time in 19 days

push eventsparklemotion/nokogiri

Mike Dalessio

commit sha 23c78c486d34383a624964468941d1c37586b5dc

Merge branch 'flavorjones-concourse-across-matrix'

view details

Mike Dalessio

commit sha 98fd3081fd77210688ff92aafc2f61ae4f0facd2

ci: try using `across` to set up a matrixed pipeline and see if it's easier to maintain and/or easier to read

view details

push time in 19 days

push eventsparklemotion/nokogiri

Mike Dalessio

commit sha 41ca71a9f1d0ac8709a6597a6be86cd386ac88ba

ci: remove docker-compose file accidentally committed

view details

Mike Dalessio

commit sha 04d91ceb8c4e062a8e4c6b78d29fcf2ccc98f617

ci: use an intermediate env var TEST_WITH_SYSTEM_LIBRARIES Now "t" is on and anything else is off for TEST_WITH_VALGRIND and TEST_WITH_SYSTEM_LIBRARIES. This better allows for on/off setting from pipeline configs, because NOKOGIRI_USE_SYSTEM_LIBRARIES is considered set even if it's present-but-blank.

view details

Mike Dalessio

commit sha b1cdf6e6ae6a36b6d9b3c932cec3c216d66aa37b

dep: update dev dep concourse to v0.34.0

view details

push time in 19 days

issue commentsparklemotion/nokogiri

An error occurred while installing nokogiri (1.9.1), and Bundler cannot continue.

Additionally, I see:

conftest.c:5:2: error: #error libxml2 is older than 2.9.3

which tells me you're compiling against system libraries, and libxml2 2.9.3 is too old for Nokogiri to support. Please use the vendored libraries if this continues to be a blocker for you.

khanimdad

comment created time in 19 days

issue closedsparklemotion/nokogiri

An error occurred while installing nokogiri (1.9.1), and Bundler cannot continue.

I am getting below error when deploying on Ubuntu 14 LTS

(Backtrace restricted to imported tasks) cap aborted! SSHKit::Runner::ExecuteError: Exception while executing as deploy@u2h3.yoopedia.com: bundle exit status: 5 bundle stdout: An error occurred while installing nokogiri (1.9.1), and Bundler cannot continue. Make sure that gem install nokogiri -v '1.9.1' succeeds before bundling. bundle stderr: /home/deploy/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/bundler-1.10.4/lib/bundler.rb:315: warning: Insecure world writable dir /home/deploy/.rbenv/versions/2.2.2/bin in PATH, mode 040777

Caused by: SSHKit::Command::Failed: bundle exit status: 5 bundle stdout: An error occurred while installing nokogiri (1.9.1), and Bundler cannot continue. Make sure that gem install nokogiri -v '1.9.1' succeeds before bundling. bundle stderr: /home/deploy/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/bundler-1.10.4/lib/bundler.rb:315: warning: Insecure world writable dir /home/deploy/.rbenv/versions/2.2.2/bin in PATH, mode 040777

Tasks: TOP => deploy:updated => bundler:install (See full trace by running task with --trace) The deploy has failed with an error: Exception while executing as deploy@u2h3.yoopedia.com: bundle exit status: 5 bundle stdout: An error occurred while installing nokogiri (1.9.1), and Bundler cannot continue. Make sure that gem install nokogiri -v '1.9.1' succeeds before bundling. bundle stderr: /home/deploy/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/bundler-1.10.4/lib/bundler.rb:315: warning: Insecure world writable dir /home/deploy/.rbenv/versions/2.2.2/bin in PATH, mode 040777

closed time in 19 days

khanimdad

issue commentsparklemotion/nokogiri

An error occurred while installing nokogiri (1.9.1), and Bundler cannot continue.

The problem, buried in this log, is:

Cannot create temporary file in /tmp/: No space left on device

I'm going to close this for now, but let us know if you have more trouble after resolving the diskspace issue.

khanimdad

comment created time in 19 days

pull request commentflavorjones/calendar-assistant

build(deps-dev): update simplecov requirement from ~> 0.17.0 to ~> 0.19.0

Please don't merge this. See 0ebf963 for information on why simplecov is pinned to v0.17.x.

dependabot-preview[bot]

comment created time in 19 days

issue commentsparklemotion/nokogiri

An error occurred while installing nokogiri (1.9.1), and Bundler cannot continue.

@khanimdad Hi, sorry you're having trouble. I'll try to help.

When you submitted this issue, you ignored the template that asks questions relevant to installation problems. In order to best help you, I have to ask you to answer those questions. Here they are:


Have you read and followed the installation tutorial at http://www.nokogiri.org/tutorials/installing_nokogiri.html?

What is the complete output of gem install?

Please make sure to escape the file contents with triple-backticks.

What are the complete contents of the mkmf.log file generated during the failed installation?

Please make sure to escape the file contents with triple-backticks.

What operating system are you using?

Are you using Linux, MacOS, Windows, a BSD variant, or something else entirely?

If MacOS, please note the version of XCode you have installed.

If Linux, please include the output of uname -a and the contents of /etc/lsb-release.

If Windows, please note whether you're installing the precompiled gems, or compiling yourself with DevKit. Also please note the version of RubyInstaller you've installed.

khanimdad

comment created time in 19 days

push eventflavorjones/concourse-gem

Mike Dalessio

commit sha 9ef4cc100d8f0df72f404ee1e884e5009d746a79

README: update TOC

view details

push time in 19 days

more