[ruby-core:40271] Can rubygems save us from "binary-compatibility hell"?

[ruby-core:40271] Can rubygems save us from "binary-compatibility hell"?

[Yusuke Endoh]

Hello, rubygems developers --

I'd like to ask some questions about rubygems and Ruby's ABI
compatibility.


Currently, the core team tries to ensure ABI compatibility in
the same Ruby series.  This means an extension library file that
is compiled with 1.9.1 can be loaded successfully with 1.9.2.

However, it is sometimes frustrating for the core team to find
an ABI-compatible way to fix some problems. [ruby-core:28281]
[ruby-core:27945]
It is also unfortunate to postpone or give up some new features
or improvements because they break ABI compatibility.
[ruby-core:39847]
In addition, the definition of ABI compatibility is ambiguous
a little.  I think "ABI compatibility" is actually ensured, but
some don't. [ruby-core:36108]


So, it would be good to discard ABI compatibility in 2.0.
Each version of Ruby (including TEENY) should have their own
library path, such as:

  /usr/lib/ruby/2.0.0/
  /usr/lib/ruby/2.0.1/
  /usr/lib/ruby/2.0.2/
  etc.

In this policy, all the users have to re-install libraries that
they are using whenever ruby is upgraded.  It's a pain.  But
recently, many libraries are gems.


So I'd like to ask a question to rubygems developers; is it
technically possible to automatically re-install all gems from
old Ruby installation to new one when Ruby is upgraded (or
during "make install")?

I have two concerns.  One is a user-custom gem that a user
created and not published in rubygems.org.  Is there a custom
gem or its source necessarily remained?

The other is a binary gem that includes a compiled extension
library.  TEENY release will be done about every year.  This
is not a technical issue, but do you think it is reasonable
to make gem developers release their binary gems when new
Ruby is released?  And, is it possible to allow users to use,
with a new Ruby, an old gem that does not support the new Ruby
with a warning?

And isn't there another concern I missed?


Note that this is currently just my idea, not a decision at
all.  What do you think?

-- 
Yusuke Endoh <mame@tsg.ne.jp>

[ruby-core:40273] Re: Can rubygems save us from "binary-compatibility hell"?

[Eric Hodel]

On Oct 21, 2011, at 7:48 PM, Yusuke Endoh wrote:
> Hello, rubygems developers --
> 
> I'd like to ask some questions about rubygems and Ruby's ABI
> compatibility.
> 
> Currently, the core team tries to ensure ABI compatibility in
> the same Ruby series.  This means an extension library file that
> is compiled with 1.9.1 can be loaded successfully with 1.9.2.
> 
> However, it is sometimes frustrating for the core team to find
> an ABI-compatible way to fix some problems. [ruby-core:28281]
> [ruby-core:27945]
> It is also unfortunate to postpone or give up some new features
> or improvements because they break ABI compatibility.
> [ruby-core:39847]
> In addition, the definition of ABI compatibility is ambiguous
> a little.  I think "ABI compatibility" is actually ensured, but
> some don't. [ruby-core:36108]
> 
> So, it would be good to discard ABI compatibility in 2.0.
> Each version of Ruby (including TEENY) should have their own
> library path, such as:
> 
>  /usr/lib/ruby/2.0.0/
>  /usr/lib/ruby/2.0.1/
>  /usr/lib/ruby/2.0.2/
>  etc.
> 
> In this policy, all the users have to re-install libraries that
> they are using whenever ruby is upgraded.  It's a pain.  But
> recently, many libraries are gems.

Yes, `gem pristine --all` will do this now:

$ GEM_HOME=~/tmp/gems GEM_PATH=~/tmp/gems gem19 pristine --all
Restoring gems to pristine condition...
Building native extensions.  This could take a while...
Restored nokogiri-1.5.0
$

(I used a gem path with just nokogiri to save time)

> So I'd like to ask a question to rubygems developers; is it
> technically possible to automatically re-install all gems from
> old Ruby installation to new one when Ruby is upgraded (or
> during "make install")?

Since 2.0.0 and 2.0.1 will have different gem directories:

$ gem19 env home
/usr/local/lib/ruby/gems/1.9.1

A different mechanism would need to be used to install gems from 2.0.0 into 2.0.1.  Using the cache directory this should be easy without needing to re-download gems, something like:

  cd `gem2.0.0 env home`; gem2.0.1 install --local -f *.gem

But in ruby, of course.

> I have two concerns.  One is a user-custom gem that a user
> created and not published in rubygems.org.  Is there a custom
> gem or its source necessarily remained?

So long as the user has not removed the .gem file from the cache directory (the source) it will work.  Users usually do not know how to do this.

> The other is a binary gem that includes a compiled extension
> library.  TEENY release will be done about every year.  This
> is not a technical issue, but do you think it is reasonable
> to make gem developers release their binary gems when new
> Ruby is released?

I think it depends on the developer.  I can make a list of recent platform gems so we can consult the authors.  With the addition of the Developer Kit for Windows rubyists it may not be as big a concern now as it has been in the past.  Luis can comment better here.

> And, is it possible to allow users to use, with a new Ruby, an old gem that does not support the new Ruby with a warning?

This would require further adjustments to RubyGems, but I think it is possible.  Some of the adjustments are related to the stdlib gems proposal (coming soon).

> And isn't there another concern I missed?
> 
> Note that this is currently just my idea, not a decision at
> all.  What do you think?

I think your idea is possible and easy to implement.

[ruby-core:40276] Fwd: Re: Can rubygems save us from "binary-compatibility hell"?

[Luis Lavena]

Forwarding this again to ruby-core as received a postmaster delivery failure.

---------- Forwarded message ----------
From: Luis Lavena <luislavena@gmail.com>
Date: Sat, Oct 22, 2011 at 11:08 AM
Subject: Re: [ruby-core:40273] Re: Can rubygems save us from
"binary-compatibility hell"?
To: ruby-core@ruby-lang.org, "RubyGems Developers (ML)"
<rubygems-developers@rubyforge.org>


Hello,

Please see comments inline...

On Sat, Oct 22, 2011 at 5:21 AM, Eric Hodel <drbrain@segment7.net> wrote:
>
>> The other is a binary gem that includes a compiled extension
>> library.  TEENY release will be done about every year.  This
>> is not a technical issue, but do you think it is reasonable
>> to make gem developers release their binary gems when new
>> Ruby is released?
>
> I think it depends on the developer.  I can make a list of recent platform gems so we can consult the authors.  With the addition of the Developer Kit for Windows rubyists it may not be as big a concern now as it has been in the past.  Luis can comment better here.
>

Well, that depends on what are the dependencies of those gems. If the
gem just use C extension for speed and don't depend on external
libraries (e.g. RedCloth, rdiscount, hpricot) then is not an issue if
there is no binary gem for the newer version of Ruby.

But, this is a problem for more complex gems like EventMachine,
nokogiri and others that depend on externals (openssl, libxml2).

More than that, there is an existing problem with RubyGems that forced
us to create "fat-binary" gems which include binaries for 1.8.x and
1.9.x so gem can be installed properly.

Projects like rake-compiler made this task a bit more easy, yet still
requires gem author certain environment adjustments.

While 1.8.x usage has been reduced lately, it is still present.

With this ABI compatibility change it will require those gem authors
wanting to provide fat-binary gems for every single ABI version in the
same gem.

I've presented this issue before at rubygems developer list, April 2009:

http://rubyforge.org/pipermail/rubygems-developers/2009-April/004522.html

>> And, is it possible to allow users to use, with a new Ruby, an old gem that does not support the new Ruby with a warning?
>
> This would require further adjustments to RubyGems, but I think it is possible.  Some of the adjustments are related to the stdlib gems proposal (coming soon).
>

Perhaps RubyGems' Specification will require to be extended to a list
of versions the gem is compatible with and not just one unique
version?

Those are my only concerns: gem authors overhead when packaging binary
gems and making more easy to gem users install them without falling
into the compile-your-own rabbit hole.

Thank you.
--
Luis Lavena
AREA 17
-
Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupéry

[ruby-core:40579] Re: Can rubygems save us from "binary-compatibility hell"?

[Vít Ondruch]

Dne 22.10.2011 4:48, Yusuke Endoh napsal(a):
> Hello, rubygems developers --
>
> I'd like to ask some questions about rubygems and Ruby's ABI
> compatibility.
>
>
> Currently, the core team tries to ensure ABI compatibility in
> the same Ruby series.  This means an extension library file that
> is compiled with 1.9.1 can be loaded successfully with 1.9.2.
>
> However, it is sometimes frustrating for the core team to find
> an ABI-compatible way to fix some problems. [ruby-core:28281]
> [ruby-core:27945]
> It is also unfortunate to postpone or give up some new features
> or improvements because they break ABI compatibility.
> [ruby-core:39847]
> In addition, the definition of ABI compatibility is ambiguous
> a little.  I think "ABI compatibility" is actually ensured, but
> some don't. [ruby-core:36108]
>
>
> So, it would be good to discard ABI compatibility in 2.0.
> Each version of Ruby (including TEENY) should have their own
> library path, such as:
>
>    /usr/lib/ruby/2.0.0/
>    /usr/lib/ruby/2.0.1/
>    /usr/lib/ruby/2.0.2/
>    etc.
>
> In this policy, all the users have to re-install libraries that
> they are using whenever ruby is upgraded.  It's a pain.  But
> recently, many libraries are gems.
>
>
> So I'd like to ask a question to rubygems developers; is it
> technically possible to automatically re-install all gems from
> old Ruby installation to new one when Ruby is upgraded (or
> during "make install")?
>
> I have two concerns.  One is a user-custom gem that a user
> created and not published in rubygems.org.  Is there a custom
> gem or its source necessarily remained?
>
> The other is a binary gem that includes a compiled extension
> library.  TEENY release will be done about every year.  This
> is not a technical issue, but do you think it is reasonable
> to make gem developers release their binary gems when new
> Ruby is released?  And, is it possible to allow users to use,
> with a new Ruby, an old gem that does not support the new Ruby
> with a warning?
>
> And isn't there another concern I missed?

Actually you have missed that there are Linux distros, which are 
repackaging the gems. I am speaking of Fedora and RHEL. It would cause a 
lot of troubles.

Vit

>
>
> Note that this is currently just my idea, not a decision at
> all.  What do you think?
>

[ruby-core:40584] Re: Can rubygems save us from "binary-compatibility hell"?

[Eric Hodel]

On Oct 31, 2011, at 2:41 PM, Vít Ondruch wrote:
> Actually you have missed that there are Linux distros, which are repackaging the gems. I am speaking of Fedora and RHEL. It would cause a lot of troubles.

What troubles?

[ruby-core:40651] Re: Can rubygems save us from "binary-compatibility hell"?

[Vít Ondruch]

Dne 1.11.2011 0:05, Eric Hodel napsal(a):
> On Oct 31, 2011, at 2:41 PM, Vít Ondruch wrote:
>> Actually you have missed that there are Linux distros, which are repackaging the gems. I am speaking of Fedora and RHEL. It would cause a lot of troubles.
> What troubles?

Because you had to rebuild packaged gems after even the tiniest update 
of Ruby. There is currently more then 200 gems in Fedora, so that is a 
lot of work to rebuild them. Even the smallest Rails stack counts almost 
50 gems.

[ruby-core:40655] Re: Can rubygems save us from "binary-compatibility hell"?

[Eric Hodel]

On Nov 1, 2011, at 2:03 PM, Vít Ondruch wrote:
> Dne 1.11.2011 0:05, Eric Hodel napsal(a):
>> On Oct 31, 2011, at 2:41 PM, Vít Ondruch wrote:
>>> Actually you have missed that there are Linux distros, which are repackaging the gems. I am speaking of Fedora and RHEL. It would cause a lot of troubles.
>> What troubles?
> 
> Because you had to rebuild packaged gems after even the tiniest update of Ruby.

I do not recall rebuilding my gems after updating between Ruby patch levels or even between versions in the 1.9 series, that is why the library version is 1.9.1.

Have I forgotten needing to rebuild?  Can you show me a ticket where API broke?

> There is currently more then 200 gems in Fedora, so that is a lot of work to rebuild them. Even the smallest Rails stack counts almost 50 gems.

This process should be automateable by Fedora and other Linux distributions.

I don't think Ruby should restrict itself because repackagers have a manual process.

If there is something Ruby can do to help Fedora, etc. automate packaging I think Ruby should do it.

[ruby-core:40659] Re: Can rubygems save us from "binary-compatibility hell"?

[Vít Ondruch]

Dne 1.11.2011 22:31, Eric Hodel napsal(a):
> On Nov 1, 2011, at 2:03 PM, Vít Ondruch wrote:
>> Dne 1.11.2011 0:05, Eric Hodel napsal(a):
>>> On Oct 31, 2011, at 2:41 PM, Vít Ondruch wrote:
>>>> Actually you have missed that there are Linux distros, which are repackaging the gems. I am speaking of Fedora and RHEL. It would cause a lot of troubles.
>>> What troubles?
>> Because you had to rebuild packaged gems after even the tiniest update of Ruby.
> I do not recall rebuilding my gems after updating between Ruby patch levels or even between versions in the 1.9 series, that is why the library version is 1.9.1.
>
> Have I forgotten needing to rebuild?  Can you show me a ticket where API broke?

No, this was not necessary for Ruby 1.9 and thats fine. However the 
Yusuke's proposal was the opposite, i.e. break ABI with every release 
and I am against it.

>> There is currently more then 200 gems in Fedora, so that is a lot of work to rebuild them. Even the smallest Rails stack counts almost 50 gems.
> This process should be automateable by Fedora and other Linux distributions.
>
> I don't think Ruby should restrict itself because repackagers have a manual process.
>
> If there is something Ruby can do to help Fedora, etc. automate packaging I think Ruby should do it.

Yes, there is one think Ruby can do and it is to preserve ABI for major 
versions, i.e. 1.8, 1.9 or 2.0.

And automated repackaging is dream as long as there is no authority 
which ensures at least some basic standards in gems. I am speaking 
namely about licenses, bundled libraries, failing unit tests etc.

[ruby-core:40660] Re: Can rubygems save us from "binary-compatibility hell"?

[Eric Hodel]

On Nov 1, 2011, at 3:25 PM, Vít Ondruch wrote:
> No, this was not necessary for Ruby 1.9 and thats fine. However the Yusuke's proposal was the opposite, i.e. break ABI with every release and I am against it.

I see.

>> If there is something Ruby can do to help Fedora, etc. automate packaging I think Ruby should do it.
> 
> Yes, there is one think Ruby can do and it is to preserve ABI for major versions, i.e. 1.8, 1.9 or 2.0.
> 
> And automated repackaging is dream as long as there is no authority which ensures at least some basic standards in gems. I am speaking namely about licenses, bundled libraries, failing unit tests etc.

What about automated repackaging just for gems shipped with Ruby?  I think Ruby already has most of these standards.

[ruby-core:40879] Re: Can rubygems save us from "binary-compatibility hell"?

[Antonio Terceiro]

[-- Attachment #1: Type: text/plain, Size: 1099 bytes --]

Vít Ondruch escreveu isso aí:
> Dne 1.11.2011 0:05, Eric Hodel napsal(a):
> >On Oct 31, 2011, at 2:41 PM, Vít Ondruch wrote:
> >>Actually you have missed that there are Linux distros, which are repackaging the gems. I am speaking of Fedora and RHEL. It would cause a lot of troubles.
> >What troubles?
> 
> Because you had to rebuild packaged gems after even the tiniest
> update of Ruby. There is currently more then 200 gems in Fedora, so
> that is a lot of work to rebuild them. Even the smallest Rails stack
> counts almost 50 gems.

Same for Debian. But the rebuild is automated and it is supposed to be
needed once a year, so it's probably ok-ish from a Debian POV.

A minor point: breaking ABI on changes to the third component of a
version number is somehow counter-intuitive, and it would be nice if
these ABI-breaking versions were 2.1.x, 2.2.x, 2.3.x, ... instead of
2.0.1, 2.0.2, 2.0.3, ...

This is how Python (2.7.x, 2.8.x, ...) and Perl (5.10.x, 5.12.x,
5.14.x,) do it.

-- 
Antonio Terceiro <terceiro@softwarelivre.org>
http://softwarelivre.org/terceiro



[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[ruby-core:40918] Re: Fwd: Re: Can rubygems save us from "binary-compatibility hell"?

[Yusuke Endoh]

Hello,

Thank you all for discussing this proposal.
Sorry I have not followed all replies yet, but I will.
This is a reply to Eric and Luis.


2011/10/22 Eric Hodel <drbrain@segment7.net>:
>> Note that this is currently just my idea, not a decision at
>> all.  What do you think?
>
> I think your idea is possible and easy to implement.

Such confidence!



2011/10/22 Luis Lavena <luislavena@gmail.com>:
> But, this is a problem for more complex gems like EventMachine,
> nokogiri and others that depend on externals (openssl, libxml2).

Sorry I can't understand.  Why does it become more difficult if
it depends on external library?


> More than that, there is an existing problem with RubyGems that forced
> us to create "fat-binary" gems which include binaries for 1.8.x and
> 1.9.x so gem can be installed properly.

You mean "fat-binary" will become fatter to include binaries for
1.8, 1.9, 2.0.0, 2.0.1, ..., right?  Indeed it looks uncool, but
so matter in practice?  Aren't Eric planning to address this issue?


> While 1.8.x usage has been reduced lately, it is still present.

Slightly off topic, 1.8 will be dying when 2.0.0 is released.
(2.0.0 will be released at Feb. 2013, and 1.8 will be abandoned
at June 2013)  So I don't feel like caring about 1.8.


> Perhaps RubyGems' Specification will require to be extended to a list
> of versions the gem is compatible with and not just one unique
> version?

Agreed.


> Those are my only concerns: gem authors overhead when packaging binary
> gems and making more easy to gem users install them without falling
> into the compile-your-own rabbit hole.

Yes, in a sense, this proposal intends to move the overhead from
the core team to gem authors.  I wonder if the overhead is too
heavy or not.  I guess the 2.0.x will be released about once a year
(or lesser frequent).

-- 
Yusuke Endoh <mame@tsg.ne.jp>

[ruby-core:40919] Re: Fwd: Re: Can rubygems save us from "binary-compatibility hell"?

[Luis Lavena]

On Thu, Nov 10, 2011 at 4:38 PM, Yusuke Endoh <mame@tsg.ne.jp> wrote:
> Hello,
>
> 2011/10/22 Luis Lavena <luislavena@gmail.com>:
>> But, this is a problem for more complex gems like EventMachine,
>> nokogiri and others that depend on externals (openssl, libxml2).
>
> Sorry I can't understand.  Why does it become more difficult if
> it depends on external library?
>

These gems depends on 3rd party libraries (OpenSSL, libxml2) which,
under the environment I'm worried about (Windows) are not present by
default.

This increases the burden on gem authors to compile binaries for every
new API version of Ruby.

>
>> More than that, there is an existing problem with RubyGems that forced
>> us to create "fat-binary" gems which include binaries for 1.8.x and
>> 1.9.x so gem can be installed properly.
>
> You mean "fat-binary" will become fatter to include binaries for
> 1.8, 1.9, 2.0.0, 2.0.1, ..., right?  Indeed it looks uncool, but
> so matter in practice?  Aren't Eric planning to address this issue?
>

See above, if gem authors don't do that, then installation of complex
gems like nokogiri, EventMachine with SSL support or others will be
complicated.

For every new Ruby version that changes ABI, gem authors will require
to release a newer binary that either bundles all the supported
versions of Ruby for his gem or drop those versions from the
dependency.

I believe what Eric will address is the be able to define a dependency
under which Ruby you can install certain gem.

>
>> While 1.8.x usage has been reduced lately, it is still present.
>
> Slightly off topic, 1.8 will be dying when 2.0.0 is released.
> (2.0.0 will be released at Feb. 2013, and 1.8 will be abandoned
> at June 2013)  So I don't feel like caring about 1.8.
>

1.8 is just an example, instead of 1.8 it will be come 1.9.1 versus
2.0.0 versus 2.0.1

Gem authors that want to support 1.9.3 and newer version of Ruby will
still need to compile their extension multiple times to be able to
generate fat-binary gems.

>
>> Those are my only concerns: gem authors overhead when packaging binary
>> gems and making more easy to gem users install them without falling
>> into the compile-your-own rabbit hole.
>
> Yes, in a sense, this proposal intends to move the overhead from
> the core team to gem authors.  I wonder if the overhead is too
> heavy or not.  I guess the 2.0.x will be released about once a year
> (or lesser frequent).
>

I think that will depend on what Eric comes up with.

Also will depend if the gem author wants to drop support for older
versions of Ruby.
-- 
Luis Lavena
AREA 17
-
Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupéry

[ruby-core:40920] Re: Fwd: Re: Can rubygems save us from "binary-compatibility hell"?

[Yusuke Endoh]

Hello,

2011/11/11 Luis Lavena <luislavena@gmail.com>:
> These gems depends on 3rd party libraries (OpenSSL, libxml2) which,
> under the environment I'm worried about (Windows) are not present by
> default.
>
> This increases the burden on gem authors to compile binaries for every
> new API version of Ruby.

Sorry I cannot get your point yet.　Gem developer already has both
developement environment and knowledge for building his/har gems.
So why does the burden increase in comparison to a simple gem?


> For every new Ruby version that changes ABI, gem authors will require
> to release a newer binary that either bundles all the supported
> versions of Ruby for his gem or drop those versions from the
> dependency.

Yes.  This proposal increases the task of all gem developers.

I know no one will be happy to have extra work.  But I think that
it is too unhappy for the core team to ensure ABI compatiblility.
So I'm searching another way to share the pain with gem authors.


> I believe what Eric will address is the be able to define a dependency
> under which Ruby you can install certain gem.
>
>>
>>> While 1.8.x usage has been reduced lately, it is still present.
>>
>> Slightly off topic, 1.8 will be dying when 2.0.0 is released.
>> (2.0.0 will be released at Feb. 2013, and 1.8 will be abandoned
>> at June 2013)  So I don't feel like caring about 1.8.
>>
>
> 1.8 is just an example, instead of 1.8 it will be come 1.9.1 versus
> 2.0.0 versus 2.0.1

I know.  It is just off topic.

Thanks!

-- 
Yusuke Endoh <mame@tsg.ne.jp>

[ruby-core:40921] Re: Fwd: Re: Can rubygems save us from "binary-compatibility hell"?

[Luis Lavena]

2011/11/10 Yusuke Endoh <mame@tsg.ne.jp>:
> Hello,
>
> 2011/11/11 Luis Lavena <luislavena@gmail.com>:
>> These gems depends on 3rd party libraries (OpenSSL, libxml2) which,
>> under the environment I'm worried about (Windows) are not present by
>> default.
>>
>> This increases the burden on gem authors to compile binaries for every
>> new API version of Ruby.
>
> Sorry I cannot get your point yet.　Gem developer already has both
> developement environment and knowledge for building his/har gems.
> So why does the burden increase in comparison to a simple gem?
>

Sorry for not making myself clear:

Lot of gem authors releases gems in "ruby" and native for Windows.
Those native for Windows package the multiple version of the
extensions.

Most of the time this is done using cross-compilation tools were the
extension is cross compiled against different versions of the cross
compiled Ruby and then packaged inside the gem.

Adding more versions of Ruby increases the burden of compilation and
testing across these versions for another platform.

You don't see that scenario because you're using an OS that do provide
these support libraries required to successfully compile a gem, but on
Windows those are lacking (or require manual installation
instructions).

By adding multiple versions of Ruby to the same gem, it will increase
the gem size, the time it takes to compile and the burden on
testing/verification between versions for gem developers.

Please apologize if my english isn't clear, I'm not a native english
speaker and trying to explain the issue the more graphical as
possible.

-- 
Luis Lavena
AREA 17
-
Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupéry

[ruby-core:40925] Re: Fwd: Re: Can rubygems save us from "binary-compatibility hell"?

[Yusuke Endoh]

Hello,

2011/11/11 Luis Lavena <luislavena@gmail.com>:
> Most of the time this is done using cross-compilation tools were the
> extension is cross compiled against different versions of the cross
> compiled Ruby and then packaged inside the gem.

I didn't know a cross-compilation is used.  Why it is needed?
Sorry for my ignorance.


I had thought the development process of gems in windows is:

1. install windows.
2. install a compiler (such as vc++ or mingw).
3. install third party's libraries they need (such as openssl).
4. install the versions of Ruby which you're likely to support.
5. develop, build and test your gems by using the compiler,
   libraries and each version of Ruby.
6. create and release a gem package.


To add support for a new version of Ruby, the extra process is:

7. install the new version of Ruby.
8. build and test your gems by using the already-installed
   compiler, libraries and the new Ruby.
9. create and release a new gem package.


I think this extra process does not change even whether the gem
depends on the third party's libraries or not, because they have
already installed the libraries.
Maybe I'm missing something.  32 bit / 64 bit?


> Please apologize if my english isn't clear, I'm not a native english
> speaker and trying to explain the issue the more graphical as
> possible.

Don't mind, my English skill is definitely poorer than you :-)
I'm sorry to trouble you.

-- 
Yusuke Endoh <mame@tsg.ne.jp>

[ruby-core:40926] Re: Fwd: Re: Can rubygems save us from "binary-compatibility hell"?

[Luis Lavena]

On Thu, Nov 10, 2011 at 6:41 PM, Yusuke Endoh <mame@tsg.ne.jp> wrote:
>
> 2011/11/11 Luis Lavena <luislavena@gmail.com>:
>> Most of the time this is done using cross-compilation tools were the
>> extension is cross compiled against different versions of the cross
>> compiled Ruby and then packaged inside the gem.
>
> I didn't know a cross-compilation is used.  Why it is needed?
> Sorry for my ignorance.
>
>
> I had thought the development process of gems in windows is:
>
> 1. install windows.
> 2. install a compiler (such as vc++ or mingw).
> 3. install third party's libraries they need (such as openssl).
> 4. install the versions of Ruby which you're likely to support.
> 5. develop, build and test your gems by using the compiler,
>   libraries and each version of Ruby.
> 6. create and release a gem package.
>

1) Most of gem authors do not have Windows.

2) Most of the gem authors do not have or want to setup an entire
development environment on Windows to be able to compile gems.

3) Provisioning of the 3rd party libraries (and it's compilation) is
not easy on Windows.

4) Cross-compilation of the 3rd party libraries and the extension of
their gems all from one environment is easier and simpler.

See rake-compiler project:

http://github.com/luislavena/rake-compiler

>
> To add support for a new version of Ruby, the extra process is:
>
> 7. install the new version of Ruby.
> 8. build and test your gems by using the already-installed
>   compiler, libraries and the new Ruby.
> 9. create and release a new gem package.
>

You're omitting the issue: a binary gem will be for a single version
of Ruby because the binary extension was compiled just for the Ruby
used.

That is the reason of the fat-binary gem approach: it can contain the
same extension compiled against different versions of Ruby shared
library (for 1.8, 1.9.1 API versions).

Please ready the thread in RubyGems-devel where I explain this problem:

http://rubyforge.org/pipermail/rubygems-developers/2009-April/004522.html

>
> I think this extra process does not change even whether the gem
> depends on the third party's libraries or not, because they have
> already installed the libraries.

Windows != your average POSIX OS, most of the time getting the correct
3rd party library *or* binaries for Windows is impossible.

You're assuming that all Ruby developers have a high knowledge on how
to use a compiler and compile dependencies by themselves, which is not
true (not even for users on OSX or Linux).

-- 
Luis Lavena
AREA 17
-
Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupéry

[ruby-core:40928] Re: Fwd: Re: Can rubygems save us from "binary-compatibility hell"?

[Yusuke Endoh]

Hello,

2011/11/11 Luis Lavena <luislavena@gmail.com>:
> 1) Most of gem authors do not have Windows.

Surprised.  I thought gem developer must get the platform (or
a contributor who has the platform) if they want to support it.

Ah, I've remembered.  You said using wine for testing.
It is an interesting and challenging project, though I do never
think it is a canonical way...


>> To add support for a new version of Ruby, the extra process is:
>>
>> 7. install the new version of Ruby.
>> 8. build and test your gems by using the already-installed
>>   compiler, libraries and the new Ruby.
>> 9. create and release a new gem package.
>>
>
> You're omitting the issue: a binary gem will be for a single version
> of Ruby because the binary extension was compiled just for the Ruby
> used.

Ah, sorry.

9. create a fat gem package by adding to the old gem a new binary
   for the new version of Ruby, and release it.

I'm not familiar with gem operation, so I'm curious; Is adding a
new binary to gem so difficult?


> Please ready the thread in RubyGems-devel where I explain this problem:
>
> http://rubyforge.org/pipermail/rubygems-developers/2009-April/004522.html

I will try...


>> I think this extra process does not change even whether the gem
>> depends on the third party's libraries or not, because they have
>> already installed the libraries.
>
> Windows != your average POSIX OS, most of the time getting the correct
> 3rd party library *or* binaries for Windows is impossible.

You mean windows binaries are rarely provided by the 3rd party,
and gem developer must compile it by themselves, right?

I know it is cumbersome, but isn't it a necessary task to make a
gem that uses the library in windows?  And, I cannot understand
how the difficulty is related with a frequent change of Ruby ABI.
Maybe there is the answer in the RubyGems-devel thread.


Thanks for your explaining patiently and sorry for my ignorance.

-- 
Yusuke Endoh <mame@tsg.ne.jp>