[ruby-core:94064] [Ruby master Feature#16029] Expose fstring related APIs to C-extensions

[ruby-core:94064] [Ruby master Feature#16029] Expose fstring related APIs to C-extensions

[jean.boussier]

Issue #16029 has been reported by byroot (Jean Boussier).

----------------------------------------
Feature #16029: Expose fstring related APIs to C-extensions
https://bugs.ruby-lang.org/issues/16029

* Author: byroot (Jean Boussier)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
----------------------------------------
As discussed with @tenderlove here: https://github.com/ruby/ruby/pull/2287#issuecomment-513865160

We'd like to update various data format parsers (JSON, MessagePack, etc) to add the possibility to deduplicate strings while parsing.

But unfortunately the `rb_fstring_*` family of functions isn't available to C-extensions, so the only available fallback is `rb_funcall(str, rb_intern("-@"))` which most parsers will likely consider too slow. So the various `rb_fstring_*` functions would need to be public.

Proposed patch: https://github.com/ruby/ruby/pull/2299



-- 
https://bugs.ruby-lang.org/

[ruby-core:94761] [Ruby master Feature#16029] Expose fstring related APIs to C-extensions

[jean.boussier]

Issue #16029 has been updated by byroot (Jean Boussier).


@mame this feature was added by @nobu in the last developer's meeting issue [#15996], but I suppose it wasn't discussed by lack of time.

Should I just add it back to the next developer's meeting issue once it is created?

----------------------------------------
Feature #16029: Expose fstring related APIs to C-extensions
https://bugs.ruby-lang.org/issues/16029#change-81375

* Author: byroot (Jean Boussier)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
----------------------------------------
As discussed with @tenderlove here: https://github.com/ruby/ruby/pull/2287#issuecomment-513865160

We'd like to update various data format parsers (JSON, MessagePack, etc) to add the possibility to deduplicate strings while parsing.

But unfortunately the `rb_fstring_*` family of functions isn't available to C-extensions, so the only available fallback is `rb_funcall(str, rb_intern("-@"))` which most parsers will likely consider too slow. So the various `rb_fstring_*` functions would need to be public.

Proposed patch: https://github.com/ruby/ruby/pull/2299



-- 
https://bugs.ruby-lang.org/

[ruby-core:94763] [Ruby master Feature#16029] Expose fstring related APIs to C-extensions

[mame]

Issue #16029 has been updated by mame (Yusuke Endoh).


@ko1 and @nobu discussed the issue.  The memo says:

> ko1, nobu: now it is difficult to expose them because of current implementation.

I didn't follow the discussion, so I don't know the detail.  @ko1 , @nobu , could you explain them?

----------------------------------------
Feature #16029: Expose fstring related APIs to C-extensions
https://bugs.ruby-lang.org/issues/16029#change-81377

* Author: byroot (Jean Boussier)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
----------------------------------------
As discussed with @tenderlove here: https://github.com/ruby/ruby/pull/2287#issuecomment-513865160

We'd like to update various data format parsers (JSON, MessagePack, etc) to add the possibility to deduplicate strings while parsing.

But unfortunately the `rb_fstring_*` family of functions isn't available to C-extensions, so the only available fallback is `rb_funcall(str, rb_intern("-@"))` which most parsers will likely consider too slow. So the various `rb_fstring_*` functions would need to be public.

Proposed patch: https://github.com/ruby/ruby/pull/2299



-- 
https://bugs.ruby-lang.org/

[ruby-core:95269] [Ruby master Feature#16029] Expose fstring related APIs to C-extensions

[sam.saffron]

Issue #16029 has been updated by sam.saffron (Sam Saffron).


I think the larger change here is allowing for a new type of API. 

From a performance perspective the people using the new API would like to avoid allocating an RVALUE unless needed, the current fstring APIs require an RVALUE unless you give it a constant string I think?

Perhaps what we need here is the ability to ask Ruby: "Hey do you have an fstring for cstr X? If so then you use it, otherwise you do the slow path of allocating an RVALUE and passing it in to the fstring function.





----------------------------------------
Feature #16029: Expose fstring related APIs to C-extensions
https://bugs.ruby-lang.org/issues/16029#change-81945

* Author: byroot (Jean Boussier)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
----------------------------------------
As discussed with @tenderlove here: https://github.com/ruby/ruby/pull/2287#issuecomment-513865160

We'd like to update various data format parsers (JSON, MessagePack, etc) to add the possibility to deduplicate strings while parsing.

But unfortunately the `rb_fstring_*` family of functions isn't available to C-extensions, so the only available fallback is `rb_funcall(str, rb_intern("-@"))` which most parsers will likely consider too slow. So the various `rb_fstring_*` functions would need to be public.

Proposed patch: https://github.com/ruby/ruby/pull/2299



-- 
https://bugs.ruby-lang.org/

[ruby-core:95276] [Ruby master Feature#16029] Expose fstring related APIs to C-extensions

[jean.boussier]

Issue #16029 has been updated by byroot (Jean Boussier).


@sam.saffron Unless I'm missing something, that's exactly what `rb_fstring_new / rb_fstring_cstr` does.

```
VALUE rb_fstring_new(const char *ptr, long len);
VALUE rb_fstring_cstr(const char *str);
```

AFAICT It does allocate an RVALUE to lookup the table, but it does it on the stack, so I think it's fine GC wise.

----------------------------------------
Feature #16029: Expose fstring related APIs to C-extensions
https://bugs.ruby-lang.org/issues/16029#change-81952

* Author: byroot (Jean Boussier)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
----------------------------------------
As discussed with @tenderlove here: https://github.com/ruby/ruby/pull/2287#issuecomment-513865160

We'd like to update various data format parsers (JSON, MessagePack, etc) to add the possibility to deduplicate strings while parsing.

But unfortunately the `rb_fstring_*` family of functions isn't available to C-extensions, so the only available fallback is `rb_funcall(str, rb_intern("-@"))` which most parsers will likely consider too slow. So the various `rb_fstring_*` functions would need to be public.

Proposed patch: https://github.com/ruby/ruby/pull/2299



-- 
https://bugs.ruby-lang.org/

[ruby-core:95285] [Ruby master Feature#16029] Expose fstring related APIs to C-extensions

[sam.saffron]

Issue #16029 has been updated by sam.saffron (Sam Saffron).


I think when it gets called it expects to reuse the memory allocated by the cstr eventually 

https://github.com//blob/96753e8475ee69537134ab3d966c3d25cb5c467c/string.c#L287-L292

So if your library is in charge of the memory for the object this is not desirable, you want to simply ask the question "do you have an fstring for the current string eg"

VALUE rb_fstring_lookup(char *ptr); 

This non const char means it would not take ownership and the thing can return Qnil if there is no fstring.

Then if 99.999% of the string your library has are already fstrings, the lookup becomes a super cheap function you can use to lookup. 



----------------------------------------
Feature #16029: Expose fstring related APIs to C-extensions
https://bugs.ruby-lang.org/issues/16029#change-81965

* Author: byroot (Jean Boussier)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
----------------------------------------
As discussed with @tenderlove here: https://github.com/ruby/ruby/pull/2287#issuecomment-513865160

We'd like to update various data format parsers (JSON, MessagePack, etc) to add the possibility to deduplicate strings while parsing.

But unfortunately the `rb_fstring_*` family of functions isn't available to C-extensions, so the only available fallback is `rb_funcall(str, rb_intern("-@"))` which most parsers will likely consider too slow. So the various `rb_fstring_*` functions would need to be public.

Proposed patch: https://github.com/ruby/ruby/pull/2299



-- 
https://bugs.ruby-lang.org/

[ruby-core:95316] [Ruby master Feature#16029] Expose fstring related APIs to C-extensions

[sam.saffron]

Issue #16029 has been updated by sam.saffron (Sam Saffron).


I was thinking something like?

```
VALUE
rb_fstring_lookup(char *ptr, rb_encoding *enc)
{
    st_data_t fstring;
    struct RString fake_str;
    setup_fake_str(&fake_str, ptr, len, ENCINDEX_US_ASCII)

    st_table *frozen_strings = rb_vm_fstring_table();

    if (!st_lookup(frozen_strings, (st_data_t)fake_str, &fstring)) {
	return Qnil;
    }

    return ret;
}

```

----------------------------------------
Feature #16029: Expose fstring related APIs to C-extensions
https://bugs.ruby-lang.org/issues/16029#change-82018

* Author: byroot (Jean Boussier)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
----------------------------------------
As discussed with @tenderlove here: https://github.com/ruby/ruby/pull/2287#issuecomment-513865160

We'd like to update various data format parsers (JSON, MessagePack, etc) to add the possibility to deduplicate strings while parsing.

But unfortunately the `rb_fstring_*` family of functions isn't available to C-extensions, so the only available fallback is `rb_funcall(str, rb_intern("-@"))` which most parsers will likely consider too slow. So the various `rb_fstring_*` functions would need to be public.

Proposed patch: https://github.com/ruby/ruby/pull/2299



-- 
https://bugs.ruby-lang.org/

[ruby-core:95381] [Ruby master Feature#16029] Expose fstring related APIs to C-extensions

[ko1]

Issue #16029 has been updated by ko1 (Koichi Sasada).


Hi.

(1) implementation

Current implementation can have issue (related to shared string) and this issue can cause something wrong behavior for C-extension.
Sorry, we need a time to confirm.

(2) naming

i think it should be `rb_str_...` prefix.



----------------------------------------
Feature #16029: Expose fstring related APIs to C-extensions
https://bugs.ruby-lang.org/issues/16029#change-82092

* Author: byroot (Jean Boussier)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
----------------------------------------
As discussed with @tenderlove here: https://github.com/ruby/ruby/pull/2287#issuecomment-513865160

We'd like to update various data format parsers (JSON, MessagePack, etc) to add the possibility to deduplicate strings while parsing.

But unfortunately the `rb_fstring_*` family of functions isn't available to C-extensions, so the only available fallback is `rb_funcall(str, rb_intern("-@"))` which most parsers will likely consider too slow. So the various `rb_fstring_*` functions would need to be public.

Proposed patch: https://github.com/ruby/ruby/pull/2299



-- 
https://bugs.ruby-lang.org/

[ruby-core:95403] [Ruby master Feature#16029] Expose fstring related APIs to C-extensions

[sam.saffron]

Issue #16029 has been updated by sam.saffron (Sam Saffron).


Koichi, 

What about rb_str_fstring_lookup and rb_str_fstring_lookup_enc? Both will not create strings so shared strings should not be a problem. 

To be honest creation can be somewhat inefficient, the one place I can see this being used is in DB drivers like PG where a consumer keeps asking for field names over and over. 

----------------------------------------
Feature #16029: Expose fstring related APIs to C-extensions
https://bugs.ruby-lang.org/issues/16029#change-82130

* Author: byroot (Jean Boussier)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
----------------------------------------
As discussed with @tenderlove here: https://github.com/ruby/ruby/pull/2287#issuecomment-513865160

We'd like to update various data format parsers (JSON, MessagePack, etc) to add the possibility to deduplicate strings while parsing.

But unfortunately the `rb_fstring_*` family of functions isn't available to C-extensions, so the only available fallback is `rb_funcall(str, rb_intern("-@"))` which most parsers will likely consider too slow. So the various `rb_fstring_*` functions would need to be public.

Proposed patch: https://github.com/ruby/ruby/pull/2299



-- 
https://bugs.ruby-lang.org/

[ruby-core:95575] [Ruby master Feature#16029] Expose fstring related APIs to C-extensions

[jean.boussier]

Issue #16029 has been updated by byroot (Jean Boussier).


> What about rb_str_fstring_lookup and rb_str_fstring_lookup_enc? 

I don't think a lookup would be enough for what I'd like to do.

A typical use case would be a JSON document with lots of duplicated strings. 

If we only lookup the `fstring` table, then only the strings already present in the codebase would be deduplicated. IMHO it would be much better to create them all as `fstrings`.

----------------------------------------
Feature #16029: Expose fstring related APIs to C-extensions
https://bugs.ruby-lang.org/issues/16029#change-82357

* Author: byroot (Jean Boussier)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
----------------------------------------
As discussed with @tenderlove here: https://github.com/ruby/ruby/pull/2287#issuecomment-513865160

We'd like to update various data format parsers (JSON, MessagePack, etc) to add the possibility to deduplicate strings while parsing.

But unfortunately the `rb_fstring_*` family of functions isn't available to C-extensions, so the only available fallback is `rb_funcall(str, rb_intern("-@"))` which most parsers will likely consider too slow. So the various `rb_fstring_*` functions would need to be public.

Proposed patch: https://github.com/ruby/ruby/pull/2299



-- 
https://bugs.ruby-lang.org/

[ruby-core:96607] [Ruby master Feature#16029] Expose fstring related APIs to C-extensions

[sam.saffron]

Issue #16029 has been updated by sam.saffron (Sam Saffron).


@byroot

I think it heavily depends on usage... MySQL gem / PG would benefit from "lookup" followed by "create fstring if missing" cause vast majority of string it is creating when querying tables. 

My proposal is for the minimal building block we could use for getting fstrings unconditionally which would offer a fast path for fstring reuse. 

----------------------------------------
Feature #16029: Expose fstring related APIs to C-extensions
https://bugs.ruby-lang.org/issues/16029#change-83575

* Author: byroot (Jean Boussier)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
----------------------------------------
As discussed with @tenderlove here: https://github.com/ruby/ruby/pull/2287#issuecomment-513865160

We'd like to update various data format parsers (JSON, MessagePack, etc) to add the possibility to deduplicate strings while parsing.

But unfortunately the `rb_fstring_*` family of functions isn't available to C-extensions, so the only available fallback is `rb_funcall(str, rb_intern("-@"))` which most parsers will likely consider too slow. So the various `rb_fstring_*` functions would need to be public.

Proposed patch: https://github.com/ruby/ruby/pull/2299



-- 
https://bugs.ruby-lang.org/

[ruby-core:96613] [Ruby master Feature#16029] Expose fstring related APIs to C-extensions

[jean.boussier]

Issue #16029 has been updated by byroot (Jean Boussier).


@sam.saffron I agree that both would be nice.

> then new fstrings can be created using todays awkward API

You mean `rb_funcall(str, rb_intern("-@"))` ?

----------------------------------------
Feature #16029: Expose fstring related APIs to C-extensions
https://bugs.ruby-lang.org/issues/16029#change-83582

* Author: byroot (Jean Boussier)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
----------------------------------------
As discussed with @tenderlove here: https://github.com/ruby/ruby/pull/2287#issuecomment-513865160

We'd like to update various data format parsers (JSON, MessagePack, etc) to add the possibility to deduplicate strings while parsing.

But unfortunately the `rb_fstring_*` family of functions isn't available to C-extensions, so the only available fallback is `rb_funcall(str, rb_intern("-@"))` which most parsers will likely consider too slow. So the various `rb_fstring_*` functions would need to be public.

Proposed patch: https://github.com/ruby/ruby/pull/2299



-- 
https://bugs.ruby-lang.org/

[ruby-core:96666] [Ruby master Feature#16029] Expose fstring related APIs to C-extensions

[sam.saffron]

Issue #16029 has been updated by sam.saffron (Sam Saffron).


@byroot, yeah. 

`rb_funcall(str, rb_intern("-@"))` works and is backwards compatible, but I entirely agree that it makes sense to add an official API as well that does not depend on dynamic invocation. 


----------------------------------------
Feature #16029: Expose fstring related APIs to C-extensions
https://bugs.ruby-lang.org/issues/16029#change-83648

* Author: byroot (Jean Boussier)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
----------------------------------------
As discussed with @tenderlove here: https://github.com/ruby/ruby/pull/2287#issuecomment-513865160

We'd like to update various data format parsers (JSON, MessagePack, etc) to add the possibility to deduplicate strings while parsing.

But unfortunately the `rb_fstring_*` family of functions isn't available to C-extensions, so the only available fallback is `rb_funcall(str, rb_intern("-@"))` which most parsers will likely consider too slow. So the various `rb_fstring_*` functions would need to be public.

Proposed patch: https://github.com/ruby/ruby/pull/2299



-- 
https://bugs.ruby-lang.org/