[ruby-core:25420] [Bug #2054] Onigurma Isn't Documented

[ruby-core:25420] [Bug #2054] Onigurma Isn't Documented

[Run Paint Run Run]

Bug #2054: Onigurma Isn't Documented
http://redmine.ruby-lang.org/issues/show/2054

Author: Run Paint Run Run
Status: Open, Priority: Normal
Category: doc
ruby -v: ruby 1.9.2dev (2009-09-05 trunk 24762) [i686-linux]

As discussed in ticket #1889, Onigurma needs documentation. http://www.geocities.jp/kosako3/oniguruma/doc/RE.txt provides a starting point. I'm happy to work on a patch, but reluctant to do so until my ARGF patch (#1635) is accepted; it's a lot of work if I'm doing it wrongly. ;-)


----------------------------------------
http://redmine.ruby-lang.org

[ruby-core:25421] [Bug #2054] Onigurma Isn't Documented

[Run Paint Run Run]

Issue #2054 has been updated by Run Paint Run Run.


("Oniguruma", even :-/).
----------------------------------------
http://redmine.ruby-lang.org/issues/show/2054

----------------------------------------
http://redmine.ruby-lang.org

[ruby-core:25449] Re: [Bug #2054] Onigurma Isn't Documented

[Yukihiro Matsumoto]

Hi,

In message "Re: [ruby-core:25420] [Bug #2054] Onigurma Isn't Documented"
    on Sun, 6 Sep 2009 04:42:06 +0900, Run Paint Run Run <redmine@ruby-lang.org> writes:

|As discussed in ticket #1889, Onigurma needs documentation. http://www.geocities.jp/kosako3/oniguruma/doc/RE.txt provides a starting point. I'm happy to work on a patch, but reluctant to do so until my ARGF patch (#1635) is accepted; it's a lot of work if I'm doing it wrongly. ;-)

FYI, #1635 is accepted and merged in to the trunk (r24764 and r24767).

							matz.

[ruby-core:25468] [Bug #2054] Onigurma Isn't Documented

[Run Paint Run Run]

Issue #2054 has been updated by Run Paint Run Run.

File 0001-Write-initial-draft-of-Oniguruma-documentation.patch added

I've attached a first draft. Is this along the right lines? If so, I'll proofread and address the omissions tomorrow.
----------------------------------------
http://redmine.ruby-lang.org/issues/show/2054

----------------------------------------
http://redmine.ruby-lang.org

[ruby-core:25486] [Bug #2054](Assigned) Onigurma Isn't Documented

[Yui NARUSE]

Issue #2054 has been updated by Yui NARUSE.

Status changed from Open to Assigned
Assigned to set to Yui NARUSE

Looking overt this, yes, your patch is along the right lines.

But some comments following:

+ *      "Go to 東京都".match(/\s\u{6771 4eac 90fd}/)
Your patch includes some non-ASCII characters.
But Ruby's C source should be in ASCII.

+ *      'haystack' =~ /hay/   #=> 0
/hay/ =~ 'haystack' is Ruby's style, bacause matching is Regexp's function.
String#match is also the same.

+ *  * <tt>/pat/n</tt> - US-ASCII
/n is ASCII-8BIT.
And these options also set Regexp::FIXED_ENCODING or ARG_ENCODING_NONE.
see [ruby-core:23193] more information.
----------------------------------------
http://redmine.ruby-lang.org/issues/show/2054

----------------------------------------
http://redmine.ruby-lang.org

[ruby-core:25491] [Bug #2054] Onigurma Isn't Documented

[Run Paint Run Run]

Issue #2054 has been updated by Run Paint Run Run.

File 0001-Re-draft-Oniguruma-documentation.patch added

Thanks, Yui. An updated patch is attached. I didn't mention ARG_ENCODING_NONE because it's not exposed by the API.
----------------------------------------
http://redmine.ruby-lang.org/issues/show/2054

----------------------------------------
http://redmine.ruby-lang.org

[ruby-core:25492] [Bug #2054] Onigurma Isn't Documented

[Run Paint Run Run]

Issue #2054 has been updated by Run Paint Run Run.

File 0001-Re-draft-Oniguruma-documentation.patch added

Tweak a rather awkward sentence.
----------------------------------------
http://redmine.ruby-lang.org/issues/show/2054

----------------------------------------
http://redmine.ruby-lang.org

[ruby-core:25495] [Bug #2054] Onigurma Isn't Documented

[Yui NARUSE]

Issue #2054 has been updated by Yui NARUSE.


+ *  literally backslash-escape that: <tt>\\\\\\</tt>.
Why 6 times?

+ *  * <tt>/\d/</tt> - A digit character (<tt>[0-9]</tt>)
+ *  * <tt>/\D/</tt> - A non-digit character (<tt>[^0-9]</tt>)
Add following:
 *  * <tt>/\h/</tt> - A hexdigit character (<tt>[0-9a-fA-F]</tt>)
 *  * <tt>/\H/</tt> - A non-hexdigit character (<tt>[^0-9a-fA-F]</tt>)

+ *  * <tt>{n,}</tt> - n or more times
Add *  * <tt>{,m}</tt> - m or less times

== Grouping

(?: ..) should be introduced also in this chapter.

+ *  Capture groups can be referred to by name when defined with the
+ *  <tt>(?<name>)</tt> construct.

Don't mention an alias (?'name'subexp) ?

+ *      /£(?<pounds>\d+)\.(?<pence>\d+)/.match("£3.67")
Pound Sign (U+00A3) is non-ASCII.
Dollar Sign (U+0024) is in ASCII (but Cent Sign is non-ASCII).

+ *    characters match <i>pat</i>, but doesn’t include those characters in the 
Right Single Quotation Mark (U+2019) is non-ASCII
Apostrophe (U+0027) is in ASCII.

+ *  A regexp can be matched only against a string whose encoding is the
+ *  same as, or a superset of, the regexp's encoding. If a match between

The word "superset" is misleading.
In Ruby 1.9, this relation is only US-ASCII and other ASCII-compatible encodings.
So "or regexp's encoding is US-ASCII and string's encoding is ASCII-compatible".

+ *  The <tt>Regexp#fixed_encoding?</tt> predicate indicates whether the regexp
Needs encoding modifiers set FIXED_ENCODING.
And /n doesn't set FIXED_ENCODING but warn.
  irb(main):001:0> /a/n=~"a\u3042"
  (irb):1: warning: regexp match /.../n against to UTF-8 string
  => 0


Don't mention following?
*  x-y     range from x to y
* ..&&..  intersection (low precedence at the next of ^)
* reluctant and possessive quantifier
* atomic group (?>subexp) 
* back reference \k<..>
* subexp call \g<..>
* substitution by literarl regexp matching
  /(?<foo>b\w+)/ =~ 'bar' #=> 0
  foo #=> "bar"

If you can describe about  /(a|b+)*/ , it will be helpful.
http://www.ruby-forum.com/topic/70726
----------------------------------------
http://redmine.ruby-lang.org/issues/show/2054

----------------------------------------
http://redmine.ruby-lang.org

[ruby-core:25496] [Bug #2054] Onigurma Isn't Documented

[Run Paint Run Run]

Issue #2054 has been updated by Run Paint Run Run.


>> + *  literally backslash-escape that: <tt>\\\\\\</tt>.
> Why 6 times?

I didn't know how else to persuade RDoc to render two consecutive literal backslashes in a fashion suitable for `ri` and HTML.

>> + *  * <tt>/\d/</tt> - A digit character (<tt>[0-9]</tt>)
>> + *  * <tt>/\D/</tt> - A non-digit character (<tt>[^0-9]</tt>)
> Add following:
>> *  * <tt>/\h/</tt> - A hexdigit character (<tt>[0-9a-fA-F]</tt>)
>> *  * <tt>/\H/</tt> - A non-hexdigit character (<tt>[^0-9a-fA-F]</tt>)

>> + *  * <tt>{n,}</tt> - n or more times
> Add *  * <tt>{,m}</tt> - m or less times

Done.

>> == Grouping

> (?: ..) should be introduced also in this chapter.

I can't introduce non-capturing parentheses until I've explained capturing, so I've merged the two sections and left the explanation where it is.

>> + *  Capture groups can be referred to by name when defined with the
>> + *  <tt>(?<name>)</tt> construct.

> Don't mention an alias (?'name'subexp) ?

It didn't seem worth it. The material is already dense, without describing alternative syntax which appears to have little merit of its own.

>> + *      /£(?<pounds>\d+)\.(?<pence>\d+)/.match("£3.67")
> Pound Sign (U+00A3) is non-ASCII.
> Dollar Sign (U+0024) is in ASCII (but Cent Sign is non-ASCII).

We need to re-think the ASCII restriction. Documenting a Unicode regular expression library without being able to use Unicode in the examples is somewhat limiting. See below for an alternative.

>> *  A regexp can be matched only against a string whose encoding is the
>> + *  same as, or a superset of, the regexp's encoding. If a match between

> The word "superset" is misleading.
>  In Ruby 1.9, this relation is only US-ASCII and other ASCII-compatible encodings.
> So "or regexp's encoding is US-ASCII and string's encoding is ASCII-compatible".

Programming Ruby uses the term in this context, and during other encoding discussions, which was why I'd adopted it. :-)

Changed to:

  A regexp can be matched against a string when they either share an encoding, or the regexp's encoding
  is  _US-ASCII_ and the string's encoding is ASCII-compatible.

I'm not keen on the phrasing, but hopefully it's more accurate.

>> + *  The <tt>Regexp#fixed_encoding?</tt> predicate indicates whether the regexp
> Needs encoding modifiers set FIXED_ENCODING.

I don't understand that part. Regexp::FIXED_ENCODING is a constant, so how can an encoding option "set" it? I think the current text makes clear that a non-ASCII encoding, i.e. one set by an encoding option, is "fixed". That is, the modifiers change the state to fixed, which the predicate queries. I'll play around with

> Don't mention following?

I didn't know I was writing a book. :-) It's already far more comprehensive that O'Reilly's Programming Ruby...

> * reluctant and possessive quantifier

I mentioned both of these under repetition. I used the more standard term of "lazy" for "reluctant".

> * subexp call \g<..>

This seemed particularly arcane; even Friedl doesn't mention it in his tome. ;-) I'll try and come up with an example use.

> * substitution by literarl regexp matching
>  /(?<foo>b\w+)/ =~ 'bar' #=> 0
>  foo #=> "bar"

Yes, I meant to mention that.

Given the scope of this tutorial, the character set limitations, the degree to which it's bloating re.c, and the way this text obscures the existing API documentation for Regexp, how about we move it to its own file? This could, presumably, either be included in re.c with RDoc's _include_ option, or, better, live separately from the API documentation yet still be accessible from `ri`. `ri Regexp` could display the API documentation and mention the existence of this document; `ri regexp` could display the current document. I can commit at least the rest of this week to writing some initial content, if necessary.
 
> If you can describe about  /(a|b+)*/ , it will be helpful.
> http://www.ruby-forum.com/topic/70726

Hmm... Constructing accurate, performant, and optimal regexps is a significant topic... I suppose I could add a "Top 10 Pitfalls" callout. 
----------------------------------------
http://redmine.ruby-lang.org/issues/show/2054

----------------------------------------
http://redmine.ruby-lang.org

[ruby-core:25497] [Bug #2054] Onigurma Isn't Documented

[Yui NARUSE]

Issue #2054 has been updated by Yui NARUSE.


> >> + *  Capture groups can be referred to by name when defined with the
> >> + *  <tt>(?<name>)</tt> construct.
> 
> > Don't mention an alias (?'name'subexp) ?
> 
> It didn't seem worth it. The material is already dense, without describing
> alternative syntax which appears to have little merit of its own.

I don't like (?'name'subexp) too,
but reference is read not only when people write code but when read code.

> >> + *      /￡(?<pounds>\d+)\.(?<pence>\d+)/.match("￡3.67")
> > Pound Sign (U+00A3) is non-ASCII.
> > Dollar Sign (U+0024) is in ASCII (but Cent Sign is non-ASCII).
> 
> We need to re-think the ASCII restriction. Documenting a Unicode regular
> expression library without being able to use Unicode in the examples is somewhat limiting.
> See below for an alternative.

Some editors still don't support UTF-8 or
some  environmets don't have fonts.

> Changed to:
> 
>   A regexp can be matched against a string when they either share an encoding, or the regexp's encoding
>   is  _US-ASCII_ and the string's encoding is ASCII-compatible.
> 
> I'm not keen on the phrasing, but hopefully it's more accurate.

good.

> >> + *  The <tt>Regexp#fixed_encoding?</tt> predicate indicates whether the regexp
> > Needs encoding modifiers set FIXED_ENCODING.
>
> I don't understand that part.

This means, /a/e equals Regexp.new("a".force_encoding("EUC-JP"), Regexp::FIXEDENCODING).

> Regexp::FIXED_ENCODING is a constant, so how can an encoding option "set" it?
> I think the current text makes clear that a non-ASCII encoding,
> i.e. one set by an encoding option, is "fixed".
> That is, the modifiers change the state to fixed, which the predicate queries.
> I'll play around with

Hm, ok.

> > Don't mention following?
>
> I didn't know I was writing a book. :-) It's already far more comprehensive that O'Reilly's Programming Ruby...

a reference shall be comprehensive :-(

> > * subexp call \g<..>
> 
> This seemed particularly arcane; even Friedl doesn't mention it in his tome. ;-)
> I'll try and come up with an example use.

exapmles are following:

  # match parenthes
  r = /\A(?<paren>\(\g<paren>*\))*\z/
  r =~ "()"       #=> 0
  r =~ "(()())()" #=> 0
  r =~ ")"        #=> nil

  # match json
  r = Regexp.compile(<<'__REGEXP__'strip, Regexp::EXTENDED)
  (?<json> \g<object> | \g<array> ){0}
  (?<begin-array>  \g<ws> \[ \g<ws> ){0}
  (?<begin-object> \g<ws> \{ \g<ws> ){0}
  (?<end-array>    \g<ws> \] \g<ws> ){0}
  (?<end-object>   \g<ws> \} \g<ws> ){0}
  (?<name-separator>  \g<ws> : \g<ws> ){0}
  (?<value-separator> \g<ws> , \g<ws> ){0}
  (?<ws> [\x20\t\n\r]* ){0}
  (?<value>  false | null | true | \g<object> | \g<array> | \g<number> | \g<string> ){0}
  (?<object> \g<begin-object> (?: \g<member> (?: \g<value-separator> \g<member> )* )? \g<end-object> ){0}
  (?<member> \g<string> \g<name-separator> \g<value> ){0}
  (?<array> \g<begin-array> (?: \g<value> (?: \g<value-separator> \g<value> )* )? \g<end-array> ){0}
  (?<number> \-? \g<int> \g<frac>? \g<exp>? ){0}
  (?<exp> [eE] [-+] [0-9]+ ){0}
  (?<frac> \. [0-9]+ ){0}
  (?<int> 0 | [1-9] [0-9]* ){0}
  (?<string> " \g<char>* " ){0}
  (?<char> [^\x00-\x1F"\\] | \\ (?: ["\\\/\b\f\n\r\t] | u [0-9a-fA-F]{4} ) ){0}
  \A\g<json>\z
  __REGEXP__
  r =~ '{"foo":1,"bar":null,"baz":true}' #=> 0

  # match simplified XML
  r = Regexp.compile(<<'__REGEXP__'strip, Regexp::EXTENDED)
  (?<element> \g<stag> \g<content>* \g<etag> ){0}
  (?<stag> < \g<name> \s* > ){0}
  (?<name> [a-zA-Z_:]+ ){0}
  (?<content> [^<&]+ (\g<element> | [^<&]+)* ){0}
  (?<etag> </ \k<name+1> >){0}
  \A\g<element>\z
  __REGEXP__

> > If you can describe about  /(a|b+)*/ , it will be helpful.
> > http://www.ruby-forum.com/topic/70726
> 
> Hmm... Constructing accurate, performant, and optimal regexps is a significant topic...
> I suppose I could add a "Top 10 Pitfalls" callout. 

This problem sometimes comes to ML.
But you can write, "there is a backtracking problem. see Mastering Regular Expression" :-)
http://swtch.com/~rsc/regexp/regexp1.html is also written in detail.

> Given the scope of this tutorial, the character set limitations, the degree to which it's
> bloating re.c, and the way this text obscures the existing API documentation for Regexp,
> how about we move it to its own file? This could, presumably, either be included in re.c
> with RDoc's _include_ option, or, better, live separately from the API documentation yet
> still be accessible from `ri`. `ri Regexp` could display the API documentation and
> mention the existence of this document; `ri regexp` could display the current document.
> I can commit at least the rest of this week to writing some initial content, if necessary.

I'm not familiar to rdoc but this seems good.
But distingish document by case seems dangerous for some case-insensitive FS, e.g. Windows.
Regexp-intro or some different name is good.
----------------------------------------
http://redmine.ruby-lang.org/issues/show/2054

----------------------------------------
http://redmine.ruby-lang.org

[ruby-core:25504] Re: [Bug #2054] Onigurma Isn't Documented

["Martin J. Dürst"]

On 2009/09/09 19:02, Yui NARUSE wrote:
> Issue #2054 has been updated by Yui NARUSE.

>>>> + *      /￡(?<pounds>\d+)\.(?<pence>\d+)/.match("￡3.67")
>>> Pound Sign (U+00A3) is non-ASCII.
>>> Dollar Sign (U+0024) is in ASCII (but Cent Sign is non-ASCII).
>> We need to re-think the ASCII restriction. Documenting a Unicode regular
>> expression library without being able to use Unicode in the examples is somewhat limiting.
>> See below for an alternative.
>
> Some editors still don't support UTF-8 or
> some  environmets don't have fonts.

I'm sure you can find examples of these, but they are getting rarer and 
rarer. If somebody cannot see all the details of an example, that's a 
pity, but I think it should be balanced against all those people who 
will see the example and understand it much better because they see the 
characters directly.

What would be more of a problem is that some committers' editors would 
mess up UTF-8 when they made changes elsewere in the file. Do we have 
such cases?

Maybe we can split out the examples with actual Unicode characters into 
a separate file?

Regards,    Martin.

-- 
#-# Martin J. Dürst, Professor, Aoyama Gakuin University
#-# http://www.sw.it.aoyama.ac.jp   mailto:duerst@it.aoyama.ac.jp

[ruby-core:25508] Re: [Bug #2054] Onigurma Isn't Documented

[NARUSE, Yui]

2009/09/10 11:31, "Martin J. Dürst" wrote:
> On 2009/09/09 19:02, Yui NARUSE wrote:
>> Issue #2054 has been updated by Yui NARUSE.
>>>>> + * /￡(?<pounds>\d+)\.(?<pence>\d+)/.match("￡3.67")
>>>> Pound Sign (U+00A3) is non-ASCII.
>>>> Dollar Sign (U+0024) is in ASCII (but Cent Sign is non-ASCII).
>>> We need to re-think the ASCII restriction. Documenting a Unicode regular
>>> expression library without being able to use Unicode in the examples
>>> is somewhat limiting.
>>> See below for an alternative.
>>
>> Some editors still don't support UTF-8 or
>> some environmets don't have fonts.
>
> I'm sure you can find examples of these, but they are getting rarer and
> rarer. If somebody cannot see all the details of an example, that's a
> pity, but I think it should be balanced against all those people who
> will see the example and understand it much better because they see the
> characters directly.
>
> What would be more of a problem is that some committers' editors would
> mess up UTF-8 when they made changes elsewere in the file. Do we have
> such cases?

My main editor on Windows, Peggy Pro's internal code is still CP932.
http://www.anchorsystems.jp/anchor/ashp/peggy/pegindex.html

If I'm only one committer who use such editor, i can migrate.

> Maybe we can split out the examples with actual Unicode characters into
> a separate file?

I heard rdoc can it.

-- 
NARUSE, Yui  <naruse@airemix.jp>

[ruby-core:25522] [Bug #2054] Onigurma Isn't Documented

[Run Paint Run Run]

Issue #2054 has been updated by Run Paint Run Run.

File 0001-Re-draft-Oniguruma-documentation.patch added

The attached addresses the concerns raised. I'll proofread it when we're finished.

I'd intended to create a module called something like _Doc_, _Ref_, or _Tut_, then add this document as the class documentation for, say, Doc::Regexp. This would address the case-sensitive filenames issue and separate tutorial content from API reference. However, this led to `ri Doc::Regexp` displaying the tutorial twice. I've yet to understand why.

I then hoped to use the RDoc _:include:_ directive in re.c to pull in the tutorial from an external file, but this is only supported for Ruby source.

For now I've just called it _RegexpRef_.
----------------------------------------
http://redmine.ruby-lang.org/issues/show/2054

----------------------------------------
http://redmine.ruby-lang.org

[ruby-core:25523] [Bug #2054] Onigurma Isn't Documented

[Run Paint Run Run]

Issue #2054 has been updated by Run Paint Run Run.

File 0002-Rename-RegexpRef-to-Doc-Regexp.patch added

Rre-installing RDoc from scratch fixed the doubling bug; renamed to _Doc::Regexp_.
----------------------------------------
http://redmine.ruby-lang.org/issues/show/2054

----------------------------------------
http://redmine.ruby-lang.org

[ruby-core:25525] Re: [Bug #2054] Onigurma Isn't Documented

[Yukihiro Matsumoto]

Hi,

In message "Re: [ruby-core:25523] [Bug #2054] Onigurma Isn't Documented"
    on Fri, 11 Sep 2009 02:58:38 +0900, Run Paint Run Run <redmine@ruby-lang.org> writes:

|File 0002-Rename-RegexpRef-to-Doc-Regexp.patch added
|
|Rre-installing RDoc from scratch fixed the doubling bug; renamed to _Doc::Regexp_.

Single request: when you add a new patch to redmine, please mention
whether the patch requires previous ones or not.

							matz.

[ruby-core:25529] Re: [Bug #2054] Onigurma Isn't Documented

[Ron Mayer]

Yukihiro Matsumoto wrote:
> Single request: when you add a new patch to redmine, please mention
> whether the patch requires previous ones or not.

Sorry for bringing up the revision control system conversation again,
but this is a good example of the kind of issues GIT (or mercurial,etc)
makes easier.   When you get at a patch from GIT you'd be likely to see
a repository showing the history that the patch depends on too.

[ruby-core:25531] [Bug #2054] Onigurma Isn't Documented

[Yui NARUSE]

Issue #2054 has been updated by Yui NARUSE.


> I'd intended to create a module called something like _Doc_, _Ref_, or _Tut_,
> then add this document as the class documentation for, say, Doc::Regexp.
> This would address the case-sensitive filenames issue and separate tutorial
> content from API reference. However, this led to `ri Doc::Regexp` displaying
> the tutorial twice. I've yet to understand why.

Your patch works on my environment.
I'll merge this after I look over, thank you!

> I then hoped to use the RDoc _:include:_ directive in re.c to pull in the tutorial from an external file, but this is only supported for Ruby source.

You can tell drbrain and fix rdoc :-)

> Single request: when you add a new patch to redmine, please mention
> whether the patch requires previous ones or not.

This patch seems from git.
And patches from git is named as [changeset number]-[patch name].patch.
So we can find this is second patch by the name 0002-Rename-...patch

> but this is a good example of the kind of issues GIT (or mercurial,etc)
> makes easier.

You can publish patch as fork of http://github.com/shyouhei/ruby/tree.
There is any problem?
----------------------------------------
http://redmine.ruby-lang.org/issues/show/2054

----------------------------------------
http://redmine.ruby-lang.org