Python extension commands in git - request for policy change

Python extension commands in git - request for policy change

[Eric S. Raymond]

git presently contains one Python extension command, Pete Wycoff's p4
importer.  If my git-weave code is merged it will acquire another.  
I think we can expect more submissions of Python extensions in the
future, for two good reasons:

1. Python has a much richer type ontology than shell; there are many
things this makes relatively easy that are quite painful in shell.

2. While Perl shares advantage #1, compared to Python it's a
maintainability mess - much more difficult to read 6 months later.

On the other hand, 

3. Attitudes in the git dev group seem to be influenced by a
perception that up-to-date Python versions are not as reliably present
on our target platforms as Perl is.

4. Python has the disadvantage that comes with robust growth; you have
to specify "version x.y or later" as a dependency, mainly because new
modules keep getting getting folded into the stock Python environment.

Previous conversation on the list suggests that there has been a tacit
policy of managing these problems by (a) discouraging (though not entirely
forbidding) Python extensions, and (b) requiring extension submitters to
document some dependency on language version.

I think this is suboptimal.  By not forbidding the Python language
entirely, we guarantee having to deal with problems 3 and 4 anyway -
but by discouraging it, we're buying significant long-term
maintainability costs. It especially disturbed me to hear of Python
commands being recoded in C - that is definitely not the right
direction for reducing expected defect counts, if only because of
memory-management issues.

We're behind the best-practices curve here.  The major Linux
distributions, which have to deal with almost the same set of
tradeoffs we do, went to Python for pretty much all glue and
administration scripts outside /etc a decade ago, and the decision has
served them well.

That, among other things, means up-to-date versions of Python are
ubiquitous unless we're looking at Windows - in which case Perl and
shell actually become much bigger portability problems.  Mac OS X 
has kept up to date, too; Lion shipped 2.7.1 and that was a major
release back at this point.

To be fair, there was a time when being a bit twitchy about Python
version skew and deployment breadth was justified, but I believe that
time is now well past us. My basis for believing this is very simple -
I maintain a lot of Python code for systems programmers with stiff
portability requirements (things like reposurgeon, coverity-submit,
freecode-submit, shipper, and the Python tools in gpsd). I know what
kinds of bug reports I get and what kinds I don't, and in the last
few years "this breaks on my Python version" has gone from unusual
to doesn't-happen.

I think my experience with gpsd is particularly instructive.  Like
git, that project has a C core with Python wrappers and extension 
components. Like git, it gets deployed in a lot of odd places by people
who cannot afford the time to be tolerant about cross-platform
problems and are quite willing to hit the maintainer with a clue-bat
when they encounter them.  The good news is - they don't have to.

I should also point out that none of Mercurial's problems seem to
have anything to do with the fact that it's written in Python...

I think we can choose a better policy based on some simple premises.

1) In 2012, we can specify a "floor" Python version of 2.6 (shipped in
2008) and be pretty much guaranteed it will be anywhere we want to
deploy except Windows.  Windows will remain a problem because Python
isn't part of the stock install, but that's an equal or worse problem
for shell and Perl - and at least the Python project ships a binary
installer for Windows.

2) Python extension commands should test the Python version on startup
and die loudly but gracefully in the rare case that they don't find
what they need.

3) We should be unconditionally be encouraging extensions to move
from shell and Perl to Python.  This would be a clear net gain is
portability and maintainability.

4) We should be encouraging C code to move to Python, too.  There's
little gain in portability on this path because modern C has cleaned
up its act a lot, but the drop in expected bug loads would be well
worth the porting effort.  Segfaults are not your friend, and the x2 to
x5 drop in line count would do very good things for long-term
maintainability.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Live free or die; death is not the worst of evils.
	-- General George Stark.

Re: Python extension commands in git - request for policy change

[Nguyen Thai Ngoc Duy]

CCing msysgit. I vaguely remember they had problems with building
Python on Windows. I don't know if it's still an issue.

On Sun, Nov 25, 2012 at 9:44 AM, Eric S. Raymond <esr@thyrsus.com> wrote:
> git presently contains one Python extension command, Pete Wycoff's p4
> importer.  If my git-weave code is merged it will acquire another.
> I think we can expect more submissions of Python extensions in the
> future, for two good reasons:
>
> 1. Python has a much richer type ontology than shell; there are many
> things this makes relatively easy that are quite painful in shell.
>
> 2. While Perl shares advantage #1, compared to Python it's a
> maintainability mess - much more difficult to read 6 months later.
>
> On the other hand,
>
> 3. Attitudes in the git dev group seem to be influenced by a
> perception that up-to-date Python versions are not as reliably present
> on our target platforms as Perl is.
>
> 4. Python has the disadvantage that comes with robust growth; you have
> to specify "version x.y or later" as a dependency, mainly because new
> modules keep getting getting folded into the stock Python environment.

These may apply to other languages as well. Where do we draw a line?


> Previous conversation on the list suggests that there has been a tacit
> policy of managing these problems by (a) discouraging (though not entirely
> forbidding) Python extensions, and (b) requiring extension submitters to
> document some dependency on language version.
>
> I think this is suboptimal.  By not forbidding the Python language
> entirely, we guarantee having to deal with problems 3 and 4 anyway -
> but by discouraging it, we're buying significant long-term
> maintainability costs. It especially disturbed me to hear of Python
> commands being recoded in C - that is definitely not the right
> direction for reducing expected defect counts, if only because of
> memory-management issues.
>
> We're behind the best-practices curve here.  The major Linux
> distributions, which have to deal with almost the same set of
> tradeoffs we do, went to Python for pretty much all glue and
> administration scripts outside /etc a decade ago, and the decision has
> served them well.
>
> That, among other things, means up-to-date versions of Python are
> ubiquitous unless we're looking at Windows - in which case Perl and
> shell actually become much bigger portability problems.  Mac OS X
> has kept up to date, too; Lion shipped 2.7.1 and that was a major
> release back at this point.
>
> To be fair, there was a time when being a bit twitchy about Python
> version skew and deployment breadth was justified, but I believe that
> time is now well past us. My basis for believing this is very simple -
> I maintain a lot of Python code for systems programmers with stiff
> portability requirements (things like reposurgeon, coverity-submit,
> freecode-submit, shipper, and the Python tools in gpsd). I know what
> kinds of bug reports I get and what kinds I don't, and in the last
> few years "this breaks on my Python version" has gone from unusual
> to doesn't-happen.
>
> I think my experience with gpsd is particularly instructive.  Like
> git, that project has a C core with Python wrappers and extension
> components. Like git, it gets deployed in a lot of odd places by people
> who cannot afford the time to be tolerant about cross-platform
> problems and are quite willing to hit the maintainer with a clue-bat
> when they encounter them.  The good news is - they don't have to.
>
> I should also point out that none of Mercurial's problems seem to
> have anything to do with the fact that it's written in Python...
>
> I think we can choose a better policy based on some simple premises.
>
> 1) In 2012, we can specify a "floor" Python version of 2.6 (shipped in
> 2008) and be pretty much guaranteed it will be anywhere we want to
> deploy except Windows.  Windows will remain a problem because Python
> isn't part of the stock install, but that's an equal or worse problem
> for shell and Perl - and at least the Python project ships a binary
> installer for Windows.
>
> 2) Python extension commands should test the Python version on startup
> and die loudly but gracefully in the rare case that they don't find
> what they need.
>
> 3) We should be unconditionally be encouraging extensions to move
> from shell and Perl to Python.  This would be a clear net gain is
> portability and maintainability.
>
> 4) We should be encouraging C code to move to Python, too.  There's
> little gain in portability on this path because modern C has cleaned
> up its act a lot, but the drop in expected bug loads would be well
> worth the porting effort.  Segfaults are not your friend, and the x2 to
> x5 drop in line count would do very good things for long-term
> maintainability.
> --
>                 <a href="http://www.catb.org/~esr/">Eric S. Raymond</a>
>
> Live free or die; death is not the worst of evils.
>         -- General George Stark.
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Duy

-- 
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to msysgit@googlegroups.com
To unsubscribe from this group, send email to
msysgit+unsubscribe@googlegroups.com
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en

Re: Python extension commands in git - request for policy change

[Eric S. Raymond]

Nguyen Thai Ngoc Duy <pclouds@gmail.com>:
> These may apply to other languages as well. Where do we draw a line?

I'm in favor of the general policy of avoiding scripting languages
other than the top three most widely deployed.  At the moment that
means shell, Python, Perl; on present trends, in a few years Perl
(dropping in popularity) might be passed by Ruby on the way up.

Or, to put it another way, I'm *not* actually arguing that we ought
to encourage extension commands in Guile or Haskell or whatever else
the in-language-of-the-week is.  It would be bad for maintainability 
to fragment git's codebase that way.

What I'm arguing is that the tradeoffs within the group {C, shell, Perl,
Python} have changed in ways that favor Python as it has become more
stable and widely deployed.  So instead of grudgingly allowing a few
Python extensions in through a back door we ought to be encouraging
more use of it.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Python extension commands in git - request for policy change

[Felipe Contreras]

On Sun, Nov 25, 2012 at 3:44 AM, Eric S. Raymond <esr@thyrsus.com> wrote:
> git presently contains one Python extension command, Pete Wycoff's p4
> importer.  If my git-weave code is merged it will acquire another.
> I think we can expect more submissions of Python extensions in the
> future, for two good reasons:

According to the Git User Survey 2012, 1% of the responders used the
'git p4' tool. I don't know how much widely used 'git weave' would be,
but I wouldn't want to star changing policies for issues that are
practically non-existent or irrelevant for the vast majority of git
users.

> We're behind the best-practices curve here.  The major Linux
> distributions, which have to deal with almost the same set of
> tradeoffs we do, went to Python for pretty much all glue and
> administration scripts outside /etc a decade ago, and the decision has
> served them well.

If your friends jump off a bridge, would you? Yes, using python has
served them well, but as opposed to what? Other scripting languages? I
don't think so.

> I should also point out that none of Mercurial's problems seem to
> have anything to do with the fact that it's written in Python...

I agree that the _current_ major problems with mercurial are not
related to python, but once those are solved, who says python won't be
an issue?. That's an exercise in guesswork, because we can't know.

> I think we can choose a better policy based on some simple premises.
>
> 1) In 2012, we can specify a "floor" Python version of 2.6 (shipped in
> 2008) and be pretty much guaranteed it will be anywhere we want to
> deploy except Windows.  Windows will remain a problem because Python
> isn't part of the stock install, but that's an equal or worse problem
> for shell and Perl - and at least the Python project ships a binary
> installer for Windows.

What if my extension only supports python 2.7? Or what if my extension
wants to support 2.0?

> 2) Python extension commands should test the Python version on startup
> and die loudly but gracefully in the rare case that they don't find
> what they need.

Yes, they should _if_ they know what version they need. In my
extensions I really have no idea.

> 3) We should be unconditionally be encouraging extensions to move
> from shell and Perl to Python.  This would be a clear net gain is
> portability and maintainability.

NO! It's up to the developer to choose what language to use, and I
find it very chauvinist of you to say "python is better, so let's all
use python". So far you have listed a few advantages of python, but
you haven't explained so far what is wrong with shell and perl.

In fact, while advancing python you have made clear a problem with
python; the version requirements. So far I have *never* encountered a
problem with git because of my bash version, or my perl version. And
we haven't touched to the python3 mess yet. To me, those are
advantages of shell and perl.

Actually, I don't care if 'git foo' is written in perl, or shell, or
c; as long as it *works*. And I would hate it if 'git rebase' ever
told me that I need a newer version of python, or worst; that I don't
have python in my system (Arch Linux ships 'python2', not 'python').

And what if X developer that wrote Y tool loves perl, and hates
python? Or loves ruby? Are we going to kick him out of the project
because (s)he refuses to switch to python? Are we going to threat him
like an outsider, a rogue developer?

> 4) We should be encouraging C code to move to Python, too.  There's
> little gain in portability on this path because modern C has cleaned
> up its act a lot, but the drop in expected bug loads would be well
> worth the porting effort.  Segfaults are not your friend, and the x2 to
> x5 drop in line count would do very good things for long-term
> maintainability.

Definitely NO! I really really doubt git in python would be able to
achieve the same performance as git in c, but to show me wrong, it
wouldn't be very difficult to run a few measurements with python
dulwich *if* we are even to begin considering this point.

And are segmentation faults really that different from python's
exceptions? Not to the user.

And why not ruby instead?

If you are serious about this, I think there's a lot more to work to
show that there's anything wrong with the current situation, and that
other alternatives (e.g. ruby) are not good solutions. I for one would
like to see more tools move away from perl/shell, and into C. And
other tools move to ruby, but that it's up to the developers of those
tools, unless I myself do it.

Cheers.

-- 
Felipe Contreras

Re: Python extension commands in git - request for policy change

[Felipe Contreras]

On Sun, Nov 25, 2012 at 6:18 AM, Eric S. Raymond <esr@thyrsus.com> wrote:
> Nguyen Thai Ngoc Duy <pclouds@gmail.com>:
>> These may apply to other languages as well. Where do we draw a line?
>
> I'm in favor of the general policy of avoiding scripting languages
> other than the top three most widely deployed.  At the moment that
> means shell, Python, Perl; on present trends, in a few years Perl
> (dropping in popularity) might be passed by Ruby on the way up.

Top three according to whom?

According to TIOBE it's python, perl, and ruby (if you don't count VB
or PHP), and perl is beating ruby only by a small margin that will
probably disappear soon. However, shell has advantages none of the
above have.

http://1.1.1.4/bmi/www.tiobe.com/content/paperinfo/tpci/images/tpci_trends.png

Cheers.

-- 
Felipe Contreras

Re: Python extension commands in git - request for policy change

[Johannes Sixt]

Am 25.11.2012 03:44, schrieb Eric S. Raymond:
> That, among other things, means up-to-date versions of Python are
> ubiquitous unless we're looking at Windows - in which case Perl and
> shell actually become much bigger portability problems.

You seem to ignore that more than a quater of users are on Windows[1].
This is not negligible.

Therefore, we *are* looking at Windows. But where is there a portability
problem? There is a POSIX shell available in all git installations on
Windows. So is Perl. Python is not.

[1]
https://git.wiki.kernel.org/index.php/GitSurvey2011#10._On_which_operating_system.28s.29_do_you_use_Git.3F

> 4) We should be encouraging C code to move to Python, too.

Absolutely not. To achieve best portability, all code should move to C
instead.

-- Hannes

Re: Python extension commands in git - request for policy change

[Eric S. Raymond]

Felipe Contreras <felipe.contreras@gmail.com>:
> If your friends jump off a bridge, would you? Yes, using python has
> served them well, but as opposed to what? Other scripting languages? I
> don't think so.

The competition that Python won was *precisely* against other scripting
languages, notably shell and Perl.  Both used to be much more heavily
used in system scripting than they are now.

> What if my extension only supports python 2.7? Or what if my extension
> wants to support 2.0?

I propose that if 2.6 can't support it, then that should be considered
grounds to reject it.

> Yes, they should _if_ they know what version they need. In my
> extensions I really have no idea.

Then you shouldn't submit those extensions to be folded into core git.

> > 3) We should be unconditionally be encouraging extensions to move
> > from shell and Perl to Python.  This would be a clear net gain is
> > portability and maintainability.
> 
> NO! It's up to the developer to choose what language to use,

I agree.  You seem to be raising a lot of straw men.  'Encouragement'
does not equate to beating anyone who makes an unpopular choice over
the head.

I am also not suggesting that the whole git core ought to be hoicked 
over to Python.  I was thinking mainly about extension subcommands, 
not what's in libgit now.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Python extension commands in git - request for policy change

[Eric S. Raymond]

Felipe Contreras <felipe.contreras@gmail.com>:
> On Sun, Nov 25, 2012 at 6:18 AM, Eric S. Raymond <esr@thyrsus.com> wrote:
> > Nguyen Thai Ngoc Duy <pclouds@gmail.com>:
> >> These may apply to other languages as well. Where do we draw a line?
> >
> > I'm in favor of the general policy of avoiding scripting languages
> > other than the top three most widely deployed.  At the moment that
> > means shell, Python, Perl; on present trends, in a few years Perl
> > (dropping in popularity) might be passed by Ruby on the way up.
> 
> Top three according to whom?

According to the LOC counts in git's codebase.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Python extension commands in git - request for policy change

[Eric S. Raymond]

Johannes Sixt <j6t@kdbg.org>:
> Am 25.11.2012 03:44, schrieb Eric S. Raymond:
> > That, among other things, means up-to-date versions of Python are
> > ubiquitous unless we're looking at Windows - in which case Perl and
> > shell actually become much bigger portability problems.
> 
> You seem to ignore that more than a quater of users are on Windows[1].
> This is not negligible.

I'm not ignoring that at all.  There are questions of fact here:

Are Perl and a POSIX shell part of the stock installation of Windows?
I believe the answer is "no".  You are free to correct me, but if that's
true they don't have any obvious portability advantage over Python.
That means the 25% percent of Windows users are not actually a reason
to prefer them.

> Absolutely not. To achieve best portability, all code should move to C
> instead.

I wrote the (first) book on C portability.  I mean that literally -
"Portable C and Unix Systems Programming", Prentice-Hall 1987.  Please
don't feel insulted when I point out that over the last 25 years I
have probably forgotten more about this topic than you know.  Just
listen when I tell you that it is not at all obvious that raw C is the
maximally portable language.

It may very well be the case that some random scripting language (not
necessarily Python) achieves greater portability simply because its
maintainers get to pay more concentrated attention to the portability
of the environment bindings at the bottom of their C implementation than
we can.

In any case, I don't believe the difference in portability between raw
C and Python is large enough in either direction to be a reason to
favor either, and I speak as a domain expert on this issue.  This is
not Python advocacy talking; the same could be said of Perl or Ruby.

The real advantages of a scripting language are in maintainability and
expected defect rates, not portability.  The three relevant things we kbnow
from large-scale studies of software defect patterns are these:

1) Expected defect counts are predictable from LOC.

2) Moving to any given scripting language from C dramatically reduces LOC,
and thus expected defects over time.

3) Moving to any scripting language from C eliminates a class of
memory-management problems that dominate C defect statistics.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Re: Python extension commands in git - request for policy change

[Pat Thoyts]

On 25 November 2012 03:15, Nguyen Thai Ngoc Duy <pclouds@gmail.com> wrote:
> CCing msysgit. I vaguely remember they had problems with building
> Python on Windows. I don't know if it's still an issue.
>
> On Sun, Nov 25, 2012 at 9:44 AM, Eric S. Raymond <esr@thyrsus.com> wrote:
>> git presently contains one Python extension command, Pete Wycoff's p4
>> importer.  If my git-weave code is merged it will acquire another.
>> I think we can expect more submissions of Python extensions in the
>> future, for two good reasons:
>>
>> 1. Python has a much richer type ontology than shell; there are many
>> things this makes relatively easy that are quite painful in shell.
>>
>> 2. While Perl shares advantage #1, compared to Python it's a
>> maintainability mess - much more difficult to read 6 months later.
>>
>> On the other hand,
>>
>> 3. Attitudes in the git dev group seem to be influenced by a
>> perception that up-to-date Python versions are not as reliably present
>> on our target platforms as Perl is.
>>
>> 4. Python has the disadvantage that comes with robust growth; you have
>> to specify "version x.y or later" as a dependency, mainly because new
>> modules keep getting getting folded into the stock Python environment.
>
> These may apply to other languages as well. Where do we draw a line?
>
>
>> Previous conversation on the list suggests that there has been a tacit
>> policy of managing these problems by (a) discouraging (though not entirely
>> forbidding) Python extensions, and (b) requiring extension submitters to
>> document some dependency on language version.
>>
>> I think this is suboptimal.  By not forbidding the Python language
>> entirely, we guarantee having to deal with problems 3 and 4 anyway -
>> but by discouraging it, we're buying significant long-term
>> maintainability costs. It especially disturbed me to hear of Python
>> commands being recoded in C - that is definitely not the right
>> direction for reducing expected defect counts, if only because of
>> memory-management issues.
>>
>> We're behind the best-practices curve here.  The major Linux
>> distributions, which have to deal with almost the same set of
>> tradeoffs we do, went to Python for pretty much all glue and
>> administration scripts outside /etc a decade ago, and the decision has
>> served them well.
>>
>> That, among other things, means up-to-date versions of Python are
>> ubiquitous unless we're looking at Windows - in which case Perl and
>> shell actually become much bigger portability problems.  Mac OS X
>> has kept up to date, too; Lion shipped 2.7.1 and that was a major
>> release back at this point.
>>
>> To be fair, there was a time when being a bit twitchy about Python
>> version skew and deployment breadth was justified, but I believe that
>> time is now well past us. My basis for believing this is very simple -
>> I maintain a lot of Python code for systems programmers with stiff
>> portability requirements (things like reposurgeon, coverity-submit,
>> freecode-submit, shipper, and the Python tools in gpsd). I know what
>> kinds of bug reports I get and what kinds I don't, and in the last
>> few years "this breaks on my Python version" has gone from unusual
>> to doesn't-happen.
>>
>> I think my experience with gpsd is particularly instructive.  Like
>> git, that project has a C core with Python wrappers and extension
>> components. Like git, it gets deployed in a lot of odd places by people
>> who cannot afford the time to be tolerant about cross-platform
>> problems and are quite willing to hit the maintainer with a clue-bat
>> when they encounter them.  The good news is - they don't have to.
>>
>> I should also point out that none of Mercurial's problems seem to
>> have anything to do with the fact that it's written in Python...
>>
>> I think we can choose a better policy based on some simple premises.
>>
>> 1) In 2012, we can specify a "floor" Python version of 2.6 (shipped in
>> 2008) and be pretty much guaranteed it will be anywhere we want to
>> deploy except Windows.  Windows will remain a problem because Python
>> isn't part of the stock install, but that's an equal or worse problem
>> for shell and Perl - and at least the Python project ships a binary
>> installer for Windows.
>>
>> 2) Python extension commands should test the Python version on startup
>> and die loudly but gracefully in the rare case that they don't find
>> what they need.
>>
>> 3) We should be unconditionally be encouraging extensions to move
>> from shell and Perl to Python.  This would be a clear net gain is
>> portability and maintainability.
>>
>> 4) We should be encouraging C code to move to Python, too.  There's
>> little gain in portability on this path because modern C has cleaned
>> up its act a lot, but the drop in expected bug loads would be well
>> worth the porting effort.  Segfaults are not your friend, and the x2 to
>> x5 drop in line count would do very good things for long-term
>> maintainability.

Git for Windows simply ships everything we need to run git - so if a
desirable module requires a version of python, we will add that
version plus any required modules into the installer. We already have
a patch to provide python in the msysgit tree - it would just require
polishing up a little. I'm certain this is no problem for the other
windows port (cygwin) either.

-- 
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to msysgit@googlegroups.com
To unsubscribe from this group, send email to
msysgit+unsubscribe@googlegroups.com
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en

Re: Re: Python extension commands in git - request for policy change

[Eric S. Raymond]

Pat Thoyts <patthoyts@gmail.com>:
> Git for Windows simply ships everything we need to run git - so if a
> desirable module requires a version of python, we will add that
> version plus any required modules into the installer. We already have
> a patch to provide python in the msysgit tree - it would just require
> polishing up a little. I'm certain this is no problem for the other
> windows port (cygwin) either.

Thank you - I think this completely disposes of the "Windows is a blocker
for scripting language X" argument, with the case X = Python in point. 
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

-- 
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to msysgit@googlegroups.com
To unsubscribe from this group, send email to
msysgit+unsubscribe@googlegroups.com
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en

Re: Python extension commands in git - request for policy change

[Michael Haggerty]

On 11/25/2012 09:53 AM, Felipe Contreras wrote:
> On Sun, Nov 25, 2012 at 3:44 AM, Eric S. Raymond <esr@thyrsus.com> wrote:
>> 1) In 2012, we can specify a "floor" Python version of 2.6 (shipped in
>> 2008) and be pretty much guaranteed it will be anywhere we want to
>> deploy except Windows.  Windows will remain a problem because Python
>> isn't part of the stock install, but that's an equal or worse problem
>> for shell and Perl - and at least the Python project ships a binary
>> installer for Windows.
> 
> What if my extension only supports python 2.7? Or what if my extension
> wants to support 2.0?

There would obviously have to be a policy like "all Python code in core
git must run on any Python interpreter with 2.6 <= version < 3.0", just
as there are policies about what C and shell features are allowed.  If
you happen to want to support earlier versions of Python, I don't see
why anybody would stop you as long as your code also runs in the
mandated versions.

(In practice, backwards compatibility within Python versions 2.x is very
good and almost any code that runs in Python 2.6 would automatically run
in all later 2.x versions.  Moreover, the Python documentation covering
what is available in each version and the deltas between versions is
high-quality and easily available online.)

There is, of course, the awkward issue of how/when to transition to
Python 3.x, which is *not* backwards compatible with Python 2.x.  I
expect that when the time comes there will be volunteers (myself
included) willing to help adapt Python scripts to the new version, but
the problem shouldn't be minimized.

Of course Perl will have the same problem if Perl6 ever materializes.

>> 2) Python extension commands should test the Python version on startup
>> and die loudly but gracefully in the rare case that they don't find
>> what they need.
> 
> Yes, they should _if_ they know what version they need. In my
> extensions I really have no idea.

Then simply (with the help of the mailing list) ensure that your
extensions run under 2.6 (or whatever the chosen minimum version is) and
everything will be OK.  It is not an error to specify 2.6 as the minimum
version even though your script happens also to run on older versions :-)

>> 3) We should be unconditionally be encouraging extensions to move
>> from shell and Perl to Python.  This would be a clear net gain is
>> portability and maintainability.
> 
> NO! It's up to the developer to choose what language to use, and I
> find it very chauvinist of you to say "python is better, so let's all
> use python". So far you have listed a few advantages of python, but
> you haven't explained so far what is wrong with shell and perl.

Given that some languages are accepted in git-core and others are not,
it's already not "up to the developer to choose what language to use".
At best there is a short list of "blessed" languages, and the developer
can choose among only those.

> In fact, while advancing python you have made clear a problem with
> python; the version requirements. So far I have *never* encountered a
> problem with git because of my bash version, or my perl version. And
> we haven't touched to the python3 mess yet. To me, those are
> advantages of shell and perl.

On the contrary, there is *constant* traffic on the mailing list about
incompatibilities between different shell implementations (sh, dash,
bash, etc), not to mention those in other utilities (sed, grep, etc)
that one is forced to work with in shell scripts.  Compatibility is a
*huge* pain when developing shell code for git.  The fact that users
typically don't encounter such problems is due to the hard work of POSIX
lawyers on the mailing list correcting the compatibility errors of
mortal programmers.

> Actually, I don't care if 'git foo' is written in perl, or shell, or
> c; as long as it *works*. And I would hate it if 'git rebase' ever
> told me that I need a newer version of python, or worst; that I don't
> have python in my system (Arch Linux ships 'python2', not 'python').

The configure script would locate the correct interpreter and the build
would adjust the scripts' shebang lines, just as things are tweaked
within Perl scripts at build time.

>> 4) We should be encouraging C code to move to Python, too.  There's
>> little gain in portability on this path because modern C has cleaned
>> up its act a lot, but the drop in expected bug loads would be well
>> worth the porting effort.  Segfaults are not your friend, and the x2 to
>> x5 drop in line count would do very good things for long-term
>> maintainability.
> 
> Definitely NO! I really really doubt git in python would be able to
> achieve the same performance as git in c, but to show me wrong, it
> wouldn't be very difficult to run a few measurements with python
> dulwich *if* we are even to begin considering this point.
> 
> And are segmentation faults really that different from python's
> exceptions? Not to the user.

There is one huge difference: it C it is all too easy to write code that
leads to a security hole due to buffer overflows and other memory
management errors.  Code written in a scripting language are largely
immune to such problems (except of course for any such bugs in the
interpreter itself, but the testing of the interpreter is shared across
many projects and users).

It would be insane to rewrite performance-critical C code in any
scripting language, but there is a huge penumbra of code that is not
performance critical and that mutates rapidly.  Such code is much easier
to write and maintain in a sane scripting language if the portability
issues can be mastered.

The most important issues to consider when imagining a future with a
hybrid of code in C and some scripting language "X" are:

* Portability: is "X" available on all platforms targeted by git, in
  usable and mutually-compatible versions?

* Startup time: Is the time to start the "X" interpreter prohibitive?
  (On my computer, "python -c pass", which starts the Python
  interpreter and does nothing, takes about 24ms.)  This overhead would
  be incurred by every command that is not pure C.

* Should the scripting language access the C functionality only by
  calling pure-C executables or by dynamically or statically linking to
  a binary module interface?  If the former, then the granularity of
  interactions between "X" and C is necessarily coarse, and "X" cannot
  be used to implement anything but the outermost layer of
  functionality.  If the latter, then the way would be clear to
  implement much more of git in "X" (and lua would also be worth
  considering).

* Learning curve for developers: how difficult is it for a typical git
  developer to become conversant with "X", considering both (1) how
  likely is it that the typical git developer already knows "X" and
  (2) how straightforward and predictable is the language "X"?
  In this category I think that Python has a huge advantage over
  Perl, though certainly opinions will differ and Ruby would also be
  a contender.

Personally, I regret wasting my time programming pointer arithmetic in
git modules that are not performance-critical (and correcting bugs by
others in these areas).  And I'm tired of having an idea to improve a
git feature only to find that it is implemented in shell, where not even
arrays are available.  I would therefore welcome more friendliness
towards a decent scripting language in the git project.

Michael

-- 
Michael Haggerty
mhagger@alum.mit.edu
http://softwareswirl.blogspot.com/

Re: Python extension commands in git - request for policy change

[Eric S. Raymond]

Michael Haggerty <mhagger@alum.mit.edu>:
> There is, of course, the awkward issue of how/when to transition to
> Python 3.x, which is *not* backwards compatible with Python 2.x.  I
> expect that when the time comes there will be volunteers (myself
> included) willing to help adapt Python scripts to the new version, but
> the problem shouldn't be minimized.

2to3 actually does a pretty good job.  It doesn't reduce the
transition cost to zero, but I find it does reduce that cost to an
easily manageable level even on quite large codebases.

> It would be insane to rewrite performance-critical C code in any
> scripting language, but there is a huge penumbra of code that is not
> performance critical and that mutates rapidly.

Indeed.  In the git architecture there is a pretty clear dividing line -
to a first approximation, plumbing should remain C but porcelain should
probably not.  (Not that I am advocating forcing such a move - but it would
be good to allow it to happen.)

The 80-20 rule (80% of the execution time is spent in 20% of the code)
helps us here.  The *other* 80% of the code can move to a scripting
language with no significant performance loss.  To find out what needs
to stay in C, run a profiler!
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Python extension commands in git - request for policy change

[Felipe Contreras]

On Sun, Nov 25, 2012 at 10:53 AM, Eric S. Raymond <esr@thyrsus.com> wrote:
> Felipe Contreras <felipe.contreras@gmail.com>:
>> If your friends jump off a bridge, would you? Yes, using python has
>> served them well, but as opposed to what? Other scripting languages? I
>> don't think so.
>
> The competition that Python won was *precisely* against other scripting
> languages, notably shell and Perl.  Both used to be much more heavily
> used in system scripting than they are now.

Against shell and perl yes, not against the rest.

>> What if my extension only supports python 2.7? Or what if my extension
>> wants to support 2.0?
>
> I propose that if 2.6 can't support it, then that should be considered
> grounds to reject it.

Seems sensible, but I don't know what "rejection" would actually mean.
My "extensions" are on the way to the contrib area. Is the contrib
area supposed to have different rules? I don't know.

Either way, making a script work on python 2.6 is probably easier than
trying to "reject" it.

>> Yes, they should _if_ they know what version they need. In my
>> extensions I really have no idea.
>
> Then you shouldn't submit those extensions to be folded into core git.

Too late.

>> > 3) We should be unconditionally be encouraging extensions to move
>> > from shell and Perl to Python.  This would be a clear net gain is
>> > portability and maintainability.
>>
>> NO! It's up to the developer to choose what language to use,
>
> I agree.  You seem to be raising a lot of straw men.  'Encouragement'
> does not equate to beating anyone who makes an unpopular choice over
> the head.

I don't see what this means in practical terms. People are going to
write code in whatever language they want to write code in. How
exactly are "we" going to "encourage" them not to do that is not
entirely clear to me.

I don't think there's such a thing as "git leadership" that would be
able to take these policy decisions, and if there was one, I don't
think the evidence presented would be enough to weigh in either way.

> I am also not suggesting that the whole git core ought to be hoicked
> over to Python.  I was thinking mainly about extension subcommands,
> not what's in libgit now.

Subcommands are also probably more efficient in c. And lets remember
that most people use git through the *official* subcommands.

Cheers.

-- 
Felipe Contreras

Re: Python extension commands in git - request for policy change

[Nguyen Thai Ngoc Duy]

On Sun, Nov 25, 2012 at 5:44 PM, Michael Haggerty <mhagger@alum.mit.edu> wrote:
> On the contrary, there is *constant* traffic on the mailing list about
> incompatibilities between different shell implementations (sh, dash,
> bash, etc), not to mention those in other utilities (sed, grep, etc)
> that one is forced to work with in shell scripts.  Compatibility is a
> *huge* pain when developing shell code for git.  The fact that users
> typically don't encounter such problems is due to the hard work of POSIX
> lawyers on the mailing list correcting the compatibility errors of
> mortal programmers.

I think we still are in the process of moving away from shell-based
commands (not the shell interface), just not enough man power to do it
fast. The only shell-based command with active development is
git-submodule. So most shell PITA is in the test suite.

> The most important issues to consider when imagining a future with a
> hybrid of code in C and some scripting language "X" are:
>
> * Portability: is "X" available on all platforms targeted by git, in
>   usable and mutually-compatible versions?
>
> * Startup time: Is the time to start the "X" interpreter prohibitive?
>   (On my computer, "python -c pass", which starts the Python
>   interpreter and does nothing, takes about 24ms.)  This overhead would
>   be incurred by every command that is not pure C.
>
> * Should the scripting language access the C functionality only by
>   calling pure-C executables or by dynamically or statically linking to
>   a binary module interface?  If the former, then the granularity of
>   interactions between "X" and C is necessarily coarse, and "X" cannot
>   be used to implement anything but the outermost layer of
>   functionality.  If the latter, then the way would be clear to
>   implement much more of git in "X" (and lua would also be worth
>   considering).
>
> * Learning curve for developers: how difficult is it for a typical git
>   developer to become conversant with "X", considering both (1) how
>   likely is it that the typical git developer already knows "X" and
>   (2) how straightforward and predictable is the language "X"?
>   In this category I think that Python has a huge advantage over
>   Perl, though certainly opinions will differ and Ruby would also be
>   a contender.

* We might also need an embedded language variant, like Jeff's lua
experiment. I'd be nice if "X" can also take this role.
-- 
Duy

Re: Python extension commands in git - request for policy change

[Felipe Contreras]

On Sun, Nov 25, 2012 at 11:44 AM, Michael Haggerty <mhagger@alum.mit.edu> wrote:
> On 11/25/2012 09:53 AM, Felipe Contreras wrote:
>> On Sun, Nov 25, 2012 at 3:44 AM, Eric S. Raymond <esr@thyrsus.com> wrote:
>>> 1) In 2012, we can specify a "floor" Python version of 2.6 (shipped in
>>> 2008) and be pretty much guaranteed it will be anywhere we want to
>>> deploy except Windows.  Windows will remain a problem because Python
>>> isn't part of the stock install, but that's an equal or worse problem
>>> for shell and Perl - and at least the Python project ships a binary
>>> installer for Windows.
>>
>> What if my extension only supports python 2.7? Or what if my extension
>> wants to support 2.0?
>
> There would obviously have to be a policy like "all Python code in core
> git must run on any Python interpreter with 2.6 <= version < 3.0", just
> as there are policies about what C and shell features are allowed.  If
> you happen to want to support earlier versions of Python, I don't see
> why anybody would stop you as long as your code also runs in the
> mandated versions.

Of course, but there are experts in C and shell around, not so many
python experts. So if somebody sneaks in a python program that makes
use of features specific to python 2.7, I doubt anybody would notice.
And if they did, I doubt that would be reason enough for rejection,
supposing that porting to 2.6 would be difficult enough.

Anyway, I think this is all guesswork.

> Of course Perl will have the same problem if Perl6 ever materializes.

It *might*, it might not be as severe.

>>> 2) Python extension commands should test the Python version on startup
>>> and die loudly but gracefully in the rare case that they don't find
>>> what they need.
>>
>> Yes, they should _if_ they know what version they need. In my
>> extensions I really have no idea.
>
> Then simply (with the help of the mailing list) ensure that your
> extensions run under 2.6 (or whatever the chosen minimum version is) and
> everything will be OK.  It is not an error to specify 2.6 as the minimum
> version even though your script happens also to run on older versions :-)

Who would do that? I don't see a lot of people.

>>> 3) We should be unconditionally be encouraging extensions to move
>>> from shell and Perl to Python.  This would be a clear net gain is
>>> portability and maintainability.
>>
>> NO! It's up to the developer to choose what language to use, and I
>> find it very chauvinist of you to say "python is better, so let's all
>> use python". So far you have listed a few advantages of python, but
>> you haven't explained so far what is wrong with shell and perl.
>
> Given that some languages are accepted in git-core and others are not,
> it's already not "up to the developer to choose what language to use".
> At best there is a short list of "blessed" languages, and the developer
> can choose among only those.

They are not because they haven't been proposed. Things change.

>> In fact, while advancing python you have made clear a problem with
>> python; the version requirements. So far I have *never* encountered a
>> problem with git because of my bash version, or my perl version. And
>> we haven't touched to the python3 mess yet. To me, those are
>> advantages of shell and perl.
>
> On the contrary, there is *constant* traffic on the mailing list about
> incompatibilities between different shell implementations (sh, dash,
> bash, etc), not to mention those in other utilities (sed, grep, etc)
> that one is forced to work with in shell scripts.  Compatibility is a
> *huge* pain when developing shell code for git.  The fact that users
> typically don't encounter such problems is due to the hard work of POSIX
> lawyers on the mailing list correcting the compatibility errors of
> mortal programmers.

*Theoretical* incompatibilities on probably obscure systems. *I* have
never seen such compatibility issues *in practice*.

>> Actually, I don't care if 'git foo' is written in perl, or shell, or
>> c; as long as it *works*. And I would hate it if 'git rebase' ever
>> told me that I need a newer version of python, or worst; that I don't
>> have python in my system (Arch Linux ships 'python2', not 'python').
>
> The configure script would locate the correct interpreter and the build
> would adjust the scripts' shebang lines, just as things are tweaked
> within Perl scripts at build time.

Arch Linux doesn't use no configure script. And what if I'm building
git myself (I've hit the issue multiple times)? Perl might have
similar issues on other systems, but not on Arch Linux; /usr/bin/perl
is there.

>>> 4) We should be encouraging C code to move to Python, too.  There's
>>> little gain in portability on this path because modern C has cleaned
>>> up its act a lot, but the drop in expected bug loads would be well
>>> worth the porting effort.  Segfaults are not your friend, and the x2 to
>>> x5 drop in line count would do very good things for long-term
>>> maintainability.
>>
>> Definitely NO! I really really doubt git in python would be able to
>> achieve the same performance as git in c, but to show me wrong, it
>> wouldn't be very difficult to run a few measurements with python
>> dulwich *if* we are even to begin considering this point.
>>
>> And are segmentation faults really that different from python's
>> exceptions? Not to the user.
>
> There is one huge difference: it C it is all too easy to write code that
> leads to a security hole due to buffer overflows and other memory
> management errors.  Code written in a scripting language are largely
> immune to such problems (except of course for any such bugs in the
> interpreter itself, but the testing of the interpreter is shared across
> many projects and users).
>
> It would be insane to rewrite performance-critical C code in any
> scripting language, but there is a huge penumbra of code that is not
> performance critical and that mutates rapidly.  Such code is much easier
> to write and maintain in a sane scripting language if the portability
> issues can be mastered.

I think git developers are perfectly able to write such a code.

> The most important issues to consider when imagining a future with a
> hybrid of code in C and some scripting language "X" are:
>
> * Portability: is "X" available on all platforms targeted by git, in
>   usable and mutually-compatible versions?
>
> * Startup time: Is the time to start the "X" interpreter prohibitive?
>   (On my computer, "python -c pass", which starts the Python
>   interpreter and does nothing, takes about 24ms.)  This overhead would
>   be incurred by every command that is not pure C.

Agree.

> * Should the scripting language access the C functionality only by
>   calling pure-C executables or by dynamically or statically linking to
>   a binary module interface?  If the former, then the granularity of
>   interactions between "X" and C is necessarily coarse, and "X" cannot
>   be used to implement anything but the outermost layer of
>   functionality.  If the latter, then the way would be clear to
>   implement much more of git in "X" (and lua would also be worth
>   considering).

I think this is very far fetched at the moment. Proposals such as
libgit2 are moving things forward, but we are pretty far from a goal
like that.

> * Learning curve for developers: how difficult is it for a typical git
>   developer to become conversant with "X", considering both (1) how
>   likely is it that the typical git developer already knows "X" and
>   (2) how straightforward and predictable is the language "X"?
>   In this category I think that Python has a huge advantage over
>   Perl, though certainly opinions will differ and Ruby would also be
>   a contender.

Right, but I have the feeling that most git developers are perfectly
familiar with C already. In order to move to something else, and all
the necessary burden of learning, or becoming more familiar with X, a
compelling argument must be put forward, and I haven't seen such an
argument.

> Personally, I regret wasting my time programming pointer arithmetic in
> git modules that are not performance-critical (and correcting bugs by
> others in these areas).  And I'm tired of having an idea to improve a
> git feature only to find that it is implemented in shell, where not even
> arrays are available.  I would therefore welcome more friendliness
> towards a decent scripting language in the git project.

Me too, if ruby was one of them.

Cheers.

-- 
Felipe Contreras

Re: Python extension commands in git - request for policy change

[Felipe Contreras]

On Sun, Nov 25, 2012 at 10:54 AM, Eric S. Raymond <esr@thyrsus.com> wrote:
> Felipe Contreras <felipe.contreras@gmail.com>:
>> On Sun, Nov 25, 2012 at 6:18 AM, Eric S. Raymond <esr@thyrsus.com> wrote:
>> > Nguyen Thai Ngoc Duy <pclouds@gmail.com>:
>> >> These may apply to other languages as well. Where do we draw a line?
>> >
>> > I'm in favor of the general policy of avoiding scripting languages
>> > other than the top three most widely deployed.  At the moment that
>> > means shell, Python, Perl; on present trends, in a few years Perl
>> > (dropping in popularity) might be passed by Ruby on the way up.
>>
>> Top three according to whom?
>
> According to the LOC counts in git's codebase.

Not according to ohloh:

1) shell 33%
2) tcl 9%
3) perl 9.7%

4) python 1.8%

And this is a non-sequitur; you are proposing to change git policies
based on numbers that are a direct result of git's policies?

https://www.ohloh.net/p/git/analyses/latest/languages_summary

-- 
Felipe Contreras

Re: Python extension commands in git - request for policy change

[David Lang]

On Sun, 25 Nov 2012, Eric S. Raymond wrote:

> Michael Haggerty <mhagger@alum.mit.edu>:
>> There is, of course, the awkward issue of how/when to transition to
>> Python 3.x, which is *not* backwards compatible with Python 2.x.  I
>> expect that when the time comes there will be volunteers (myself
>> included) willing to help adapt Python scripts to the new version, but
>> the problem shouldn't be minimized.
>
> 2to3 actually does a pretty good job.  It doesn't reduce the
> transition cost to zero, but I find it does reduce that cost to an
> easily manageable level even on quite large codebases.
>
>> It would be insane to rewrite performance-critical C code in any
>> scripting language, but there is a huge penumbra of code that is not
>> performance critical and that mutates rapidly.
>
> Indeed.  In the git architecture there is a pretty clear dividing line -
> to a first approximation, plumbing should remain C but porcelain should
> probably not.  (Not that I am advocating forcing such a move - but it would
> be good to allow it to happen.)
>
> The 80-20 rule (80% of the execution time is spent in 20% of the code)
> helps us here.  The *other* 80% of the code can move to a scripting
> language with no significant performance loss.  To find out what needs
> to stay in C, run a profiler!

Remember that old code is tested code. The mere act of re-writing it from 
scratch is likely to introduce new bugs due to 'simplifications' by the person 
re-writing the code.

If a particular piece of code has a track record of being buggy, this may be 
overwelmed by the fresh start and new attention (plus whatever theoretical 
advantage any particular language provides), but unless it's suspect, re-writing 
it for the sole reason of changing the language is unlikely to be a win.

In addition, a good programmer working in a 'bad' language that they are very 
familiar with is going to write better code than that same programmer would 
write in a 'good' language that they are not familiar with.

I git, the programmers are very familiar with C and Bash, but far less familiar 
with either Perl or Python (although from what I see, far more familiar with 
Perl than Python)

If it's something going into contrib, where the core developers are not needing 
to maintain it, the language it's written in matters far less than if it's 
something that's going to be in the core. If it's in the core, it needs to be in 
a language that the core developers are comforatable with.

You may think that C and Bash are poor choices, but that is what the community 
is familar with.

You are far from the first person to say that git should be re-written (or at 
least large portions of it) in the language-of-the-day, and you won't be the 
last (even, or especially if it does get re-written in Python ;-)

David Lang

Re: Python extension commands in git - request for policy change

[Stefano Lattarini]

Hi David.  One minor but important correction ...

On 11/25/2012 12:51 PM, David Lang wrote:
>
> You may think that C and Bash are poor choices, but that is what the
> community is familar with.
>
Actually, it is C and POSIX shell -- not merely bash.  Indeed, the shell
code in Git is expected to work with the Solaris Korn shell, the BSD
/bin/sh, the dash shell (which is now the default /bin/sh on Debian and
Ubuntu), etc.

(Oh, and on the python vs. C vs. shell diatribe I'm mostly neutral,
mostly because I'm no Git developer, and I have no "cents to throw").

Regards,
  Stefano

Re: Re: Python extension commands in git - request for policy change

[Erik Faye-Lund]

On Sun, Nov 25, 2012 at 11:33 AM, Eric S. Raymond <esr@thyrsus.com> wrote:
> Pat Thoyts <patthoyts@gmail.com>:
>> Git for Windows simply ships everything we need to run git - so if a
>> desirable module requires a version of python, we will add that
>> version plus any required modules into the installer. We already have
>> a patch to provide python in the msysgit tree - it would just require
>> polishing up a little. I'm certain this is no problem for the other
>> windows port (cygwin) either.
>
> Thank you - I think this completely disposes of the "Windows is a blocker
> for scripting language X" argument, with the case X = Python in point.

As the one who wrote that patch; not at all. That patch is a horrible
mess, and it is not yet proven that the resulting python executable
works any more than a basic hello world.

-- 
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to msysgit@googlegroups.com
To unsubscribe from this group, send email to
msysgit+unsubscribe@googlegroups.com
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en

Re: Re: Python extension commands in git - request for policy change

[Johannes Schindelin]

Hi,

thank you Duy for thinking of Cc:ing the msysGit mailing list. We indeed
do not have a working Python in Git for Windows yet (mainly because I did
not review kusma's patch yet thanks to a non-fun-at-all side track).

On Sun, 25 Nov 2012, Eric S. Raymond wrote:

> Nguyen Thai Ngoc Duy <pclouds@gmail.com>:
> > These may apply to other languages as well. Where do we draw a line?
> 
> I'm in favor of the general policy of avoiding scripting languages
> other than the top three most widely deployed.

It is one thing to allow users to use the scripting languages of their
choice to do their work.

It is a different thing completely to allow the core of an important piece
of software like Git to consist of a hodge podge of languages. There are
so many problems already, both technical and social ones [*1*], that I would
really like to caution against letting even more languages creep into the
core. It is bad enough already.

Ciao,
Dscho

Footnote [*1*]: Technical problems include serious performance issues on
Windows when using shell/Perl scripting (see the many, many complaints
about git-svn just as an example), portability problems (I am thankful
that Junio seems to insist at least on POSIX compatibility of shell
scripts still even if there are very vocal forces trying to get lazy on
that front).

And do not underestimate the social problems with *requiring* contributors
to know yet another language well just because you let a core part be
written in that language. There is even a rule of thumb: increase the
number of languages used in your program == halve the number of potential
contributors. And if you think that this is theoretical: look at the mails
we got about Git GUI being written in Tcl/Tk (hardly a difficult language
to learn) and losing contributors over it.

-- 
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to msysgit@googlegroups.com
To unsubscribe from this group, send email to
msysgit+unsubscribe@googlegroups.com
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en

Re: Python extension commands in git - request for policy change

[Eric S. Raymond]

Felipe Contreras <felipe.contreras@gmail.com>:
> Seems sensible, but I don't know what "rejection" would actually mean.

Why is this mysterious?  We reject a patch when we don't choose to merge it.

> My "extensions" are on the way to the contrib area. Is the contrib
> area supposed to have different rules? I don't know.

I don't have a strong opinion about this.  I lean towards looser rules
for contrib because, among other things, it's a place for experiments
and we disclaim responsibility for maintaining it. But requiring 2.6
compatibility for Python scripts is not really onerous.

> Too late.

I'd be happy to help you out by auditing them for version dependencies.

> I don't see what this means in practical terms. People are going to
> write code in whatever language they want to write code in. How
> exactly are "we" going to "encourage" them not to do that is not
> entirely clear to me.

One way is by having clear guidelines for good practice that *include*
Python, and tell people exactly what the requirements are.

> Subcommands are also probably more efficient in c. And lets remember
> that most people use git through the *official* subcommands.

See my remarks on the 80-20 rule elsewhere in the thread.  Execessive
worship of "efficiency" is a great way to waste effort and pile up
hidden costs in maintainance problems.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Python extension commands in git - request for policy change

[Eric S. Raymond]

Felipe Contreras <felipe.contreras@gmail.com>:
> Of course, but there are experts in C and shell around, not so many
> python experts. So if somebody sneaks in a python program that makes
> use of features specific to python 2.7, I doubt anybody would notice.

I would.

> And if they did, I doubt that would be reason enough for rejection,
> supposing that porting to 2.6 would be difficult enough.

In cases like that, backporting is usually pretty easy.  Been there, done that.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Python extension commands in git - request for policy change

[Eric S. Raymond]

David Lang <david@lang.hm>:
> You may think that C and Bash are poor choices, but that is what the
> community is familar with.

I don't think C is a "poor" choice.  bash, on the other hand...so
many dependencies on tool quirks!

> You are far from the first person to say that git should be
> re-written (or at least large portions of it) in the
> language-of-the-day, and you won't be the last (even, or especially
> if it does get re-written in Python ;-)

I think you're overinterpreting.  Trying for One Big Rewrite in language
X is almost never a good idea and I don't advocate it.  Encouraging people
to migrate pieces as they feel motivated and resdy is a different matter.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Python extension commands in git - request for policy change

[Eric S. Raymond]

Felipe Contreras <felipe.contreras@gmail.com>:
> Not according to ohloh:
> 
> 1) shell 33%
> 2) tcl 9%
> 3) perl 9.7%
> 
> 4) python 1.8%

Look in the Makefile - all that tcl code is buried in gitk.  We're
very, very lucky the author did such a good job, because it's a
potentially serious headache; who can maintain it?
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Python extension commands in git - request for policy change

[Felipe Contreras]

On Sun, Nov 25, 2012 at 6:50 PM, Eric S. Raymond <esr@thyrsus.com> wrote:
> Felipe Contreras <felipe.contreras@gmail.com>:
>> Not according to ohloh:
>>
>> 1) shell 33%
>> 2) tcl 9%
>> 3) perl 9.7%
>>
>> 4) python 1.8%
>
> Look in the Makefile - all that tcl code is buried in gitk.  We're
> very, very lucky the author did such a good job, because it's a
> potentially serious headache; who can maintain it?

And gitk is an integral part of git. But if you have different
numbers, what are they?

-- 
Felipe Contreras

Re: Python extension commands in git - request for policy change

[Felipe Contreras]

On Sun, Nov 25, 2012 at 6:36 PM, Eric S. Raymond <esr@thyrsus.com> wrote:
> Felipe Contreras <felipe.contreras@gmail.com>:
>> Of course, but there are experts in C and shell around, not so many
>> python experts. So if somebody sneaks in a python program that makes
>> use of features specific to python 2.7, I doubt anybody would notice.
>
> I would.

And are you going to be around to spot them? It seems my patches for
git-remote-hg slipped by your watch, because it seems they use stuff
specific to python 2.7.

>> And if they did, I doubt that would be reason enough for rejection,
>> supposing that porting to 2.6 would be difficult enough.
>
> In cases like that, backporting is usually pretty easy.  Been there, done that.

Exactly. Why would you reject something you can fix easily?

-- 
Felipe Contreras

Re: Python extension commands in git - request for policy change

[Krzysztof Mazur]

On Sat, Nov 24, 2012 at 09:44:51PM -0500, Eric S. Raymond wrote:
> 
> We're behind the best-practices curve here.  The major Linux
> distributions, which have to deal with almost the same set of
> tradeoffs we do, went to Python for pretty much all glue and
> administration scripts outside /etc a decade ago, and the decision has
> served them well.
> 
> That, among other things, means up-to-date versions of Python are
> ubiquitous unless we're looking at Windows - in which case Perl and
> shell actually become much bigger portability problems.  Mac OS X 
> has kept up to date, too; Lion shipped 2.7.1 and that was a major
> release back at this point.
> 

What about embedded systems? git is also useful there. C and shell is
everywhere, python is not. Adding additional dependency if it's not
really needed it's not a good idea.

Also not everyone uses up-to-date systems and sometimes you just
care about some critical parts and do not touch everything else and
there is probably quote large number of systems with python < 2.6.
And even when you keep your system up-to-date, there are some GNU/Linux
distros that are still supported, but does not provide recent python - for
instance PLD Ac, which I still use on some systems and will use
until the hardware dies, provides only python 2.4.6 (by the way,
important packages like git are of course quite recent there - 1.7.11.1).

Krzysiek

Re: Python extension commands in git - request for policy change

[Felipe Contreras]

On Sun, Nov 25, 2012 at 6:32 PM, Eric S. Raymond <esr@thyrsus.com> wrote:
> Felipe Contreras <felipe.contreras@gmail.com>:
>> Seems sensible, but I don't know what "rejection" would actually mean.
>
> Why is this mysterious?  We reject a patch when we don't choose to merge it.

Why would you reject it? If, according to you, it's very simple to fix
the portability, then presumably it would take you less time to fix
it, than to reject it (and everything that implies).

>> Too late.
>
> I'd be happy to help you out by auditing them for version dependencies.

Be my guest:
http://git.kernel.org/?p=git/git.git;a=tree;f=contrib/remote-helpers;h=adfdcc164e634c74024c8f69bb0cdb9f3b4a9f18;hb=7b4a70c62f3a83fbd8b44bf712141754a5f64205

Some patches might be missing, so:
https://github.com/felipec/git/tree/fc/remote/hg

>> I don't see what this means in practical terms. People are going to
>> write code in whatever language they want to write code in. How
>> exactly are "we" going to "encourage" them not to do that is not
>> entirely clear to me.
>
> One way is by having clear guidelines for good practice that *include*
> Python, and tell people exactly what the requirements are.

The key word being guideline, which is different from a strict rule.

>> Subcommands are also probably more efficient in c. And lets remember
>> that most people use git through the *official* subcommands.
>
> See my remarks on the 80-20 rule elsewhere in the thread.  Execessive
> worship of "efficiency" is a great way to waste effort and pile up
> hidden costs in maintainance problems.

According to the results of the last survey, our users do care about
performance, so I don't think there's anything excessive about it. Are
there any hidden costs in maintenance problems? I don't think so.

The people that like to improve the performance of git, would keep
doing so, and the people that want to use fancy scripts to do fancy
stuff, will keep doing so. It just happens that the former have
actually managed to do it, and go all the way into the mainline.

It would be great if we had a finished libgit2 with all the essential
stuff, and good bindings for python (and other languages), and it
would be great if python really was this touted language, that is easy
to read, and would make things more maintainable. Unfortunately,
that's not the case.

I could write an endless list of what things in the python language
don't make any sense, and how in ruby, for example, they do.
Fortunately, I don't have to.

Git does have problems, but they have nothing to do with maintenance,
or C; they have to do with the user interface, and the documentation
(again, according to our users (and me)). So, I don't see why worry
about moving code from C to python when barely any code in git is
python, specially if it doesn't fix any real issue.

Cheers.

-- 
Felipe Contreras

Re: Python extension commands in git - request for policy change

[Eric S. Raymond]

Felipe Contreras <felipe.contreras@gmail.com>:
> And gitk is an integral part of git. But if you have different
> numbers, what are they?

I looked at the Makefile.  I saw that there are shell variables that collect
C commands, shell command, Perl commands, and Python commands.  There are no
collections of other commands.  That makes them the top languages in the
universe we are concerned about

Please don't waste further time on quibbling.  We all know that gitk is
an uncomfortable special case and that the project would be far better
off, maintainability-wise, if it were successfully ported to one if these
other languages.  Trying to catch me out by triumphantly pointing at gitk 
is...juvenile.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

-- 
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to msysgit@googlegroups.com
To unsubscribe from this group, send email to
msysgit+unsubscribe@googlegroups.com
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en

Re: Python extension commands in git - request for policy change

[Eric S. Raymond]

Felipe Contreras <felipe.contreras@gmail.com>:
> And are you going to be around to spot them? It seems my patches for
> git-remote-hg slipped by your watch, because it seems they use stuff
> specific to python 2.7.

The dev group hasn't decided (in whatever way it decides these
things) to require 2.6 yet.  When and if it does, I will volunteer my
services as a Python expert to audit the in-tree Python code for 2.6
conformance and assist the developers in backporting if required.
I will also make myself available to audit future submissions.  

I think you know who I am. Junio and the other senior devs certainly
know where to find me. I've been making promises like this, and
*keeping* them, for decades.  Please stop wasting our time with
petulant display.

> Exactly. Why would you reject something you can fix easily?

I wouldn't.  The point of a policy like this is not to kick incoming
submissions over the horizon as though that were some sort of
accomplishment, it's to let submitters know what is required of
them so they can code up to a standard that supports maintainability.
It would be no different than any of our other portability requirements.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Python extension commands in git - request for policy change

[Eric S. Raymond]

Felipe Contreras <felipe.contreras@gmail.com>:
> > I'd be happy to help you out by auditing them for version dependencies.
> 
> Be my guest:
> http://git.kernel.org/?p=git/git.git;a=tree;f=contrib/remote-helpers;h=adfdcc164e634c74024c8f69bb0cdb9f3b4a9f18;hb=7b4a70c62f3a83fbd8b44bf712141754a5f64205
> 
> Some patches might be missing, so:
> https://github.com/felipec/git/tree/fc/remote/hg

OK, here's what I look for:  use of argparse, use of unittest, use
of Collections.counters, use or ordered dictionaries, use of set literals,
use of multiple context managers in one "with", use of memoryview, use of
the comma format specifier.  I'm not worried about the changes in repr()
for floating point; I'd be astonished if they mattered in code like this.
Likewise for PyCapsule and importlib.

I don't see obvious problems in that code.  Looks pretty vanilla, actually;
the latest version-related blocker I can see is the import of json,
which would have been a problem before 2.5.

You wrote the code.  Do you *know* of 2.7-specific constructions in
there that I've missed?  If you do, and think of this as a way to
catch me in a mistake and dance triumphantly, you lose - our goal
should be to cooperate to improve the auditing process, not score
silly points.

> > One way is by having clear guidelines for good practice that *include*
> > Python, and tell people exactly what the requirements are.
> 
> The key word being guideline, which is different from a strict rule.

Agreed. It's a matter for the dev group to decide when we need rules
and when we need guidelines.  I think we need a rule about Python version
conformance that protects older systems, but other things can be guidelines.

> According to the results of the last survey, our users do care about
> performance, so I don't think there's anything excessive about it. Are
> there any hidden costs in maintenance problems? I don't think so.

Then you're either pretending or very naive. Three decades of
experience as a C programmer tells me that C code at any volume is a
*serious* maintainance problem relative to almost any language with
GC.  Prudent architects confine it is much as possible.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Python extension commands in git - request for policy change

[Eric S. Raymond]

Krzysztof Mazur <krzysiek@podlesie.net>:
> What about embedded systems? git is also useful there. C and shell is
> everywhere, python is not.

Supposing this is true (and I question it with regard to shell) if you
tell me how you live without gitk and the Perl pieces I'll play that
right back at you as your answer.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Python extension commands in git - request for policy change

[Sitaram Chamarty]

On Mon, Nov 26, 2012 at 4:17 AM, Eric S. Raymond <esr@thyrsus.com> wrote:
> Krzysztof Mazur <krzysiek@podlesie.net>:
>> What about embedded systems? git is also useful there. C and shell is
>> everywhere, python is not.
>
> Supposing this is true (and I question it with regard to shell) if you
> tell me how you live without gitk and the Perl pieces I'll play that
> right back at you as your answer.

gitk is unlikely to be used on an embedded system, the perl pieces more so.

I have never understood why people complain about readability in perl.
 Just because it uses the ascii charset a bit more?  You expect a
mathematician or indeed any scientist to use special symbols to mean
special things, why not programmers?

Perhaps people should be forced to use COBOL for a few years (like I
did, a long while ago) to appreciate brevity.

Re: Python extension commands in git - request for policy change

[Krzysztof Mazur]

On Mon, Nov 26, 2012 at 10:40:00AM +0530, Sitaram Chamarty wrote:
> On Mon, Nov 26, 2012 at 4:17 AM, Eric S. Raymond <esr@thyrsus.com> wrote:
> > Krzysztof Mazur <krzysiek@podlesie.net>:
> >> What about embedded systems? git is also useful there. C and shell is
> >> everywhere, python is not.
> >
> > Supposing this is true (and I question it with regard to shell) if you
> > tell me how you live without gitk and the Perl pieces I'll play that
> > right back at you as your answer.
> 
> gitk is unlikely to be used on an embedded system, the perl pieces more so.

Currently even perl is used only for few very high level commands that
are not really needed there. I think that python is ok for pieces
that use perl now, but I think that it shouldn't be used for
basic porcelain commands. I also don't think that we should prefer
python over other languages and especially I don't think that some
existing code should be rewritten to python.

Even if python is really better, I think that the natural migration is
much better.

> 
> I have never understood why people complain about readability in perl.
>  Just because it uses the ascii charset a bit more?  You expect a
> mathematician or indeed any scientist to use special symbols to mean
> special things, why not programmers?
> 

Because some perl programmers really create write-only code, but the
maintainer can just reject that code. It's not the language that
create non-readable code, but the programmer. I think that the perl
code in git is readable, at least is parts I've seen.

Krzysiek

Re: Python extension commands in git - request for policy change

[Andreas Ericsson]

On 11/25/2012 11:44 PM, Eric S. Raymond wrote:
> Felipe Contreras <felipe.contreras@gmail.com>:
>> According to the results of the last survey, our users do care about
>> performance, so I don't think there's anything excessive about it. Are
>> there any hidden costs in maintenance problems? I don't think so.
> 
> Then you're either pretending or very naive. Three decades of
> experience as a C programmer tells me that C code at any volume is a
> *serious* maintainance problem relative to almost any language with
> GC.  Prudent architects confine it is much as possible.
> 

Prudent architects also avoid rewrites as much as possible, since it's
inevitable that bugs will be introduced that have been fixed in the
"official" version.

Personally, I think if you'd left your suggestion on "It would be great
to have guidelines for python scripts. I propose 2.6 as the minimum
required python verison" and left it at that, there would have been
very little disagreement. The suggestion that things should be rewritten
in python for some spurious long-term savings seems mostly designed to
refuel everyone's favourite flamethrower, and you know as well as I do
that it just won't happen unless there's at least a chance of some
substantial technical benefits from doing so.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Considering the successes of the wars on alcohol, poverty, drugs and
terror, I think we should give some serious thought to declaring war
on peace.

Re: Python extension commands in git - request for policy change

[Felipe Contreras]

On Sun, Nov 25, 2012 at 10:56 PM, Eric S. Raymond <esr@thyrsus.com> wrote:
> Felipe Contreras <felipe.contreras@gmail.com>:
>> And gitk is an integral part of git. But if you have different
>> numbers, what are they?
>
> I looked at the Makefile.  I saw that there are shell variables that collect
> C commands, shell command, Perl commands, and Python commands.  There are no
> collections of other commands.

I suppose you are talking about BUILT_INS, and SCRIPT_FOO, but tcl
scripts don't need no SCRIPT_FOO stuff, because they don't need to be
regenerated, in fact, I don't think the shell scripts need to be
regenerated, but that's not important, what is important is this:

all::
ifndef NO_TCLTK
	$(QUIET_SUBDIR0)git-gui $(QUIET_SUBDIR1) gitexecdir='$(gitexec_instdir_SQ)' all
	$(QUIET_SUBDIR0)gitk-git $(QUIET_SUBDIR1) all
endif

Why are you ignoring that?

> That makes them the top languages in the universe we are concerned about

According to whom?

I find it very curious how you are arguing for a change in the status
quo to move more towards python, and the basis you are using for
choosing python, and not ruby, or other scripting language is
precisely the status quo. However, the only scripts using python are
these:

SCRIPT_PYTHON += git-remote-testgit.py
SCRIPT_PYTHON += git-p4.py

I already re-wrote git-remote-testgit in bash, and it's dubious
whether or not git-remote-testpy (the new name for this old test) will
fulfill any service. Than means 43% of the current python code is
gone.

And what happens if I rewrite git-p4 in ruby? Would you then argue
that ruby is the way to go because 1% of the *current* code-base uses
it?

Interestingly, according to Wikipedia git is written in: C, Bourne
Shell, Tcl, Perl. That seems to be the case.

> Please don't waste further time on quibbling.  We all know that gitk is
> an uncomfortable special case and that the project would be far better
> off, maintainability-wise, if it were successfully ported to one if these
> other languages.

As I said, gitk is integral to the git experience. Of course you are
free to disagree, but according to the last user survey 57% of the
responders used some kind of graphical tool (e.g. gitk, tig). How many
use gitk, and how many use something else, we don't really know, but
what we know is that gitk is distributed *by default*.

Nobody is arguing that gitk should not be distributed by default, just
like nobody is arguing that git-p4 shouldn't, but we *know* very few
people use git-p4 (1% according to the survey), and we can reasonably
assume that many more use gitk.

You cannot have your cake and eat it at the same time. If you use the
amount of code as a measure, then you have to agree that Tcl/Tk is a
way bigger language than python in the mainline git world. If not,
then by all means, show us the numbers. But you can't say "the
important languages are A, B, and D, C doesn't count because I don't
like it, and E doesn't count either because we should draw the line at
three", that seems awfully convenient to push your agenda.

And I don't agree that the project would be better off with something
else, if it was, somebody would have proposed an alternative by now,
but there aren't any. I have tried gitg, and giggle, and I have even
contributed to them, but they are just not as good and useful as plain
old gitk, I always keep coming back.

gitk:
 * is blazing fast to start
 * doesn't have a lot of dependencies: just tcl/tk
 * works on Windows without a major fuzz
 * is actively maintained
 * shows a proper graph (unlike gitg or giggle)

Now, show me an alternative that fulfills all these points. And I'm
pretty sure you won't find one, because if you did, it would have been
already proposed for mainline git... there isn't any. And if you did,
we would start with oh, but it's GTK+, or it's Qt, and how do you make
it work on Windows. No, gitk is just fine, and works great.

Tcl/Tk might not be your cup of tea, and indeed it's rather unmodern,
but that only tells you how an awful job the modern toolkits have done
with regards to portability and flexibility.

You were arguing for portability, well, Tcl/Tk works on all platforms,
here, have a look, there's no other tool that fulfills this:

http://www.mediawiki.org/wiki/Git/Graphical_User_Interfaces

> Trying to catch me out by triumphantly pointing at gitk is...juvenile.

Isn't that what you are doing by triumphantly pointing at git-p4?

Sorry, if you want to cut the line for the languages that git should
use right now at three, then python is out. Maybe in a couple of
years. Maybe. But I doubt it.

Cheers.

-- 
Felipe Contreras

Re: Python extension commands in git - request for policy change

[Felipe Contreras]

On Sun, Nov 25, 2012 at 11:11 PM, Eric S. Raymond <esr@thyrsus.com> wrote:
> Felipe Contreras <felipe.contreras@gmail.com>:
>> And are you going to be around to spot them? It seems my patches for
>> git-remote-hg slipped by your watch, because it seems they use stuff
>> specific to python 2.7.
>
> The dev group hasn't decided (in whatever way it decides these
> things) to require 2.6 yet.  When and if it does, I will volunteer my
> services as a Python expert to audit the in-tree Python code for 2.6
> conformance and assist the developers in backporting if required.
> I will also make myself available to audit future submissions.

What dev group?

> I think you know who I am. Junio and the other senior devs certainly
> know where to find me. I've been making promises like this, and
> *keeping* them, for decades.  Please stop wasting our time with
> petulant display.

All right, you don't wand feedback, fine.

If you need me I'll be rewriting python code to ruby.

Cheers.

-- 
Felipe Contreras

Re: Python extension commands in git - request for policy change

[David Aguilar]

On Mon, Nov 26, 2012 at 5:11 AM, Felipe Contreras
<felipe.contreras@gmail.com> wrote:
> And I don't agree that the project would be better off with something
> else, if it was, somebody would have proposed an alternative by now,
> but there aren't any. I have tried gitg, and giggle, and I have even
> contributed to them, but they are just not as good and useful as plain
> old gitk, I always keep coming back.
>
> gitk:
>  * is blazing fast to start
>  * doesn't have a lot of dependencies: just tcl/tk
>  * works on Windows without a major fuzz
>  * is actively maintained
>  * shows a proper graph (unlike gitg or giggle)
>
> Now, show me an alternative that fulfills all these points. And I'm
> pretty sure you won't find one, because if you did, it would have been
> already proposed for mainline git... there isn't any. And if you did,
> we would start with oh, but it's GTK+, or it's Qt, and how do you make
> it work on Windows. No, gitk is just fine, and works great.
>
> Tcl/Tk might not be your cup of tea, and indeed it's rather unmodern,
> but that only tells you how an awful job the modern toolkits have done
> with regards to portability and flexibility.
>
> You were arguing for portability, well, Tcl/Tk works on all platforms,
> here, have a look, there's no other tool that fulfills this:
>
> http://www.mediawiki.org/wiki/Git/Graphical_User_Interfaces

*cough* git-cola *cough*

it runs everywhere.  Yes, windows too. It's written in python.
It's been actively maintained since 2007.

It's "modern" and has features that don't exist anywhere else.

It even has tests.  It even comes with a building full of willing
guinea-pigs^Wtesters that let me know right away when
anything goes wrong.

It uses Qt but that's really the whole point of Qt -> cross-platform.
(not sure how that wiki page ended up saying Gnome/GTK?)

The DAG aka git-dag (in its master branch, about to be released)
is nicer looking then gitk IMO.  gitk still has some features
that are better too--there's no silver bullet, but the delta
is pretty small.

The only point this doesn't fulfill is dependency-free-ness.
With that requirement the answer can *only* be tcl/tk.
So saying, "go look for one you won't find it" is really
a tautology.  It's not even an entertaining one.

http://xkcd.com/703/

When the requirement is, "what is the best user experience
possible", then you use a mature GUI library.  These are different
requirements and probably different use cases.  For the gitk use
case, gitk is the perfect tool.

Anyways, just sayin', you make it sound like this stuff doesn't
exist, but it does.  I've never proposed it for mainline
git because I'm very aware of the dependency requirements.

But, if git "recommended" it I would very much appreciate the
extra eyes and contributions.  Being in mainline git could
possibly help with that.  A submodule under contrib/
would be an interesting experiment.

In any case, I think documenting the python standards
(even if within contrib/ only) is a good thing.

We'd be increasing the overall portability by documenting
what we support and sticking to it, just
like what is done for shell scripts and perl versions.
Eric is helping make that happen, let's not  throw
out the baby with the bathwater.  FWIW, I would also make
my python expertise available.

This thread has gotten into meta-meta territory --
it's discussing code that has not yet even been written,
and going off on all sorts of tangents.

Let's stay on target.  I think bringing up python
as an extension language would help in a these key areas:

- There are a few python modules floating around that
  are similar to the ruby grit library.  Having an official
  python module would be good.

- Commands on the edges of the git experience (GUIs,
  import/export/etc) can benefit from python.  git-p4
  is one example.  git-weave is another.

Are there any arguments against it for these use cases?


BTW, Felipe, if you're going to be rewriting python code to ruby,
please, pretty please rewrite that darn GUI ;-)

You can call it "git-new-cola: project kansas"

http://en.wikipedia.org/wiki/New_Coke

I expect a patch by the morning ;-p

I kid, of course, but if you do pull it off I WILL buy you a soda-pop!
-- 
David

Re: Python extension commands in git - request for policy change

[Felipe Contreras]

On Tue, Nov 27, 2012 at 8:54 AM, David Aguilar <davvid@gmail.com> wrote:
> On Mon, Nov 26, 2012 at 5:11 AM, Felipe Contreras
> <felipe.contreras@gmail.com> wrote:

>> http://www.mediawiki.org/wiki/Git/Graphical_User_Interfaces
>
> *cough* git-cola *cough*
>
> it runs everywhere.  Yes, windows too. It's written in python.
> It's been actively maintained since 2007.

% sudo pacman -S git-cola
error: target not found: git-cola

http://aur.archlinux.org/packages/gi/git-cola/git-cola.tar.gz
% makepkg

==> Missing Dependencies:
  -> python2-pyqt>=4.3
==> Checking buildtime dependencies...
==> Missing Dependencies:
  -> asciidoc
  -> docbook-xsl
  -> rsync
  -> xmlto
  -> python2-sphinx>=1.1.3

Sorry, no.

I'm not sure if you have been following, but msysgit doesn't seem to
have good support for python, let alone Qt. In my view the fact that
it uses python is not a good thing.

> It's "modern" and has features that don't exist anywhere else.
>
> It even has tests.  It even comes with a building full of willing
> guinea-pigs^Wtesters that let me know right away when
> anything goes wrong.
>
> It uses Qt but that's really the whole point of Qt -> cross-platform.
> (not sure how that wiki page ended up saying Gnome/GTK?)

Yes, Qt is cross-platform *in theory*, but have you used any Qt
application in Windows? I haven't.

> The DAG aka git-dag (in its master branch, about to be released)
> is nicer looking then gitk IMO.  gitk still has some features
> that are better too--there's no silver bullet, but the delta
> is pretty small.

If you mean this one:
http://1.1.1.5/bmi/git-cola.github.com/images/dag.png

Then I wholeheartedly disagree.

> The only point this doesn't fulfill is dependency-free-ness.
> With that requirement the answer can *only* be tcl/tk.
> So saying, "go look for one you won't find it" is really
> a tautology.  It's not even an entertaining one.

That's the whole point; there is nothing else. If there was something
else, there would be something else. But there isn't.

> When the requirement is, "what is the best user experience
> possible", then you use a mature GUI library.  These are different
> requirements and probably different use cases.

But those are not the requirements.

> Anyways, just sayin', you make it sound like this stuff doesn't
> exist, but it does.  I've never proposed it for mainline
> git because I'm very aware of the dependency requirements.

A lot of stuff exists. And people use a lot of those. But they don't
fulfill the requirements that I think gitk does perfectly.

> But, if git "recommended" it I would very much appreciate the
> extra eyes and contributions.  Being in mainline git could
> possibly help with that.  A submodule under contrib/
> would be an interesting experiment.

It might be, if somebody actually tried to submit the code. But I
honestly doubt so.

> In any case, I think documenting the python standards
> (even if within contrib/ only) is a good thing.
>
> We'd be increasing the overall portability by documenting
> what we support and sticking to it, just
> like what is done for shell scripts and perl versions.
> Eric is helping make that happen, let's not  throw
> out the baby with the bathwater.  FWIW, I would also make
> my python expertise available.

Nobody has argued that there shouldn't be guidelines for python code.
What I have objected is to 'strict rules'.

> This thread has gotten into meta-meta territory --
> it's discussing code that has not yet even been written,
> and going off on all sorts of tangents.

That is the point; why should we change the policy for code that
hasn't been written yet? That's not how things evolve in git as far as
I have seen.

> BTW, Felipe, if you're going to be rewriting python code to ruby,
> please, pretty please rewrite that darn GUI ;-)

I would need to write a widget toolkit first =/ I think I'll pass. gitk is fine.

Cheers.

-- 
Felipe Contreras

-- 
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to msysgit@googlegroups.com
To unsubscribe from this group, send email to
msysgit+unsubscribe@googlegroups.com
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en

Re: Python extension commands in git - request for policy change

[Sitaram Chamarty]

On Tue, Nov 27, 2012 at 1:24 PM, David Aguilar <davvid@gmail.com> wrote:

> *cough* git-cola *cough*
>
> it runs everywhere.  Yes, windows too. It's written in python.
> It's been actively maintained since 2007.
>
> It's "modern" and has features that don't exist anywhere else.
>
> It even has tests.  It even comes with a building full of willing
> guinea-pigs^Wtesters that let me know right away when
> anything goes wrong.
>
> It uses Qt but that's really the whole point of Qt -> cross-platform.
> (not sure how that wiki page ended up saying Gnome/GTK?)
>
> The DAG aka git-dag (in its master branch, about to be released)
> is nicer looking then gitk IMO.  gitk still has some features
> that are better too--there's no silver bullet, but the delta
> is pretty small.

Gitk does a lot of things that people don't realise, since they're not
really documented and you have to scrounge around on the UI.  The
thing is, it's just about the most awesome tool for code archeology I
have seen.

I realise (from looking at the doc page) that git-cola helps you do
all sorts of things, but those are all things I am happier doing at
the command line.

Gitk does precisely those things which *require* a GUI, where the
amount of information presented overwhelms a text interface.  The
display is concisely designed to give you the maximum information at a
minimum space use.  For example, a little black square when a commit
has a note attached.  Even hovering over the arrow-heads, on complex
trees where the line gets broken, does something meaningful.

if I had to pin it down, the feature I use most often is "Show origin
of this line".  Other features I use often are
  - review a commit file by file (f and b keys, also spacebar and 'd')
  - search by SHA1 (4 digits appear to be enough, regardless of how
big your repo is),
  - search for commits changing path/dir (while still showing all the
commits; i.e., this is not 'git-dag -- README.txt' but within gitk you
search up and down for commits touching README.txt
  - and navigating the commit tree looking for stuff

http://sitaramc.github.com/1-basic-usage/gitk.html is my attempt to
document some of the stuff I have found and use.

One final point: the DAG on the right wastes enormous amounts of
space.  Purely subjectively, it is almost jarring on the senses.  (If
you reduce it, it becomes unreadable).

With all due respect, git-cola/dag isn't anywhere near what gitk does,
at least for people who are not afraid of the command line and only
need the GUI to visualise a truly complex tree.

-- 
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to msysgit@googlegroups.com
To unsubscribe from this group, send email to
msysgit+unsubscribe@googlegroups.com
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en

Re: Python extension commands in git - request for policy change

[David Aguilar]

On Tue, Nov 27, 2012 at 1:17 AM, Sitaram Chamarty <sitaramc@gmail.com> wrote:
> On Tue, Nov 27, 2012 at 1:24 PM, David Aguilar <davvid@gmail.com> wrote:
>
>> *cough* git-cola *cough*
>>
>> it runs everywhere.  Yes, windows too. It's written in python.
>> It's been actively maintained since 2007.
>>
>> It's "modern" and has features that don't exist anywhere else.
>>
>> It even has tests.  It even comes with a building full of willing
>> guinea-pigs^Wtesters that let me know right away when
>> anything goes wrong.
>>
>> It uses Qt but that's really the whole point of Qt -> cross-platform.
>> (not sure how that wiki page ended up saying Gnome/GTK?)
>>
>> The DAG aka git-dag (in its master branch, about to be released)
>> is nicer looking then gitk IMO.  gitk still has some features
>> that are better too--there's no silver bullet, but the delta
>> is pretty small.
>
> Gitk does a lot of things that people don't realise, since they're not
> really documented and you have to scrounge around on the UI.  The
> thing is, it's just about the most awesome tool for code archeology I
> have seen.
>
> I realise (from looking at the doc page) that git-cola helps you do
> all sorts of things, but those are all things I am happier doing at
> the command line.

Ditto.  There's actually a few small things I use it for,
mainly for teasing apart commits.  These days you can use git-gui
for that, but in the old days it was the only way to interactively
select individual lines and stage/unstage/revert them, etc.
I don't think we can line-by-line revert in git-gui yet, though.

Some other small things that I use: ctrl-g, type something
for grep, hit enter twice and I'm in my editor on that
(or any other selected) line.  'spacebar' does xdg-open,
and 'enter' launches the editor in the status widget;
small things.  I, too, do most stuff on the command line.

The grep thing is a good example.  You have tons of output,
you see the one line that you care about, and you want to jump
there.  Clicking on that line and hitting enter is the minimal
effort to do that.  You don't have to click because we also
have keyboard navigation.  I have a feeling that there's probably
something I'm missing, though.. another way of working (emacs?)
that would render all of this custom GUI stuff pointless.

What I learned about users:

The commit editor is the #1 thing that got my coworkers finally
writing better commit messages. It forces the subject/description
separation and shows yellow, red when the subject gets too long.
It also auto-wraps.  IMO it makes sense for git-gui to do
the same these days.

> Gitk does precisely those things which *require* a GUI, where the
> amount of information presented overwhelms a text interface.  The
> display is concisely designed to give you the maximum information at a
> minimum space use.  For example, a little black square when a commit
> has a note attached.  Even hovering over the arrow-heads, on complex
> trees where the line gets broken, does something meaningful.
>
> if I had to pin it down, the feature I use most often is "Show origin
> of this line".  Other features I use often are
>   - review a commit file by file (f and b keys, also spacebar and 'd')
>   - search by SHA1 (4 digits appear to be enough, regardless of how
> big your repo is),
>   - search for commits changing path/dir (while still showing all the
> commits; i.e., this is not 'git-dag -- README.txt' but within gitk you
> search up and down for commits touching README.txt
>   - and navigating the commit tree looking for stuff
>
> http://sitaramc.github.com/1-basic-usage/gitk.html is my attempt to
> document some of the stuff I have found and use.

Wow, this is awesome.

> One final point: the DAG on the right wastes enormous amounts of
> space.  Purely subjectively, it is almost jarring on the senses.  (If
> you reduce it, it becomes unreadable).
>
> With all due respect, git-cola/dag isn't anywhere near what gitk does,
> at least for people who are not afraid of the command line and only
> need the GUI to visualise a truly complex tree.

This is really great feedback.
cc:ing Guillaume since he had similar ideas.

thx,
-- 
David

-- 
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to msysgit@googlegroups.com
To unsubscribe from this group, send email to
msysgit+unsubscribe@googlegroups.com
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en

Re: Python extension commands in git - request for policy change

[Magnus Bäck]

On Sunday, November 25, 2012 at 06:40 EST,
     Felipe Contreras <felipe.contreras@gmail.com> wrote:

> On Sun, Nov 25, 2012 at 11:44 AM, Michael Haggerty
> <mhagger@alum.mit.edu> wrote:

[...]

> > On the contrary, there is *constant* traffic on the mailing list
> > about incompatibilities between different shell implementations (sh,
> > dash, bash, etc), not to mention those in other utilities (sed,
> > grep, etc) that one is forced to work with in shell scripts.
> > Compatibility is a *huge* pain when developing shell code for git.
> > The fact that users typically don't encounter such problems is due
> > to the hard work of POSIX lawyers on the mailing list correcting the
> > compatibility errors of mortal programmers.
>
> *Theoretical* incompatibilities on probably obscure systems. *I* have
> never seen such compatibility issues *in practice*.

While "constant traffic" probably overstates the issue, these are not
theoretical problems. I recall at least three cases in the last year
or so where Git has seen breakage with Solaris or Mac OS X because
of sed or tr incompatibilities, and I don't even read this list that
thoroughly.

[...]

-- 
Magnus Bäck
baeck@google.com

Re: Re: Python extension commands in git - request for policy change

[Johannes Schindelin]

Hi David,

On Mon, 26 Nov 2012, David Aguilar wrote:

> *cough* git-cola *cough*

If you had a couple of free cycles to help us get Python/Qt compiled in
msysGit, I will be happy to make a Git for Windows package including
git-cola.

Ciao,
Dscho

-- 
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to msysgit@googlegroups.com
To unsubscribe from this group, send email to
msysgit+unsubscribe@googlegroups.com
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en

Re: Python extension commands in git - request for policy change

[Eric S. Raymond]

Magnus Bäck <baeck@google.com>:
> While "constant traffic" probably overstates the issue, these are not
> theoretical problems. I recall at least three cases in the last year
> or so where Git has seen breakage with Solaris or Mac OS X because
> of sed or tr incompatibilities, and I don't even read this list that
> thoroughly.

This is exactly the sort of of pain experience would lead me to
expect.  

OK, this is where I assume the full Old Fart position (30-year
old-school Unix guy, author of "The Art of Unix Programming", can
remember the years before Perl and still has sh idioms in his
fingertips) and say "Get with the 21st century, people! Or at least
1990..."

As a general scripting language shell sucks *really badly* compared to
anything new-school. Performance, portability, you name it, it's a
mess.  It's not so much the shell interpreters itself that are the
portabilty problem, but (as Magnus implicitly points out) all those
userland dependencies on sed and tr and awk and even variants of
expr(!) that get dragged in the second you try to get any actual work
done.

Can we cease behaving like we're still pounding keys on 110-baud
teletypes now?  Some old-school Unix habits have persisted long past
the point that they're even remotely sane.  Shell programming at any
volume above a few lines of throwaway code is one of them - it's
*nuts* and we should *stop doing it*.

(Yes, I too still make this mistake occasionally out of ancient reflex.
But I know I shouldn't, and I always end up regretting it.)
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Python extension commands in git - request for policy change

[Sitaram Chamarty]

On Wed, Nov 28, 2012 at 12:05 AM, Eric S. Raymond <esr@thyrsus.com> wrote:
> Magnus Bäck <baeck@google.com>:
>> While "constant traffic" probably overstates the issue, these are not
>> theoretical problems. I recall at least three cases in the last year
>> or so where Git has seen breakage with Solaris or Mac OS X because
>> of sed or tr incompatibilities, and I don't even read this list that
>> thoroughly.
>
> This is exactly the sort of of pain experience would lead me to
> expect.
>
> OK, this is where I assume the full Old Fart position (30-year
> old-school Unix guy, author of "The Art of Unix Programming", can
> remember the years before Perl and still has sh idioms in his
> fingertips) and say "Get with the 21st century, people! Or at least
> 1990..."
>
> As a general scripting language shell sucks *really badly* compared to
> anything new-school. Performance, portability, you name it, it's a
> mess.  It's not so much the shell interpreters itself that are the
> portabilty problem, but (as Magnus implicitly points out) all those
> userland dependencies on sed and tr and awk and even variants of
> expr(!) that get dragged in the second you try to get any actual work
> done.

Not always.  There are several situations where a shell script that
makes good use of grep, cut, etc., is definitely much cleaner and more
elegant than anything you can do in a "propah" programming language.

If the price of doing that is sticking to a base set of primitives,
it's a small price to pay, not much different from sticking to python
2.7 or perl 5.8 or whatever.

Shell *is* the universal scripting language, not perl (even though we
all know it is what God himself used to create the world -- see xkcd
224 if you don't believe me!), not python, not Ruby.

-- 
sitaram

Re: Python extension commands in git - request for policy change

[Guillaume DE BURE]

Le mardi 27 novembre 2012 02:51:04 David Aguilar a écrit :
> On Tue, Nov 27, 2012 at 1:17 AM, Sitaram Chamarty <sitaramc@gmail.com> 
wrote:
> > On Tue, Nov 27, 2012 at 1:24 PM, David Aguilar <davvid@gmail.com> wrote:
> >> *cough* git-cola *cough*
> >> 
> >> it runs everywhere.  Yes, windows too. It's written in python.
> >> It's been actively maintained since 2007.
> >> 
> >> It's "modern" and has features that don't exist anywhere else.
> >> 
> >> It even has tests.  It even comes with a building full of willing
> >> guinea-pigs^Wtesters that let me know right away when
> >> anything goes wrong.
> >> 
> >> It uses Qt but that's really the whole point of Qt -> cross-platform.
> >> (not sure how that wiki page ended up saying Gnome/GTK?)
> >> 
> >> The DAG aka git-dag (in its master branch, about to be released)
> >> is nicer looking then gitk IMO.  gitk still has some features
> >> that are better too--there's no silver bullet, but the delta
> >> is pretty small.
> > 
> > Gitk does a lot of things that people don't realise, since they're not
> > really documented and you have to scrounge around on the UI.  The
> > thing is, it's just about the most awesome tool for code archeology I
> > have seen.
> > 
> > I realise (from looking at the doc page) that git-cola helps you do
> > all sorts of things, but those are all things I am happier doing at
> > the command line.
> 
> Ditto.  There's actually a few small things I use it for,
> mainly for teasing apart commits.  These days you can use git-gui
> for that, but in the old days it was the only way to interactively
> select individual lines and stage/unstage/revert them, etc.
> I don't think we can line-by-line revert in git-gui yet, though.
> 
> Some other small things that I use: ctrl-g, type something
> for grep, hit enter twice and I'm in my editor on that
> (or any other selected) line.  'spacebar' does xdg-open,
> and 'enter' launches the editor in the status widget;
> small things.  I, too, do most stuff on the command line.
> 
> The grep thing is a good example.  You have tons of output,
> you see the one line that you care about, and you want to jump
> there.  Clicking on that line and hitting enter is the minimal
> effort to do that.  You don't have to click because we also
> have keyboard navigation.  I have a feeling that there's probably
> something I'm missing, though.. another way of working (emacs?)
> that would render all of this custom GUI stuff pointless.
> 
> What I learned about users:
> 
> The commit editor is the #1 thing that got my coworkers finally
> writing better commit messages. It forces the subject/description
> separation and shows yellow, red when the subject gets too long.
> It also auto-wraps.  IMO it makes sense for git-gui to do
> the same these days.
> 
> > Gitk does precisely those things which *require* a GUI, where the
> > amount of information presented overwhelms a text interface.  The
> > display is concisely designed to give you the maximum information at a
> > minimum space use.  For example, a little black square when a commit
> > has a note attached.  Even hovering over the arrow-heads, on complex
> > trees where the line gets broken, does something meaningful.
> > 
> > if I had to pin it down, the feature I use most often is "Show origin
> > of this line".  Other features I use often are
> > 
> >   - review a commit file by file (f and b keys, also spacebar and 'd')
> >   - search by SHA1 (4 digits appear to be enough, regardless of how
> > 
> > big your repo is),
> > 
> >   - search for commits changing path/dir (while still showing all the
> > 
> > commits; i.e., this is not 'git-dag -- README.txt' but within gitk you
> > search up and down for commits touching README.txt
> > 
> >   - and navigating the commit tree looking for stuff
> > 
> > http://sitaramc.github.com/1-basic-usage/gitk.html is my attempt to
> > document some of the stuff I have found and use.
> 
> Wow, this is awesome.
> 
> > One final point: the DAG on the right wastes enormous amounts of
> > space.  Purely subjectively, it is almost jarring on the senses.  (If
> > you reduce it, it becomes unreadable).
> > 
> > With all due respect, git-cola/dag isn't anywhere near what gitk does,
> > at least for people who are not afraid of the command line and only
> > need the GUI to visualise a truly complex tree.
> 
> This is really great feedback.
> cc:ing Guillaume since he had similar ideas.
> 
> thx,

Thanks David for cc:ing me. I'm sorry I was quite below the radar lately, had 
too much work to get back on the dag. What I would really like for the dag is 
to become as useful as the treeview in git extensions 
(http://code.google.com/p/gitextensions/), where it is the central place for 
checkout, merge, cherry picks... 

My only complaint with git extensions is that it is poorly integrated on my 
linux desktop, due to framework choice (let's not start a flame war here, it's 
not the point). On the other hand, git-cola is quite easy to hack on thanks to 
its python roots, well integrated on all OS thanks to Qt, so I thought it 
would be great to make it at least on par.

Had an opportunity to rework the visuals on the dag, but not yet its 
functionalities... As soon as I'm back on normal life, I'll pickup the subject 
again :)

Cheers,

Guillaume

-- 
Skrooge, a free, Open Source, personal finances software for linux, Mac OS, 
Windows
http://skrooge.org

Re: Python extension commands in git - request for policy change

[Felipe Contreras]

On Tue, Nov 27, 2012 at 3:35 PM, Magnus Bäck <baeck@google.com> wrote:
> On Sunday, November 25, 2012 at 06:40 EST,
>      Felipe Contreras <felipe.contreras@gmail.com> wrote:
>
>> On Sun, Nov 25, 2012 at 11:44 AM, Michael Haggerty
>> <mhagger@alum.mit.edu> wrote:
>
> [...]
>
>> > On the contrary, there is *constant* traffic on the mailing list
>> > about incompatibilities between different shell implementations (sh,
>> > dash, bash, etc), not to mention those in other utilities (sed,
>> > grep, etc) that one is forced to work with in shell scripts.
>> > Compatibility is a *huge* pain when developing shell code for git.
>> > The fact that users typically don't encounter such problems is due
>> > to the hard work of POSIX lawyers on the mailing list correcting the
>> > compatibility errors of mortal programmers.
>>
>> *Theoretical* incompatibilities on probably obscure systems. *I* have
>> never seen such compatibility issues *in practice*.
>
> While "constant traffic" probably overstates the issue, these are not
> theoretical problems. I recall at least three cases in the last year
> or so where Git has seen breakage with Solaris or Mac OS X because
> of sed or tr incompatibilities, and I don't even read this list that
> thoroughly.

Most of the *constant* traffic is about *theoretical*
incompatibilities, how much of that are real incompatibilities, it's
not known. _Some_ of the traffic is about real incompatibilities,
sure, but you could count only three cases *in a year*. It's not a
huge amount. And then, how man this year?

Also, I would like references to those incompatibilities.

Cheers.

-- 
Felipe Contreras

Re: Python extension commands in git - request for policy change

[Felipe Contreras]

On Tue, Nov 27, 2012 at 7:35 PM, Eric S. Raymond <esr@thyrsus.com> wrote:

> As a general scripting language shell sucks *really badly* compared to
> anything new-school. Performance, portability, you name it, it's a
> mess.  It's not so much the shell interpreters itself that are the
> portabilty problem, but (as Magnus implicitly points out) all those
> userland dependencies on sed and tr and awk and even variants of
> expr(!) that get dragged in the second you try to get any actual work
> done.

Somehow it has worked perfectly fine for us.

Also, you are ignoring all the advantages that shell has and python does not.

> Can we cease behaving like we're still pounding keys on 110-baud
> teletypes now?  Some old-school Unix habits have persisted long past
> the point that they're even remotely sane.  Shell programming at any
> volume above a few lines of throwaway code is one of them - it's
> *nuts* and we should *stop doing it*.

Yes, let's all switch to ruby! Nah, everybody is still using Unix and
shells, they work. If they didn't, we wouldn't be having this
conversation.

What is the big problem? Somebody said we hit a few issues per year?
I've already hit more than that writing code for python just the last
month.

Cheers.

-- 
Felipe Contreras

Re: Python extension commands in git - request for policy change

[Jeff King]

On Wed, Nov 28, 2012 at 01:10:34AM +0100, Felipe Contreras wrote:

> > While "constant traffic" probably overstates the issue, these are not
> > theoretical problems. I recall at least three cases in the last year
> > or so where Git has seen breakage with Solaris or Mac OS X because
> > of sed or tr incompatibilities, and I don't even read this list that
> > thoroughly.
> 
> Most of the *constant* traffic is about *theoretical*
> incompatibilities, how much of that are real incompatibilities, it's
> not known. _Some_ of the traffic is about real incompatibilities,
> sure, but you could count only three cases *in a year*. It's not a
> huge amount. And then, how man this year?
> 
> Also, I would like references to those incompatibilities.

Try:

  git log --grep='portab' -- '*.sh'

Not all of the hits are shell portability fixes, but most of them are,
and they are all in response to real, reported issues. The usual
culprits are Solaris, BSD (including OS X), and the occasional GNUism.
And that is only the ones that we fixed in response to bug reports for
commits in the wild. Many more have been caught in review before needing
a separate patch (grepping the list archive for 'portable' and '\.sh'
yields 1800 messages).

So dealing with shell portability is definitely something we do, and it
is a minor pain.

But like you, I think we have been getting along reasonably well with
shell scripts (and where it is not powerful enough, writing C). No
solution is going to be free of portability issues (whether because of
differing versions, because the tool is uncommon on certain platforms,
or whatever). And because git-core's official API is shell-executable
commands, any other language you write ends up looking a lot like shell
anyway.

If people are really hankering to write sub-commands of git in python
(or whatever), I would suggest checking out libgit2 which has a nice set
of python bindings (and ruby bindings, and C# bindings, and so on). It
doesn't have feature parity with git-core yet, but they have been making
a lot of progress.

-Peff

Re: Python extension commands in git - request for policy change

[Felipe Contreras]

On Wed, Nov 28, 2012 at 1:51 AM, Jeff King <peff@peff.net> wrote:
> On Wed, Nov 28, 2012 at 01:10:34AM +0100, Felipe Contreras wrote:
>
>> > While "constant traffic" probably overstates the issue, these are not
>> > theoretical problems. I recall at least three cases in the last year
>> > or so where Git has seen breakage with Solaris or Mac OS X because
>> > of sed or tr incompatibilities, and I don't even read this list that
>> > thoroughly.
>>
>> Most of the *constant* traffic is about *theoretical*
>> incompatibilities, how much of that are real incompatibilities, it's
>> not known. _Some_ of the traffic is about real incompatibilities,
>> sure, but you could count only three cases *in a year*. It's not a
>> huge amount. And then, how man this year?
>>
>> Also, I would like references to those incompatibilities.
>
> Try:
>
>   git log --grep='portab' -- '*.sh'

% git log --oneline --grep='portab' --since='2 years ago' --no-merges -- '*.sh'
6eafa6d submodules: don't stumble over symbolic links when cloning recursively

Somebody mentioned 'portable', but no problem was hit.

2718781 t9400: fix gnuism in grep

It's a test, and nobody was hit by any problem.

0dbe659 t5704: fix nonportable sed/grep usages

Apparently there was an issue, but this is a test.

93d5e0c t7810: avoid unportable use of "echo"

A problem, but in tests.

482ce70 tests: avoid nonportable {foo,bar} glob

Again, tests.

77e5726 t0050: fix printf format strings for portability

Tests.


So, this search didn't throw a single issue that affected users in two
years. Moving git sub-commands to python wouldn't change a thing.
Maybe shell wasn't the right language for the test system, but I don't
see anybody proposing it to be changed.

> Not all of the hits are shell portability fixes, but most of them are,
> and they are all in response to real, reported issues. The usual
> culprits are Solaris, BSD (including OS X), and the occasional GNUism.
> And that is only the ones that we fixed in response to bug reports for
> commits in the wild. Many more have been caught in review before needing
> a separate patch (grepping the list archive for 'portable' and '\.sh'
> yields 1800 messages).
>
> So dealing with shell portability is definitely something we do, and it
> is a minor pain.

First you have to separate the issues with the test system, because
that's not going to be changed. And then you have to separate the
*potential* issues and the *real* ones. You can spend all your time
doing "shell portability", but does that mean that you actually have a
problem? Maybe if you hadn't done anything, nobody would have noticed
there was a problem.

Sure, you will argue that we don't see the *real* issues, because they
were fixed preemptively, but the fact of the matter is that we will
never know. All we know is the reality we can observe, and the reality
is that we hit very few *real* issues outside the test system (feel
free to provide evidence to the contrary).

> But like you, I think we have been getting along reasonably well with
> shell scripts (and where it is not powerful enough, writing C). No
> solution is going to be free of portability issues (whether because of
> differing versions, because the tool is uncommon on certain platforms,
> or whatever). And because git-core's official API is shell-executable
> commands, any other language you write ends up looking a lot like shell
> anyway.

Agreed.

Cheers.

-- 
Felipe Contreras

Re: Python extension commands in git - request for policy change

[Jeff King]

On Wed, Nov 28, 2012 at 02:22:09AM +0100, Felipe Contreras wrote:

> Sure, you will argue that we don't see the *real* issues, because they
> were fixed preemptively, but the fact of the matter is that we will
> never know. All we know is the reality we can observe, and the reality
> is that we hit very few *real* issues outside the test system (feel
> free to provide evidence to the contrary).

I think reports of breakage in the test scripts are relevant, because
they are indicative that people _do_ run platforms that care about these
issues, and if we were to write a lot of shell scripts, we would run
across them more frequently. But the fact of the matter is that we don't
write a lot of non-test shell scripts these days, which is part of the
reason limiting your search to the last 2 years did not turn up many
fixes outside the tests.

There was a big push in 2006 and 2007 to port some of the hairier
scripts to C. Try:

  git log --no-renames --diff-filter=D \
          --diff-filter=D --format='%ad %s' --date=short \
          -- 'git-*.sh'

A lot of it was motivated by portability and decent performance for
common commands under Windows.

Anyway, there is not much point in debating the exact level of pain that
shell portability causes us. Even if you accept that there is some, it
is clearly not a major problem for the project.

-Peff

Re: Python extension commands in git - request for policy change

[Felipe Contreras]

On Wed, Nov 28, 2012 at 2:39 AM, Jeff King <peff@peff.net> wrote:
> On Wed, Nov 28, 2012 at 02:22:09AM +0100, Felipe Contreras wrote:
>
>> Sure, you will argue that we don't see the *real* issues, because they
>> were fixed preemptively, but the fact of the matter is that we will
>> never know. All we know is the reality we can observe, and the reality
>> is that we hit very few *real* issues outside the test system (feel
>> free to provide evidence to the contrary).
>
> I think reports of breakage in the test scripts are relevant, because
> they are indicative that people _do_ run platforms that care about these
> issues, and if we were to write a lot of shell scripts, we would run
> across them more frequently. But the fact of the matter is that we don't
> write a lot of non-test shell scripts these days, which is part of the
> reason limiting your search to the last 2 years did not turn up many
> fixes outside the tests.

If we were to write a lot of shell scripts, and we were to apply the
same standards as we do with the tests, which most likely wouldn't be
the case; end-user scripts are way more important, specially
porcelain.

> There was a big push in 2006 and 2007 to port some of the hairier
> scripts to C. Try:
>
>   git log --no-renames --diff-filter=D \
>           --diff-filter=D --format='%ad %s' --date=short \
>           -- 'git-*.sh'
>
> A lot of it was motivated by portability and decent performance for
> common commands under Windows.

Good stuff indeed. I look forward to the day all main git porcelain
commands are written in C (git-rebase I'm looking at you), there are
not many left:

git-am
git-bisect
git-citool
git-gui
git-pull
git-rebase
git-stash
git-submodule

> Anyway, there is not much point in debating the exact level of pain that
> shell portability causes us. Even if you accept that there is some, it
> is clearly not a major problem for the project.

Indeed.

-- 
Felipe Contreras

Re: Python extension commands in git - request for policy change

[Felipe Contreras]

On Sun, Nov 25, 2012 at 10:56 PM, Eric S. Raymond <esr@thyrsus.com> wrote:
> Felipe Contreras <felipe.contreras@gmail.com>:
>> And gitk is an integral part of git. But if you have different
>> numbers, what are they?

> Please don't waste further time on quibbling.  We all know that gitk is
> an uncomfortable special case and that the project would be far better
> off, maintainability-wise, if it were successfully ported to one if these
> other languages.  Trying to catch me out by triumphantly pointing at gitk
> is...juvenile.

Another bit of information I just realized, 'man git' lists gitk as a
'Main porcelain command' as high level as any git command can get.

-- 
Felipe Contreras

-- 
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to msysgit@googlegroups.com
To unsubscribe from this group, send email to
msysgit+unsubscribe@googlegroups.com
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en

Re: Python extension commands in git - request for policy change

[Joshua Jensen]

----- Original Message -----
From: Michael Haggerty
Date: 11/25/2012 3:44 AM
> * Startup time: Is the time to start the "X" interpreter prohibitive?
>    (On my computer, "python -c pass", which starts the Python
>    interpreter and does nothing, takes about 24ms.)  This overhead would
>    be incurred by every command that is not pure C.
FWIW, on Windows 7 running on a Core i7 1.6GHz quad core processor, a 
cold run of "python -c pass" after a reboot results in 0.74 seconds.  
The second run takes 0.13 seconds.

Because Lua was mentioned in another message of this thread, I'll 
provide the following:

* A cold run of 'lua5.1 -e ""' takes 0.4 seconds.  The second run takes 
0.03 seconds.
* A cold run of LuaJIT takes the same.

-Josh

Re: Python extension commands in git - request for policy change

[Magnus Bäck]

On Tuesday, November 27, 2012 at 19:10 EST,
     Felipe Contreras <felipe.contreras@gmail.com> wrote:

> On Tue, Nov 27, 2012 at 3:35 PM, Magnus Bäck <baeck@google.com> wrote:
>
> > While "constant traffic" probably overstates the issue, these are
> > not theoretical problems. I recall at least three cases in the last
> > year or so where Git has seen breakage with Solaris or Mac OS X
> > because of sed or tr incompatibilities, and I don't even read this
> > list that thoroughly.
>
> Most of the *constant* traffic is about *theoretical*
> incompatibilities, how much of that are real incompatibilities, it's
> not known. _Some_ of the traffic is about real incompatibilities,
> sure, but you could count only three cases *in a year*. It's not a
> huge amount. And then, how man this year?
>
> Also, I would like references to those incompatibilities.

I don't remember the details of the Mac OS X problem, but just searching
the archives for "xpg4" revealed the following Solaris problems since
April:

http://article.gmane.org/gmane.comp.version-control.git/195010
http://article.gmane.org/gmane.comp.version-control.git/195966
http://article.gmane.org/gmane.comp.version-control.git/207944

-- 
Magnus Bäck
baeck@google.com

Re: Python extension commands in git - request for policy change

[Philippe Vaucher]

> Also, you are ignoring all the advantages that shell has and python does not.

Out of curiosity, can you list the advantages? From what I gathered:

- no need to install bash
- git contributors are more used to bash
- there's only one "version" of bash (no real need to handle different
versions compared to py26, py27, etc)

Are there any "language" advantage or are all the advantages in the
"pragmatic" section?
I don't really have an opinion on this topic. A "real language" seems
better, but bash's pragmaticity seems to outweight the cons,
especially the "contributors want bash" argument.

Thanks,
Philippe

Re: Python extension commands in git - request for policy change

[Felipe Contreras]

On Mon, Dec 3, 2012 at 3:45 PM, Philippe Vaucher
<philippe.vaucher@gmail.com> wrote:
>> Also, you are ignoring all the advantages that shell has and python does not.
>
> Out of curiosity, can you list the advantages? From what I gathered:
>
> - no need to install bash

Unless you are in Windows or OS X. OS X has a shell, but not bash.

> - git contributors are more used to bash

I don't see this as a big advantage.

> - there's only one "version" of bash (no real need to handle different
> versions compared to py26, py27, etc)

This one is.

The language doesn't change much from different versions of bash either way.

- the language is part of POSIX

Which means you don't need bash, you can use dash, or zsh, or many other shells.

- the language is simple

It's so simple it's used in shells.

- no need for extensions

All you need is the shell binary, that's it.

Cheers.

-- 
Felipe Contreras

Re: Python extension commands in git - request for policy change

[Stephen Bash]

----- Original Message -----
> From: "Felipe Contreras" <felipe.contreras@gmail.com>
> Sent: Tuesday, December 4, 2012 9:19:18 AM
> Subject: Re: Python extension commands in git - request for policy change
> 
> > > Also, you are ignoring all the advantages that shell has and
> > > python does not.
> >
> > Out of curiosity, can you list the advantages? From what I
> > gathered:
> >
> > - no need to install bash
> 
> Unless you are in Windows or OS X. OS X has a shell, but not bash.

>From http://support.apple.com/kb/TA27005:

"The default shell (or command-line interface) used in Mac OS X 10.0 through 10.2.8 is tcsh (with 10.3 and 10.4 it's bash). With Mac OS X 10.2 or later, other interactive shells are included, such as bash and zsh."

Stephen

Re: Python extension commands in git - request for policy change

[Martin Langhoff]

On Sat, Nov 24, 2012 at 9:44 PM, Eric S. Raymond <esr@thyrsus.com> wrote:
> git presently contains one Python extension command, Pete Wycoff's p4
> importer.  If my git-weave code is merged it will acquire another.

Write a really compelling tool. Don't argue languages. Make it
wonderful. The git maintainers, tool maintainers (you!) and overall
development community have shown flexibility and smarts in the past.

cheers,


m
--
 martin.langhoff@gmail.com
 martin@laptop.org -- Software Architect - OLPC
 - ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 - http://wiki.laptop.org/go/User:Martinlanghoff

Re: Python extension commands in git - request for policy change

[Patrick Donnelly]

Sorry I'm late to this party...

I'm an Nmap developer that is casually interested in git development.
I've been lurking for a while and thought I'd post my thoughts on this
thread.

On Sun, Nov 25, 2012 at 6:25 AM, Nguyen Thai Ngoc Duy <pclouds@gmail.com> wrote:
>> The most important issues to consider when imagining a future with a
>> hybrid of code in C and some scripting language "X" are:
>>
>> * Portability: is "X" available on all platforms targeted by git, in
>>   usable and mutually-compatible versions?
>>
>> * Startup time: Is the time to start the "X" interpreter prohibitive?
>>   (On my computer, "python -c pass", which starts the Python
>>   interpreter and does nothing, takes about 24ms.)  This overhead would
>>   be incurred by every command that is not pure C.
>>
>> * Should the scripting language access the C functionality only by
>>   calling pure-C executables or by dynamically or statically linking to
>>   a binary module interface?  If the former, then the granularity of
>>   interactions between "X" and C is necessarily coarse, and "X" cannot
>>   be used to implement anything but the outermost layer of
>>   functionality.  If the latter, then the way would be clear to
>>   implement much more of git in "X" (and lua would also be worth
>>   considering).
>>
>> * Learning curve for developers: how difficult is it for a typical git
>>   developer to become conversant with "X", considering both (1) how
>>   likely is it that the typical git developer already knows "X" and
>>   (2) how straightforward and predictable is the language "X"?
>>   In this category I think that Python has a huge advantage over
>>   Perl, though certainly opinions will differ and Ruby would also be
>>   a contender.
>
> * We might also need an embedded language variant, like Jeff's lua
> experiment. I'd be nice if "X" can also take this role.

Lua has been an incredible success for Nmap [2](and other projects).
As an embedded scripting language, it's unrivaled in terms of ease of
embedding, ease of use for users, and performance. I would strongly
recommend the git developers to seriously consider it.

Addressing the points listed above:

>> * Portability: is "X" available on all platforms targeted by git, in
>>   usable and mutually-compatible versions?

Lua is written in ANSI C and so compiles basically anywhere (certainly
anywhere git does).

>> * Startup time: Is the time to start the "X" interpreter prohibitive?
>>   (On my computer, "python -c pass", which starts the Python
>>   interpreter and does nothing, takes about 24ms.)  This overhead would
>>   be incurred by every command that is not pure C.

As mentioned elsewhere in this thread by Joshua:

"Because Lua was mentioned in another message of this thread, I'll
provide the following:

* A cold run of 'lua5.1 -e ""' takes 0.4 seconds.  The second run
takes 0.03 seconds.
* A cold run of LuaJIT takes the same."

The runtime figures would probably be much lower if git modules (e.g.
pull) were capable of calling other modules without forking, all
within the same Lua environment.

>> * Should the scripting language access the C functionality only by
>>   calling pure-C executables or by dynamically or statically linking to
>>   a binary module interface?  If the former, then the granularity of
>>   interactions between "X" and C is necessarily coarse, and "X" cannot
>>   be used to implement anything but the outermost layer of
>>   functionality.  If the latter, then the way would be clear to
>>   implement much more of git in "X" (and lua would also be worth
>>   considering).

Using Lua as a glue between a proper git C API and modules would be
optimal in my opinion and experience.

>> * Learning curve for developers: how difficult is it for a typical git
>>   developer to become conversant with "X", considering both (1) how
>>   likely is it that the typical git developer already knows "X" and
>>   (2) how straightforward and predictable is the language "X"?
>>   In this category I think that Python has a huge advantage over
>>   Perl, though certainly opinions will differ and Ruby would also be
>>   a contender.

Lua is remarkably easy to learn and is engineered so it's approachable
by novice (or non-) programmers. Still, it offers all the features you
expect from a modern scripting language including GC, real lexical
scoping and closure, tables/arrays, and *coroutines*. While Nmap
occasionally gets questions about why we didn't use Python, no one
complains about Lua itself.


As for version considerations, Lua changes at a glacial pace compared
to other popular languages like Python or Ruby. Lua 5.2 was released
last year and 5.1 was released 5 years before that [1]. Still, while
users (people who bind Lua to applications) are certainly encouraged
to upgrade Lua, there is no real need to. Binding Lua to an
application statically is not a significant cost as it compiles to
less than 200 KB including base libraries; so, it's simple and cheap
to remain independent of the system git runs on. It isn't unreasonable
-- indeed, it is common -- to select one version of Lua and keep it
for a significant lifetime of the project.

[I'd be happy to answer any questions concerning Lua and/or scripting
Git. I'd also be happy to assist in embedding Lua in Git.]

[1] http://www.lua.org/versions.html
[2] http://nmap.org/book/nse.html

--
- Patrick Donnelly

Re: Python extension commands in git - request for policy change

[Sitaram Chamarty]

On Tue, Dec 11, 2012 at 11:14 AM, Patrick Donnelly <batrick@batbytes.com> wrote:
> Sorry I'm late to this party...
>
> I'm an Nmap developer that is casually interested in git development.
> I've been lurking for a while and thought I'd post my thoughts on this
> thread.
>
> On Sun, Nov 25, 2012 at 6:25 AM, Nguyen Thai Ngoc Duy <pclouds@gmail.com> wrote:
>>> The most important issues to consider when imagining a future with a
>>> hybrid of code in C and some scripting language "X" are:
>>>
>>> * Portability: is "X" available on all platforms targeted by git, in
>>>   usable and mutually-compatible versions?
>>>
>>> * Startup time: Is the time to start the "X" interpreter prohibitive?
>>>   (On my computer, "python -c pass", which starts the Python
>>>   interpreter and does nothing, takes about 24ms.)  This overhead would
>>>   be incurred by every command that is not pure C.
>>>
>>> * Should the scripting language access the C functionality only by
>>>   calling pure-C executables or by dynamically or statically linking to
>>>   a binary module interface?  If the former, then the granularity of
>>>   interactions between "X" and C is necessarily coarse, and "X" cannot
>>>   be used to implement anything but the outermost layer of
>>>   functionality.  If the latter, then the way would be clear to
>>>   implement much more of git in "X" (and lua would also be worth
>>>   considering).
>>>
>>> * Learning curve for developers: how difficult is it for a typical git
>>>   developer to become conversant with "X", considering both (1) how
>>>   likely is it that the typical git developer already knows "X" and
>>>   (2) how straightforward and predictable is the language "X"?
>>>   In this category I think that Python has a huge advantage over
>>>   Perl, though certainly opinions will differ and Ruby would also be
>>>   a contender.
>>
>> * We might also need an embedded language variant, like Jeff's lua
>> experiment. I'd be nice if "X" can also take this role.
>
> Lua has been an incredible success for Nmap [2](and other projects).
> As an embedded scripting language, it's unrivaled in terms of ease of
> embedding, ease of use for users, and performance. I would strongly
> recommend the git developers to seriously consider it.

[snipping the rest; all valid points no doubt]

Does lua have os.putenv() yet?  The inability to even *set* an env var
before calling something else was a killer for me when I last tried
it.

That may make it fine as an embedded language (called *by* something
else) but it is a bit too "frugal" to use as a glue language (calls
other things).

Re: Python extension commands in git - request for policy change

[Patrick Donnelly]

Hi Sitaram,

On Tue, Dec 11, 2012 at 7:09 PM, Sitaram Chamarty <sitaramc@gmail.com> wrote:
> On Tue, Dec 11, 2012 at 11:14 AM, Patrick Donnelly <batrick@batbytes.com> wrote:
>> Lua has been an incredible success for Nmap [2](and other projects).
>> As an embedded scripting language, it's unrivaled in terms of ease of
>> embedding, ease of use for users, and performance. I would strongly
>> recommend the git developers to seriously consider it.
>
> [snipping the rest; all valid points no doubt]
>
> Does lua have os.putenv() yet?  The inability to even *set* an env var
> before calling something else was a killer for me when I last tried
> it.

Lua is pretty strict about being entirely ANSI C and makes very few
exceptions (e.g. dlopen). The exceptions that do exist are only in
base libraries which can easily be thrown out.

> That may make it fine as an embedded language (called *by* something
> else) but it is a bit too "frugal" to use as a glue language (calls
> other things).

As a glue language, this is a feature. The Programming in Lua (written
by the architect of Lua) preface [1] contains the philosophy of Lua on
this issue.

For cases where you do need access to functions like putenv, it's
trivial to either write wrappers for the functions you want to expose
or to incorporate a library that does it all, e.g. luaposix which
contains the majority of POSIX's system calls.

[1] http://www.lua.org/pil/p1.html

--
- Patrick Donnelly

Re: Python extension commands in git - request for policy change

[Tomas Carnecky]

On Wed, 12 Dec 2012 05:39:43 +0530, Sitaram Chamarty <sitaramc@gmail.com> wrote:
> Does lua have os.putenv() yet?  The inability to even *set* an env var
> before calling something else was a killer for me when I last tried
> it.

If it doesn't, it would be trivial to add. It's a one-liner. It's been a while
since I used Lua, but it would be something like this:

    void L_putenv(lua_State *L) {
        putenv(lua_tostring(L, 1));
    }

and then somewhere during setup:

    lua_register(L, "putenv", L_putenv);

Re: Python extension commands in git - request for policy change

[Nguyen Thai Ngoc Duy]

On Wed, Dec 12, 2012 at 7:53 AM, Tomas Carnecky
<tomas.carnecky@gmail.com> wrote:
> If it doesn't, it would be trivial to add. It's a one-liner. It's been a while
> since I used Lua, but it would be something like this:
>
>     void L_putenv(lua_State *L) {
>         putenv(lua_tostring(L, 1));
>     }
>
> and then somewhere during setup:
>
>     lua_register(L, "putenv", L_putenv);

I should have done my homework before asking, but well.. is there any
way to automate this? If we use lua for writing "builtin" commands,
we'll need to export a lot of C functions and writing wrappers like
this is boring and time consuming. Also, assume I export fn(char*,int)
to Lua, then I change the prototype to fn(char*, char*), can Lua spot
all the call sites at compile time (or something) so I can update
them?
-- 
Duy

Re: Python extension commands in git - request for policy change

[Tomas Carnecky]

On Wed, 12 Dec 2012 08:50:27 +0700, Nguyen Thai Ngoc Duy <pclouds@gmail.com> wrote:
> On Wed, Dec 12, 2012 at 7:53 AM, Tomas Carnecky
> <tomas.carnecky@gmail.com> wrote:
> > If it doesn't, it would be trivial to add. It's a one-liner. It's been a while
> > since I used Lua, but it would be something like this:
> >
> >     void L_putenv(lua_State *L) {
> >         putenv(lua_tostring(L, 1));
> >     }
> >
> > and then somewhere during setup:
> >
> >     lua_register(L, "putenv", L_putenv);
> 
> I should have done my homework before asking, but well.. is there any
> way to automate this? If we use lua for writing "builtin" commands,
> we'll need to export a lot of C functions and writing wrappers like
> this is boring and time consuming. Also, assume I export fn(char*,int)
> to Lua, then I change the prototype to fn(char*, char*), can Lua spot
> all the call sites at compile time (or something) so I can update
> them?

A Patrick mentioned in an earlier email, there is luaposix which includes lots
of these functions [1]. There may be tools which generate these bindings
automatically, but I'm not aware of any. Likewise, I'm not aware of any static
analyzer which would be required to spot changes in the prototypes (though you
could cover some(most?) of it through tests). The last time I seriously used
Lua and its C bindings was many years ago, so I am not the best person to
answer these questions.

[1]: http://luaposix.github.com/luaposix/docs/index.html

Re: Python extension commands in git - request for policy change

[Patrick Donnelly]

Hi Duy,

On Tue, Dec 11, 2012 at 8:50 PM, Nguyen Thai Ngoc Duy <pclouds@gmail.com> wrote:
> On Wed, Dec 12, 2012 at 7:53 AM, Tomas Carnecky
> <tomas.carnecky@gmail.com> wrote:
>> If it doesn't, it would be trivial to add. It's a one-liner. It's been a while
>> since I used Lua, but it would be something like this:
>>
>>     void L_putenv(lua_State *L) {
>>         putenv(lua_tostring(L, 1));
>>     }
>>
>> and then somewhere during setup:
>>
>>     lua_register(L, "putenv", L_putenv);
>
> I should have done my homework before asking, but well.. is there any
> way to automate this?

If you want these basic POSIX functions, use an existing library.

If you want to automate adding a number of application specific
functions, you can use swig or similar. AFAIK, all languages rely on
third party tools like swig to assist in automated binding generation.
Although, automated binding generation is usually used to make it easy
to export bindings for multiple languages easily. If Lua is going to
be used as a "standard" module glue language, using swig is really
overkill.

> If we use lua for writing "builtin" commands,
> we'll need to export a lot of C functions and writing wrappers like
> this is boring and time consuming. Also, assume I export fn(char*,int)
> to Lua, then I change the prototype to fn(char*, char*), can Lua spot
> all the call sites at compile time (or something) so I can update
> them?

If the API calls are generic (don't require special handling), you can
use some preprocessor magic to save time/space.

--
- Patrick Donnelly

Re: Python extension commands in git - request for policy change

[Eric S. Raymond]

Sitaram Chamarty <sitaramc@gmail.com>:
> [snipping the rest; all valid points no doubt]

I meant to respond to Patrick's post earlier.

I haven't actually written any code in lua yet, but I've read the book;
I think I get it.  I've seen the effects of lua integration on another
large project, Battle for Wesnoth.

I'm not, despite conclusions some people here might have jumped to,
religiously attached to Python.  So I can say this: I think lua as a
language is an *excellent* design.  It is clever, economical,
minimalist, and (other than the one ugly detail of 1-origin indexing)
shows consistent good taste.

It might be a good fit for extending git; I wouldn't be very surprised if
that worked. However, I do have concerns about the "Oh, we'll just
lash together a binding to C" attitude common among lua programmers; I
foresee maintainability problems and the possibility of slow death by
low-level details as that strategy tries to scale up.

And, of course, one problem with calling back into C a lot is that
you walk back into C's resource-management issues.

My sense is that git's use cases are better served by a glue language
in the Python/Perl/Ruby class rather than an extension langage. But
my mind is open on this issue.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Python extension commands in git - request for policy change

[Joshua Jensen]

----- Original Message -----
From: Eric S. Raymond
Date: 12/11/2012 8:30 PM
> It might be a good fit for extending git; I wouldn't be very surprised if
> that worked. However, I do have concerns about the "Oh, we'll just
> lash together a binding to C" attitude common among lua programmers; I
> foresee maintainability problems and the possibility of slow death by
> low-level details as that strategy tries to scale up.
I don't understand this statement: "Oh, we'll just lash together a 
binding to C" attitude.

??
> My sense is that git's use cases are better served by a glue language
> in the Python/Perl/Ruby class rather than an extension langage. But
> my mind is open on this issue.
I spend nearly 100% of my Git time on Windows.

Spawning new processes in Windows is dog slow.  Using 'git rebase', 
arguably my favorite Git command, is time-waiting torture.  I'm also on 
about as fast of a Windows machine as money can buy these days.

I have a Git add-on similar to git-media that uses the smudge and clean 
filters to read/write large binary files into a separate storage 
location.  When checking out a workspace, Git shells out to run a filter 
for each file it needs to write to the workspace.

I can get a maximum of 100 processes per second with this technique, 
resulting in just 100 files being written to disk.  However, I tend to 
see closer to 60 files written to disk.

So, I patched Git to allow the smudge/clean filters to load up a DLL 
that executes a Lua script.  The Lua script properly retrieves+caches a 
file locally, or it puts the file on a network share.

The in-process DLL checkout ends up being every bit as fast as when we 
use Perforce to sync files to our local workspace.  Git, then, can be a 
Perforce replacement for our needs.

(For those who don't know, Perforce handles large workspaces with 
massive binary files very efficiently.)

Anyway, my preference is to allow scripts to run in-process within Git, 
because it is far, far faster on Windows.  I imagine it is faster than 
forking processes on non-Windows machines, too, but I have no statistics 
to back that up.

Python, Perl, or Ruby can be embedded, too, but Lua probably embeds the 
easiest and smallest out of those other 3 languages.

And shell scripts tend to be the slowest on Windows due to the excessive 
numbers of process invocations needed to get anything reasonable done.

-Josh

Re: Python extension commands in git - request for policy change

[Joshua Jensen]

----- Original Message -----
From: Patrick Donnelly
Date: 12/11/2012 7:26 PM
>> If we use lua for writing "builtin" commands,
>> we'll need to export a lot of C functions and writing wrappers like
>> this is boring and time consuming. Also, assume I export fn(char*,int)
>> to Lua, then I change the prototype to fn(char*, char*), can Lua spot
>> all the call sites at compile time (or something) so I can update
>> them?
> If the API calls are generic (don't require special handling), you can
> use some preprocessor magic to save time/space.
Since I mostly use C++, thanks to template metaprogramming, I get to 
register functions like so:

         void SomeExistingFunction(char* str, int num);

luaState->GetGlobals().RegisterDirect("SomeExistingFunction", 
SomeExistingFunction);

I certainly am not suggesting C++ be used within Git, but in this case, 
C++ has some nice compile-time advantages.

:)

-Josh

Re: Python extension commands in git - request for policy change

[Jeff King]

On Tue, Dec 11, 2012 at 10:30:43PM -0500, Eric S. Raymond wrote:

> My sense is that git's use cases are better served by a glue language
> in the Python/Perl/Ruby class rather than an extension langage. But
> my mind is open on this issue.

I think there are really two separate use cases to consider:

  1. Providing snippets of script to Git to get Turing-complete behavior
     for existing Git features. For example, selecting commits during a
     traversal (e.g., a better "log --grep"), formatting output (e.g., a
     better "log --format" or "for-each-ref --format").

  2. Writing whole new git commands in a language that is quicker or
     easier to develop in than C.

I think (1) is a good match for lua. It's light-weight and easy to
embed, we can map the few bits of information we want for each snippet
into the language (e.g., a commit object as a lua table), and the
language ecosystem is not that important (the user is more interested in
writing readable one-liners manipulating data provided by git than they
are in calling out to third-party modules).

But for (2), you are going to care a lot more about the language and its
ecosystem (because you'll be interacting more with the world outside of
git), and about having bindings to lots of different parts of git
(because you'll want to do more interesting things than just examine a
few data structures).  We provide that right now with executable
plumbing commands. That's convenient for shell scripts, and you can
build bindings for other languages on top (e.g., see perl/Git.pm).

It's nicely universal, but of course there are some drawbacks: it's slow
(fork and pipe overhead), and it's sometimes awkward (parsing, quoting,
no interactivity between caller and plumbing). The other obvious choice
for a lingua franca is a linkable C library, with bindings in your
language of choice.

It would take a lot of effort to expose git-core's internals in a clean
way; you'd probably be better off starting from scratch and rewriting
large parts in a friendly library-like manner. Fortunately, there is
already a project underway to do so: libgit2.  It does not yet have
feature parity with git, but it can do quite a bit.  And there are
already ruby and python bindings.

-Peff

Re: Python extension commands in git - request for policy change

[Patrick Donnelly]

Hi Jeff,

On Wed, Dec 12, 2012 at 1:32 AM, Jeff King <peff@peff.net> wrote:
> It would take a lot of effort to expose git-core's internals in a clean
> way; you'd probably be better off starting from scratch and rewriting
> large parts in a friendly library-like manner. Fortunately, there is
> already a project underway to do so: libgit2.  It does not yet have
> feature parity with git, but it can do quite a bit.  And there are
> already ruby and python bindings.

Of course, this comes back to the issue of whether it's a good idea to
use perl/ruby/python as a front-end to regular git commands
(pull/push/etc.). While, yes, bindings can be made for these
languages, you are now making git depend on the presence of one of
these languages in order for git to function. With Lua, the (static)
dependence is very small yet brings much to git in terms of
extensibility and maintainability.

As for Lua's suitability for your (2) point, I admit I'm not familiar
with how much "interacting with the outside world" the git commands
do; however, I would suspect that it is not significant enough to rule
Lua out?

--
- Patrick Donnelly

Re: Python extension commands in git - request for policy change

[Patrick Donnelly]

Hi Eric,

On Tue, Dec 11, 2012 at 10:30 PM, Eric S. Raymond <esr@thyrsus.com> wrote:
> It might be a good fit for extending git; I wouldn't be very surprised if
> that worked. However, I do have concerns about the "Oh, we'll just
> lash together a binding to C" attitude common among lua programmers; I
> foresee maintainability problems and the possibility of slow death by
> low-level details as that strategy tries to scale up.

I think this is quite a prediction? Could you give an example
scenario? How would another language (e.g. Python) mitigate this?

> And, of course, one problem with calling back into C a lot is that
> you walk back into C's resource-management issues.

C resource management can be effectively dealt with by relying on
Lua's GC to track C resources via userdata.

> My sense is that git's use cases are better served by a glue language
> in the Python/Perl/Ruby class rather than an extension langage. But
> my mind is open on this issue.

I don't see how these languages are more appropriate based on your concerns.

--
- Patrick Donnelly

Re: Python extension commands in git - request for policy change

[Jeff King]

On Wed, Dec 12, 2012 at 02:03:56AM -0500, Patrick Donnelly wrote:

> On Wed, Dec 12, 2012 at 1:32 AM, Jeff King <peff@peff.net> wrote:
> > It would take a lot of effort to expose git-core's internals in a clean
> > way; you'd probably be better off starting from scratch and rewriting
> > large parts in a friendly library-like manner. Fortunately, there is
> > already a project underway to do so: libgit2.  It does not yet have
> > feature parity with git, but it can do quite a bit.  And there are
> > already ruby and python bindings.
> 
> Of course, this comes back to the issue of whether it's a good idea to
> use perl/ruby/python as a front-end to regular git commands
> (pull/push/etc.).

Yeah, I think that is a separate issue, though. I cannot see us ever
writing core commands like "git pull" in any scripting language besides
POSIX shell due to dependency issues. So language bindings are really
for things that are not going to go into git-core, or are ancillary
commands that people can live without (e.g., git-add--interactive,
remote helpers, etc).

> While, yes, bindings can be made for these languages, you are now
> making git depend on the presence of one of these languages in order
> for git to function. With Lua, the (static) dependence is very small
> yet brings much to git in terms of extensibility and maintainability.

And I would include Lua in my list of "I cannot see..." above. It can be
statically linked, so it is not a run-time dependency, but it would
still be a build-time dependency. The community has historically been
pretty resistant to dependencies (I do not care too much myself,
though).

I think doing anything significant in Lua would have the same problem as
doing anything significant in Python: there would need to be substantial
internal cleanup to make sane bindings. And again, that is what libgit2
is doing (and yes, there are Lua bindings for it already).

Using libgit2 bindings would introduce a new dependency, of course, but
that is on par with a Lua dependency.

> As for Lua's suitability for your (2) point, I admit I'm not familiar
> with how much "interacting with the outside world" the git commands
> do; however, I would suspect that it is not significant enough to rule
> Lua out?

I did not mean to rule it out for point (2); I only meant that it is
probably the only reasonable thing for point (1), whereas for point (2),
we have many more options.  I suspect Lua would do just fine given the
right set of modules, though I tend to prefer other languages myself
when embeddedness is not an issue.

As for "interacting with the outside world", I was specifically thinking
of stuff like git-send-email (currently in perl) and git-imap-send
(written in C). They need to open network sockets and speak standard
protocols. I suspect Lua would need a module or custom bindings to do
the former at all, and certainly the code would be much simpler if we
re-used standard modules for speaking SMTP and IMAP (which of course
increases our dependencies again...).

-Peff

Re: Python extension commands in git - request for policy change

[Eric S. Raymond]

Joshua Jensen <jjensen@workspacewhiz.com>:
> Anyway, my preference is to allow scripts to run in-process within
> Git, because it is far, far faster on Windows.  I imagine it is
> faster than forking processes on non-Windows machines, too, but I
> have no statistics to back that up.
> 
> Python, Perl, or Ruby can be embedded, too, but Lua probably embeds
> the easiest and smallest out of those other 3 languages.
> 
> And shell scripts tend to be the slowest on Windows due to the
> excessive numbers of process invocations needed to get anything
> reasonable done.

I don't think there's *any* dimension along which lua is not clearly
better than shell for this sort of thing, so no argument there.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Python extension commands in git - request for policy change

[Eric S. Raymond]

Jeff King <peff@peff.net>:
> I think there are really two separate use cases to consider:
> 
>   1. Providing snippets of script to Git to get Turing-complete behavior
>      for existing Git features. For example, selecting commits during a
>      traversal (e.g., a better "log --grep"), formatting output (e.g., a
>      better "log --format" or "for-each-ref --format").
> 
>   2. Writing whole new git commands in a language that is quicker or
>      easier to develop in than C.

That's good analysis.  I agree with your use-case split, I guess I'm just not
very aware of the places in git where (1) is important.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Python extension commands in git - request for policy change

[Jeff King]

On Wed, Dec 12, 2012 at 07:26:25AM -0500, Eric S. Raymond wrote:

> Jeff King <peff@peff.net>:
> > I think there are really two separate use cases to consider:
> > 
> >   1. Providing snippets of script to Git to get Turing-complete behavior
> >      for existing Git features. For example, selecting commits during a
> >      traversal (e.g., a better "log --grep"), formatting output (e.g., a
> >      better "log --format" or "for-each-ref --format").
> > 
> >   2. Writing whole new git commands in a language that is quicker or
> >      easier to develop in than C.
> 
> That's good analysis.  I agree with your use-case split, I guess I'm just not
> very aware of the places in git where (1) is important.

Yeah, I don't think (1) is your use case at all. But when people talk
about "Jeff's lua experiment", they are talking about some patches I had
to do (1), which covered "log --format" (but ultimately would need more
cleanup to be acceptable upstream). Maybe that clears up the discussion
a little bit.

-Peff

Re: Python extension commands in git - request for policy change

[Eric S. Raymond]

Patrick Donnelly <batrick@batbytes.com>:
> On Tue, Dec 11, 2012 at 10:30 PM, Eric S. Raymond <esr@thyrsus.com> wrote:
> > It might be a good fit for extending git; I wouldn't be very surprised if
> > that worked. However, I do have concerns about the "Oh, we'll just
> > lash together a binding to C" attitude common among lua programmers; I
> > foresee maintainability problems and the possibility of slow death by
> > low-level details as that strategy tries to scale up.
> 
> I think this is quite a prediction? Could you give an example
> scenario?

Everything old is new again.  I'm going by experience with Tcl back in the day.

>        How would another language (e.g. Python) mitigate this?

The way you mitigate this sort of problem is to have a good set of
high-level bindings for standard services (like socket I/O) built in
your extension language and using its abstractions, so you don't get a
proliferation of low-level semi-custom APIs for doing the same stuff.

I have elsewhere referred to this as "the harsh lesson of Perl", which
I do not love but which was the first scripting language to get this
right.  There is a reason Tcl and a couple of earlier designs like csh
that we would now call "scripting languages" were displaced by Python
and Perl; this is it.

My favorite present-day example of getting this right is the Python bindings
for GTK.  They're lovely.  A work of art.

> I don't see how these languages are more appropriate based on your concerns.

Your previous exchange with Jeff King indicates that you don't
understand glue scripting very well.  Your puzzlement here just
confirms that.  Trust both of us on this, it's important.  And
reread my previous three paragraphs.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Python extension commands in git - request for policy change

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> I think there are really two separate use cases to consider:
>
>   1. Providing snippets of script to Git to get Turing-complete behavior
>      for existing Git features. For example, selecting commits during a
>      traversal (e.g., a better "log --grep"), formatting output (e.g., a
>      better "log --format" or "for-each-ref --format").
>
>   2. Writing whole new git commands in a language that is quicker or
>      easier to develop in than C.
>
> I think (1) is a good match for lua....
>
> But for (2), you are going to care a lot more about the language and its
> ecosystem (because you'll be interacting more with the world outside of
> git), and about having bindings to lots of different parts of git
> (because you'll want to do more interesting things than just examine a
> few data structures).

Good summary.  We also need to realize that adding a native
subcommand written in C has become much easier over time as our
internal API has evolved and matured.  These days, we still do a
whole new command in scripts (and we have whole commands still in
scripts) not because "quicker or easier to develop" (that is still
true for throw-away experiments) but primarily because that is
easier to modify and experiment over time until we find a solid
design.

Among the more important subcommands that are still scripts, I think
"add -i", "repack", "pull", "stash" and possibly "rebase" have
interfaces facing to both the end users and to the core part
solidified enough that they can now be ported to C.  Others either
are not important enough or still have rooms to improve the
interfaces in either direction that it would still be better to
leave them in scripts (e.g. "bisect" with "<used-to-be, now-is> vs
<good, bad>" issue unsettled, "submodule" with "floating" issue
unsettled, etc.).

Re: Python extension commands in git - request for policy change

[Andrew Ardill]

On 13 December 2012 04:49, Junio C Hamano <gitster@pobox.com> wrote:
> "bisect" with "<used-to-be, now-is> vs
> <good, bad>" issue unsettled

Would you want to see this issue resolved in-script before a porting
attempt was started?

Regards,

Andrew Ardill

Re: Python extension commands in git - request for policy change

[Junio C Hamano]

Andrew Ardill <andrew.ardill@gmail.com> writes:

> On 13 December 2012 04:49, Junio C Hamano <gitster@pobox.com> wrote:
>> "bisect" with "<used-to-be, now-is> vs
>> <good, bad>" issue unsettled
>
> Would you want to see this issue resolved in-script before a porting
> attempt was started?

Honestly, I do not care too much either way, but for the people who
want to work either on the rewrite-to-C or on the semantics issue,
it would be easier to manage it that way.

And that "issue resolved in-script" does not have to be "implemented
in-script".  The resolution could be to declare that it is not worth
it and a promise to call the two states <good, bad> and with no
other names.  It would give a semantics for the rewriters-to-C can
start working on that is stable enough ;-).

Re: Python extension commands in git - request for policy change

[Patrick Donnelly]

On Wed, Dec 12, 2012 at 7:43 AM, Eric S. Raymond <esr@thyrsus.com> wrote:
> Patrick Donnelly <batrick@batbytes.com>:
>>        How would another language (e.g. Python) mitigate this?
>
> The way you mitigate this sort of problem is to have a good set of
> high-level bindings for standard services (like socket I/O) built in
> your extension language and using its abstractions, so you don't get a
> proliferation of low-level semi-custom APIs for doing the same stuff.
>
> I have elsewhere referred to this as "the harsh lesson of Perl", which
> I do not love but which was the first scripting language to get this
> right.  There is a reason Tcl and a couple of earlier designs like csh
> that we would now call "scripting languages" were displaced by Python
> and Perl; this is it.

Okay, I understand what you were trying to say earlier.

I'm not going to say Lua is a silver bullet for all embedded language
needs. If you seriously need an exotic suite of libraries built into
the language, then Lua is not really going to work well for you. In
reality though, many projects that require an extension language do
not need all the system programming facilities thrown in. In fact,
many don't want them due to bloat or security considerations. So, you
take on a hyperopic viewpoint by ruling out Lua simply because it
lacks a suite of system libraries.

With Jeff's response:

> As for "interacting with the outside world", I was specifically thinking
> of stuff like git-send-email (currently in perl) and git-imap-send
> (written in C). They need to open network sockets and speak standard
> protocols. I suspect Lua would need a module or custom bindings to do
> the former at all, and certainly the code would be much simpler if we
> re-used standard modules for speaking SMTP and IMAP (which of course
>increases our dependencies again...).

I would think this can perhaps be exported into another script Lua
could exec as needed. Or luasocket may be sufficient. These
dependencies would need to be examined in detail. I wouldn't recommend
selecting a language because of one odd network protocol dependency
satisfied by an obscure built-in library (I realize Jeff's example was
exactly that, an example).

On Wed, Dec 12, 2012 at 7:43 AM, Eric S. Raymond <esr@thyrsus.com> wrote:
>> I don't see how these languages are more appropriate based on your concerns.
>
> Your previous exchange with Jeff King indicates that you don't
> understand glue scripting very well.  Your puzzlement here just
> confirms that.  Trust both of us on this, it's important.  And
> reread my previous three paragraphs.

What I didn't understand coming into this thread was Git's ecosystem.
I understand embedded scripting languages very well and have been
working with Lua for years.

What does puzzle me is your dismissal of Lua because it doesn't have
the library suite Python does. Lua is not a system programming
language and I could argue Python is not really an embedded language.
I came here to try to stimulate discussion about what Git actually
needs/wants from a higher level language. If a small embedded language
would fit well, the Lua should be a candidate for consideration.

--
- Patrick Donnelly