[ruby-core:89842] [Ruby trunk Bug#15313] [PATCH] Let debuggers know when a tail call happens

[ruby-core:89842] [Ruby trunk Bug#15313] [PATCH] Let debuggers know when a tail call happens

[alanwucanada]

Issue #15313 has been reported by alanwu (Alan Wu).

----------------------------------------
Bug #15313: [PATCH] Let debuggers know when a tail call happens
https://bugs.ruby-lang.org/issues/15313

* Author: alanwu (Alan Wu)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
* ruby -v: 
* Backport: 2.3: UNKNOWN, 2.4: UNKNOWN, 2.5: UNKNOWN
----------------------------------------
## Background
The popular debugger "byebug" relies on tracepoint events to implement a few core functionality such as "next".
The "next" command needs to pause execution when the VM reaches the next line on the same stack frame. As there is no direct way, using the public
API, to tell the execution progress of a frame, Byebug keeps track of the currently executing frame by maintaining a counter using the "call" and "return"
tracepoint event. https://github.com/deivid-rodriguez/byebug/blob/58ee5114aa856ec49483532a86fd159a877dd6ab/ext/byebug/byebug.c#L162-L170

Byebug's counter becomes incorrect when the interpreter performs a tail call, since after a tail call, the "return" event doesn't fire for the caller. #15303
This cause the "next" command to misbehave when stepping over code that performs tail call. https://github.com/deivid-rodriguez/byebug/issues/481

## Proposed solution
I implemented a new method in TracePoint that lets Byebug, or any other debugger to differentiate between a normal method call and a tail call. Using this API,
Byebug can maintain the counter properly in the event of a tail call.

Here are some other solutions
  - Some sort of public api that exposes execution progress of control frames. This might take a lot of effort.
  - Some sort of public api that give ids to stack frames. After a tail call, debuggers can use this to tell that a frame is different, even though it's on the same location on the stack.
  - Turning off tail call optimization in Forwardable. This side steps the problem but the core issue still remain. Third party libraries could still use tco
     and byebug wouldn't be able to work with them.

Here are some downsides to my solution:
  - This is a pretty niche feature that only debuggers want
  - I'm adding a vm frame flag. While we have a few more bits to play with, it is a limited resource.




---Files--------------------------------
0001-Add-a-tail-call-predicate-for-TracePoint.patch (18.1 KB)


-- 
https://bugs.ruby-lang.org/

[ruby-core:89928] [Ruby trunk Bug#15313] [PATCH] Let debuggers know when a tail call happens

[ko1]

Issue #15313 has been updated by ko1 (Koichi Sasada).

Assignee set to ko1 (Koichi Sasada)

fix #15303 (invoke return event) doesn't solve it?

BTW, if you provide an API, sample code will help us to understand.


----------------------------------------
Bug #15313: [PATCH] Let debuggers know when a tail call happens
https://bugs.ruby-lang.org/issues/15313#change-75011

* Author: alanwu (Alan Wu)
* Status: Open
* Priority: Normal
* Assignee: ko1 (Koichi Sasada)
* Target version: 
* ruby -v: 
* Backport: 2.3: UNKNOWN, 2.4: UNKNOWN, 2.5: UNKNOWN
----------------------------------------
## Background
The popular debugger "byebug" relies on tracepoint events to implement a few core functionality such as "next".
The "next" command needs to pause execution when the VM reaches the next line on the same stack frame. As there is no direct way, using the public
API, to tell the execution progress of a frame, Byebug keeps track of the currently executing frame by maintaining a counter using the "call" and "return"
tracepoint event. https://github.com/deivid-rodriguez/byebug/blob/58ee5114aa856ec49483532a86fd159a877dd6ab/ext/byebug/byebug.c#L162-L170

Byebug's counter becomes incorrect when the interpreter performs a tail call, since after a tail call, the "return" event doesn't fire for the caller. #15303
This causes the "next" command to misbehave when stepping over code that performs tail call. https://github.com/deivid-rodriguez/byebug/issues/481

## Proposed solution
I implemented a new method in TracePoint that lets Byebug, or any other debugger to differentiate between a normal method call and a tail call. Using this API,
Byebug can maintain the counter properly in the event of a tail call.

Here are some other solutions
  - Some sort of public api that exposes execution progress of control frames. This might take a lot of effort.
  - Some sort of public api that give ids to stack frames. After a tail call, debuggers can use this to tell that a frame is different, even though it's on the same location on the stack.
  - Turning off tail call optimization in Forwardable. This side steps the problem but the core issue still remain. Third party libraries could still use tco
     and byebug wouldn't be able to work with them.

Here are some downsides to my solution:
  - This is a pretty niche feature that only debuggers want
  - I'm adding a vm frame flag. While we have a few more bits to play with, it is a limited resource.




---Files--------------------------------
0001-Add-a-tail-call-predicate-for-TracePoint.patch (18.1 KB)


-- 
https://bugs.ruby-lang.org/

[ruby-core:89936] [Ruby trunk Bug#15313] [PATCH] Let debuggers know when a tail call happens

[alanwucanada]

Issue #15313 has been updated by alanwu (Alan Wu).


Yes if the return events can fire as usual even when tail calls happen it would be perfect and would fix #15303.

I don't see a way to fire the return events in the normal order without extra allocations, though. Maybe it's possible? 

We could fire the return event before the tailcall happens, #return_value method would be confusing though. 

> BTW, if you provide an API, sample code will help us to understand.

Thank you for letting me know. The patch has a spec in it and I thought that was enough. Next time, I will remember to post examples on the tracker too. 


----------------------------------------
Bug #15313: [PATCH] Let debuggers know when a tail call happens
https://bugs.ruby-lang.org/issues/15313#change-75025

* Author: alanwu (Alan Wu)
* Status: Open
* Priority: Normal
* Assignee: ko1 (Koichi Sasada)
* Target version: 
* ruby -v: 
* Backport: 2.3: UNKNOWN, 2.4: UNKNOWN, 2.5: UNKNOWN
----------------------------------------
## Background
The popular debugger "byebug" relies on tracepoint events to implement a few core functionality such as "next".
The "next" command needs to pause execution when the VM reaches the next line on the same stack frame. As there is no direct way, using the public
API, to tell the execution progress of a frame, Byebug keeps track of the currently executing frame by maintaining a counter using the "call" and "return"
tracepoint event. https://github.com/deivid-rodriguez/byebug/blob/58ee5114aa856ec49483532a86fd159a877dd6ab/ext/byebug/byebug.c#L162-L170

Byebug's counter becomes incorrect when the interpreter performs a tail call, since after a tail call, the "return" event doesn't fire for the caller. #15303
This causes the "next" command to misbehave when stepping over code that performs tail call. https://github.com/deivid-rodriguez/byebug/issues/481

## Proposed solution
I implemented a new method in TracePoint that lets Byebug, or any other debugger to differentiate between a normal method call and a tail call. Using this API,
Byebug can maintain the counter properly in the event of a tail call.

Here are some other solutions
  - Some sort of public api that exposes execution progress of control frames. This might take a lot of effort.
  - Some sort of public api that give ids to stack frames. After a tail call, debuggers can use this to tell that a frame is different, even though it's on the same location on the stack.
  - Turning off tail call optimization in Forwardable. This side steps the problem but the core issue still remain. Third party libraries could still use tco
     and byebug wouldn't be able to work with them.

Here are some downsides to my solution:
  - This is a pretty niche feature that only debuggers want
  - I'm adding a vm frame flag. While we have a few more bits to play with, it is a limited resource.




---Files--------------------------------
0001-Add-a-tail-call-predicate-for-TracePoint.patch (18.1 KB)


-- 
https://bugs.ruby-lang.org/

[ruby-core:89981] [Ruby trunk Bug#15313] [PATCH] Let debuggers know when a tail call happens

[alanwucanada]

Issue #15313 has been updated by alanwu (Alan Wu).

File 0001-Add-a-tail-call-predicate-for-TracePoint.patch added
File deleted (0001-Add-a-tail-call-predicate-for-TracePoint.patch)

I noticed that the original patch doesn't compile with GCC.
-Werror=declaration-after-statement doesn't seem to have any effect on Clang: https://godbolt.org/z/8Qykk4 (switch to gcc to see it fail to compile. It compiles on Clang)



----------------------------------------
Bug #15313: [PATCH] Let debuggers know when a tail call happens
https://bugs.ruby-lang.org/issues/15313#change-75087

* Author: alanwu (Alan Wu)
* Status: Open
* Priority: Normal
* Assignee: ko1 (Koichi Sasada)
* Target version: 
* ruby -v: 
* Backport: 2.3: UNKNOWN, 2.4: UNKNOWN, 2.5: UNKNOWN
----------------------------------------
## Background
The popular debugger "byebug" relies on tracepoint events to implement a few core functionality such as "next".
The "next" command needs to pause execution when the VM reaches the next line on the same stack frame. As there is no direct way, using the public
API, to tell the execution progress of a frame, Byebug keeps track of the currently executing frame by maintaining a counter using the "call" and "return"
tracepoint event. https://github.com/deivid-rodriguez/byebug/blob/58ee5114aa856ec49483532a86fd159a877dd6ab/ext/byebug/byebug.c#L162-L170

Byebug's counter becomes incorrect when the interpreter performs a tail call, since after a tail call, the "return" event doesn't fire for the caller. #15303
This causes the "next" command to misbehave when stepping over code that performs tail call. https://github.com/deivid-rodriguez/byebug/issues/481

## Proposed solution
I implemented a new method in TracePoint that lets Byebug, or any other debugger to differentiate between a normal method call and a tail call. Using this API,
Byebug can maintain the counter properly in the event of a tail call.

Here are some other solutions
  - Some sort of public api that exposes execution progress of control frames. This might take a lot of effort.
  - Some sort of public api that give ids to stack frames. After a tail call, debuggers can use this to tell that a frame is different, even though it's on the same location on the stack.
  - Turning off tail call optimization in Forwardable. This side steps the problem but the core issue still remain. Third party libraries could still use tco
     and byebug wouldn't be able to work with them.

Here are some downsides to my solution:
  - This is a pretty niche feature that only debuggers want
  - I'm adding a vm frame flag. While we have a few more bits to play with, it is a limited resource.




---Files--------------------------------
0001-Add-a-tail-call-predicate-for-TracePoint.patch (18.1 KB)


-- 
https://bugs.ruby-lang.org/

[ruby-core:90296] [Ruby trunk Bug#15313] [PATCH] Let debuggers know when a tail call happens

[alanwucanada]

Issue #15313 has been updated by alanwu (Alan Wu).


Could I ask for some feedback on this?

----------------------------------------
Bug #15313: [PATCH] Let debuggers know when a tail call happens
https://bugs.ruby-lang.org/issues/15313#change-75403

* Author: alanwu (Alan Wu)
* Status: Open
* Priority: Normal
* Assignee: ko1 (Koichi Sasada)
* Target version: 
* ruby -v: 
* Backport: 2.3: UNKNOWN, 2.4: UNKNOWN, 2.5: UNKNOWN
----------------------------------------
## Background
The popular debugger "byebug" relies on tracepoint events to implement a few core functionality such as "next".
The "next" command needs to pause execution when the VM reaches the next line on the same stack frame. As there is no direct way, using the public
API, to tell the execution progress of a frame, Byebug keeps track of the currently executing frame by maintaining a counter using the "call" and "return"
tracepoint event. https://github.com/deivid-rodriguez/byebug/blob/58ee5114aa856ec49483532a86fd159a877dd6ab/ext/byebug/byebug.c#L162-L170

Byebug's counter becomes incorrect when the interpreter performs a tail call, since after a tail call, the "return" event doesn't fire for the caller. #15303
This causes the "next" command to misbehave when stepping over code that performs tail call. https://github.com/deivid-rodriguez/byebug/issues/481

## Proposed solution
I implemented a new method in TracePoint that lets Byebug, or any other debugger to differentiate between a normal method call and a tail call. Using this API,
Byebug can maintain the counter properly in the event of a tail call.

Here are some other solutions
  - Some sort of public api that exposes execution progress of control frames. This might take a lot of effort.
  - Some sort of public api that give ids to stack frames. After a tail call, debuggers can use this to tell that a frame is different, even though it's on the same location on the stack.
  - Turning off tail call optimization in Forwardable. This side steps the problem but the core issue still remain. Third party libraries could still use tco
     and byebug wouldn't be able to work with them.

Here are some downsides to my solution:
  - This is a pretty niche feature that only debuggers want
  - I'm adding a vm frame flag. While we have a few more bits to play with, it is a limited resource.




---Files--------------------------------
0001-Add-a-tail-call-predicate-for-TracePoint.patch (18.1 KB)


-- 
https://bugs.ruby-lang.org/

[ruby-core:90301] [Ruby trunk Bug#15313] [PATCH] Let debuggers know when a tail call happens

[ko1]

Issue #15313 has been updated by ko1 (Koichi Sasada).


how about https://bugs.ruby-lang.org/issues/15303#note-2 ?


----------------------------------------
Bug #15313: [PATCH] Let debuggers know when a tail call happens
https://bugs.ruby-lang.org/issues/15313#change-75408

* Author: alanwu (Alan Wu)
* Status: Open
* Priority: Normal
* Assignee: ko1 (Koichi Sasada)
* Target version: 
* ruby -v: 
* Backport: 2.3: UNKNOWN, 2.4: UNKNOWN, 2.5: UNKNOWN
----------------------------------------
## Background
The popular debugger "byebug" relies on tracepoint events to implement a few core functionality such as "next".
The "next" command needs to pause execution when the VM reaches the next line on the same stack frame. As there is no direct way, using the public
API, to tell the execution progress of a frame, Byebug keeps track of the currently executing frame by maintaining a counter using the "call" and "return"
tracepoint event. https://github.com/deivid-rodriguez/byebug/blob/58ee5114aa856ec49483532a86fd159a877dd6ab/ext/byebug/byebug.c#L162-L170

Byebug's counter becomes incorrect when the interpreter performs a tail call, since after a tail call, the "return" event doesn't fire for the caller. #15303
This causes the "next" command to misbehave when stepping over code that performs tail call. https://github.com/deivid-rodriguez/byebug/issues/481

## Proposed solution
I implemented a new method in TracePoint that lets Byebug, or any other debugger to differentiate between a normal method call and a tail call. Using this API,
Byebug can maintain the counter properly in the event of a tail call.

Here are some other solutions
  - Some sort of public api that exposes execution progress of control frames. This might take a lot of effort.
  - Some sort of public api that give ids to stack frames. After a tail call, debuggers can use this to tell that a frame is different, even though it's on the same location on the stack.
  - Turning off tail call optimization in Forwardable. This side steps the problem but the core issue still remain. Third party libraries could still use tco
     and byebug wouldn't be able to work with them.

Here are some downsides to my solution:
  - This is a pretty niche feature that only debuggers want
  - I'm adding a vm frame flag. While we have a few more bits to play with, it is a limited resource.




---Files--------------------------------
0001-Add-a-tail-call-predicate-for-TracePoint.patch (18.1 KB)


-- 
https://bugs.ruby-lang.org/