Re: Fwd: [GSOC][Proposal] Move existing tests to a unit testing framework

[Patrick Steinhardt <ps@pks.im>]

[-- Attachment #1: Type: text/plain, Size: 11210 bytes --]

On Wed, Mar 27, 2024 at 08:06:06PM +0530, Sanchit Jindal wrote:
> Hello,
> 
> Please find my proposal for GSOC 2024 for the project "Move existing
> tests to a unit testing framework" at
> https://docs.google.com/document/d/1yWP5GFRksdQEOKtYIEXbNSVb1qlJ2szRvpTcXXVvgTk/edit?usp=sharing.
> I am also adding the text version for convenience.

Thanks a lot for your proposal!

> Thank You,
> Sanchit Jindal
> --------------------
> 
> Google Summer Of Code Proposal
> 
> GIT (Move existing tests to a unit testing framework)

Sorry to be nitpicky, but it's spelled "Git", not "GIT" :)

> ### Personal Information
> 
> Name:          Sanchit Jindal
> Email:         sanchit1053@gmail.com
> Qualification: B.Tech in Computer Science and Engineering
> 
> 
> Greetings,
> 
> My name is Sanchit Jindal, currently pursuing my B.E. in Computer
> Science at the Indian Institute of Technology, Bombay, I will complete
> my degree in May 2024. I am writing to put forward my proposal for the
> Git Project to transition the unit tests to the new Unit Testing
> Framework.
> 
> You can find my Resume at
> https://www.linkedin.com/in/sanchit-jindal-0a953621a/ in the featured
> section

I would recommend to include highlights of your resume in the proposal
directly, either instead or in addition to linking to it. Not everyone
has a LinkedIn account (I don't), and thus I'm unable to view most of
your profile.

If you already did that, then it might be worthwhile to point out that
you did include highlights already. Otherwise I'm left wondering whether
I miss out on any important information.

> Throughout my academic journey and internships, I've gained valuable
> experience across diverse domains, ranging from machine learning to
> fundamental system components like compilers and operating systems.
> This breadth of exposure has honed my ability to swiftly grasp new
> concepts and adapt to different codebases. Proficient in both C and
> C++, I am well-equipped to contribute effectively to the project.
> 
> For this GSOC project, I aspire to make meaningful contributions to
> the open-source community. My interest in software development drives
> me to create solutions that will offer utility to others.
> 
> 
> ### Overview
> 
> Proposed Abstract
> Git has a lot of test cases that need to be migrated to use a new unit
> testing framework. This typically involves moving code from both:
> * a “t/helper/test-*.c” test helper in C, and
> * a “t/*.sh” test script in shell that invokes the test helper
> 
> 
> over to a single “t/unit-tests/t-*.c” in C using the unit testing framework.

I guess these formatting issues here come from the fact that you copied
the text over from the linked doc? Not a huge issue in that case.

> ### Details
> 
> The Project entails porting the current testing framework (which is a
> collection of helper files written in c that provide various
> utilities, and various shell scripts) to using a new Unit-Testing
> Library. The shell scripts in the original code base setup the
> environment and , using the helper functions, test the functionalities
> provided by git. The new framework utilises the test-lib library that

s/git/Git

> will combine the functionality of both the helper functions and the
> shell scripts into one unit-test.
> Advantages of this approach
> * Unit Tests are a great way to check any program and to test all
> possible corner cases and functions
> * Using this new testing framework will provide a more modular
> approach to unit testing, allowing to create more extensive tests, and
> to provide better debugging features. Such as better debug Messages or

s/Messages/messages

> a better CLI to choose and run tests.
> * The tests will also improve the runtime as instead of running
> multiple process (with each use of shell commands), the program can
> compile and run a single process
> * The approach will also future proof the unit testing as having a c
> library will provide a better base than shell scripts to test
> libraries and other internal features

I wouldn't necessarily say "better" here. It's not like the unit tests
are expected to eventually replace all ouf our shell script tests. Those
test suites serve different purposes, and in general it is recommended
to continue writing shell script tests. They exercise the code at the
user level and thus directly mirror the experience a user would have,
which is generally more helpful than only asserting that one small part
of the codebase works. You can kind of think of these as integration or
end-to-end tests

So the unit testing framework isn't necessarily better or even preferred
in the general case. But it does allow us to exercise low-level
functionality or specific edge cases that were previously cumbersome to
test.

So I wouldn't say "better", but rather that those unit tests address a
different need than the scripts.

> ### Basic Structure of the Changes
> 
> * For each file in the helper directory I will create a corresponding
> unit-test file. (Some files which are being used by multiple test can
> be ported later)

Not all files in the helper directory are actually unit tests. Some of
them are, some of them expose functionality that would otherwise not be
exposed by Git in a direct way. "test-tool refstore" comes to mind,
which allows you to munge the ref store in ways that git-update-ref(1)
wouldn't allow you to do. There are others.

Furthermore, it may not be realistic to port _all_ existing tests over
to the new functionality. It is perfectly fine to say that you plan to
only address a subset of unit tests, and you don't necessarily have to
say which ones exactly you want to port over now.

> * Each test in the script will be corresponding to a function that
> will be called in the format
> 
> '''
> TEST(funtion_name(), “SUCCESS MESSAGE”);
> '''
> 
> * The function body will contain the main implementation of the test
> corresponding to the c file ( The helper function will ideally be the
> same with no or minimal changes just to allow it to be called as
> required )

With "helper function" you probably mean the test logic, right? I guess
it depends on the function whether it would be ideal to have only
minimal changes. Given that we also convert them to use our new testing
framework I do expect that some of them may change significantly, which
wouldn't be all that bad, either.

> * Where the assert usually given at the end of the test will be made
> using the utility functions such as check_int, check_str and others,
> (or creating more function as they are required)
> * The Make files will be updated to remove the use of shell script
> with compiling and running the new c programs
> 
> ### Previous Contributions
> 
> I have worked on the micro-project
> Use test_path_is_* functions in test scripts
> 
> Thread Link: https://lore.kernel.org/git/b8d0620d4104106210ecf6a34ada591adf01cff8.1711049963.git.gitgitgadget@gmail.com/
> Status: Open
> 
> I had some difficulties in understanding the usage of gitgitGadget and
> git send-email which are being employed as the Pull request
> alternatives. Having tried gitgitGadget I am confident that I will be
> able to utilise it hereafter. I am having some trouble using git
> send-email with the network at my institute but I am optimistic that I
> will be able to use it after the completion of my degree.

While many folks use git-send-email(1) directly it is perfectly fine if
you don't and want to use GitGitGadget instead. As long as you are
familiar with one of those workflows you don't necessarily have to set
up the other one.

> With the guidance of Eric Sunshine and Junio C Hamano, I have iterated
> through different versions of the pull request, and familiarised
> myself with the git commit pattern and formalities
> 
> 
> ### Deliverables
> 
> The plan is to port the helper functions from helper directory to the
> unit-test directory along with the test defined in the shell scripts.
> Some of the helper tools which have a more basic functionality and are
> being used by multiple test scripts can be left untouched.
> As a stretch goal I will try to port the shell scripts to the new
> testing interface creating new functions for the library as required
> I also plan to keep documenting all the changes made and to keep in
> constant contact with the mentors regarding any bugs or reviews
> regarding the code.
> 
> 
> ### Availability
> 
> I am expecting to be free from my academic responsibilities by 4th
> May. I have no further commitments for the summer and will be able to
> provide about 40-50 hours of work per week. I am confident that I
> should be able to complete about 2-3 files each week depending on the
> files.

I would like to caution you a bit about the "2-3 files each week" goal.
The Git project is comparatively slow-moving, and it very often takes
multiple weeks for a patch series until it lands in the "master" branch.
Furthermore, the quality bar is quite high in the Git project, which may
require you to undergo several revisions for a patch series before it is
accepted.

So I would recommend to account for all these facts. It may make sense
to compare to below related work as an estimate of how much work you can
get done realistically.

> Related Work
> The project has been worked on by Achu Luma for the Outreachy Internship
> Test-ctype:
> https://lore.kernel.org/git/20240112102743.1440-1-ach.lumap@gmail.com/#t
> Test-advise:
> https://lore.kernel.org/git/20240112102122.1422-1-ach.lumap@gmail.com/
> Test-data:
> https://lore.kernel.org/git/20240205162506.1835-2-ach.lumap@gmail.com/
> Test-hash:
> https://lore.kernel.org/git/20240229054004.3807-2-ach.lumap@gmail.com/
> Test-strcmp-offset:
> https://lore.kernel.org/git/20240310144819.4379-1-ach.lumap@gmail.com/
> 
> 
> Another testcase has also been handled by Ghanshyam Thakkar
> Test–oid-array:
> https://lore.kernel.org/git/20240223193257.9222-1-shyamthakkar001@gmail.com/
> 
> 
> 
> 
> Timeline (Tentative)
> 
> Community Bonding
> (1 May- 26 May)
> Be in contact with mentors and figure out the best way to migrate the
> test-cases, Familiarise myself with the library and previous work done
> during the outreachy program.
> 
> Phase I
> (27 May - 11 July)
> Begin tackling the harder or longer testcases that will require more
> knowledge of the implementation
> 
> Phase II
> (12 July - 18 Aug)
> Keep working on the testcases following the mentors feedback

Note that you'll not only get feedback from mentors, but also from the
community.

> Final Week
> (19 Aug - 26 Aug)
> Finish up the remaining testcases, fixing any bugs that may be
> discovered in the earlier implementations
> 
> ### Acknowledgement
> I would like to thank Eric Sunshine and Junio C Hamano for helping me
> with the microproject and for the guidance on the gitgitGadget and git
> send-email features, And also helping me get acquainted with the git
> PR guidelines.

Patrick

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]