[PATCH 0/2] cleanup compat/ sources using spatch

[PATCH 0/2] cleanup compat/ sources using spatch

[Denton Liu]

Sources under compat/ were excluded from cleanup using spatch since some
of them were taken from an upstream and we didn't want to modify them so
that merging in future updates would be simpler. However, most of the
files in compat/ were created by us so it should be safe to apply
cleanup to those files.


Denton Liu (2):
  mingw: apply array.cocci rule
  compat/*.[ch]: remove extern from function declarations using spatch

 compat/mingw.c         | 4 ++--
 compat/mingw.h         | 6 +++---
 compat/win32/pthread.h | 6 +++---
 3 files changed, 8 insertions(+), 8 deletions(-)

-- 
2.23.0.248.g3a9dd8fb08

[PATCH 1/2] mingw: apply array.cocci rule

[Denton Liu]

After running Coccinelle on all sources inside compat/ that were created
by us[1], it was found that compat/mingw.c violated an array.cocci rule
in two places and, thus, a patch was generated. Apply this patch so that
all compat/ sources created by us follows all cocci rules.

[1]: Do not run Coccinelle on files that are taken from some upstream
because in case we need to pull updates from them, we would like to have
diverged as little as possible in order to make merging updates simpler.

The following sources were determined to have been taken from some
upstream:

* compat/regex/
* compat/inet_ntop.c
* compat/inet_pton.c
* compat/nedmalloc/
* compat/obstack.{c,h}
* compat/poll/

Signed-off-by: Denton Liu <liu.denton@gmail.com>
---
 compat/mingw.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/compat/mingw.c b/compat/mingw.c
index 738f0a826a..a3b1e9e3bb 100644
--- a/compat/mingw.c
+++ b/compat/mingw.c
@@ -1265,7 +1265,7 @@ static wchar_t *make_environment_block(char **deltaenv)
 		}
 
 		ALLOC_ARRAY(result, size);
-		memcpy(result, wenv, size * sizeof(*wenv));
+		COPY_ARRAY(result, wenv, size);
 		FreeEnvironmentStringsW(wenv);
 		return result;
 	}
@@ -1309,7 +1309,7 @@ static wchar_t *make_environment_block(char **deltaenv)
 			continue;
 
 		size = wcslen(array[i]) + 1;
-		memcpy(p, array[i], size * sizeof(*p));
+		COPY_ARRAY(p, array[i], size);
 		p += size;
 	}
 	*p = L'\0';
-- 
2.23.0.248.g3a9dd8fb08

[PATCH 2/2] compat/*.[ch]: remove extern from function declarations using spatch

[Denton Liu]

In 554544276a (*.[ch]: remove extern from function declarations using
spatch, 2019-04-29), we removed externs from function declarations using
spatch but we intentionally excluded files under compat/ since some are
directly copied from an upstream and we should avoid churning them so
that manually merging future updates will be simpler.

In the last commit, we determined the files which taken from an upstream
so we can exclude them and run spatch on the remainder.

This was the Coccinelle patch used:

	@@
	type T;
	identifier f;
	@@
	- extern
	  T f(...);

and it was run with:

	$ git ls-files compat/\*\*.{c,h} |
		xargs spatch --sp-file contrib/coccinelle/noextern.cocci --in-place
	$ git checkout -- \
		compat/regex/ \
		compat/inet_ntop.c \
		compat/inet_pton.c \
		compat/nedmalloc/ \
		compat/obstack.{c,h} \
		compat/poll/

Coccinelle has some trouble dealing with `__attribute__` and varargs so
we ran the following to ensure that no remaining changes were left
behind:

	$ git ls-files compat/\*\*.{c,h} |
		xargs sed -i'' -e 's/^\(\s*\)extern \([^(]*([^*]\)/\1\2/'
	$ git checkout -- \
		compat/regex/ \
		compat/inet_ntop.c \
		compat/inet_pton.c \
		compat/nedmalloc/ \
		compat/obstack.{c,h} \
		compat/poll/

Signed-off-by: Denton Liu <liu.denton@gmail.com>
---
 compat/mingw.h         | 6 +++---
 compat/win32/pthread.h | 6 +++---
 2 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/compat/mingw.h b/compat/mingw.h
index a03e40e6e2..9ad204c57c 100644
--- a/compat/mingw.h
+++ b/compat/mingw.h
@@ -11,7 +11,7 @@ typedef _sigset_t sigset_t;
 #undef _POSIX_THREAD_SAFE_FUNCTIONS
 #endif
 
-extern int mingw_core_config(const char *var, const char *value, void *cb);
+int mingw_core_config(const char *var, const char *value, void *cb);
 #define platform_core_config mingw_core_config
 
 /*
@@ -443,7 +443,7 @@ static inline void convert_slashes(char *path)
 			*path = '/';
 }
 #define PATH_SEP ';'
-extern char *mingw_query_user_email(void);
+char *mingw_query_user_email(void);
 #define query_user_email mingw_query_user_email
 #if !defined(__MINGW64_VERSION_MAJOR) && (!defined(_MSC_VER) || _MSC_VER < 1800)
 #define PRIuMAX "I64u"
@@ -580,4 +580,4 @@ int main(int argc, const char **argv);
 /*
  * Used by Pthread API implementation for Windows
  */
-extern int err_win_to_posix(DWORD winerr);
+int err_win_to_posix(DWORD winerr);
diff --git a/compat/win32/pthread.h b/compat/win32/pthread.h
index c6cb8dd219..f1cfe73de9 100644
--- a/compat/win32/pthread.h
+++ b/compat/win32/pthread.h
@@ -50,7 +50,7 @@ typedef struct {
 	DWORD tid;
 } pthread_t;
 
-extern int pthread_create(pthread_t *thread, const void *unused,
+int pthread_create(pthread_t *thread, const void *unused,
 			  void *(*start_routine)(void*), void *arg);
 
 /*
@@ -59,10 +59,10 @@ extern int pthread_create(pthread_t *thread, const void *unused,
  */
 #define pthread_join(a, b) win32_pthread_join(&(a), (b))
 
-extern int win32_pthread_join(pthread_t *thread, void **value_ptr);
+int win32_pthread_join(pthread_t *thread, void **value_ptr);
 
 #define pthread_equal(t1, t2) ((t1).tid == (t2).tid)
-extern pthread_t pthread_self(void);
+pthread_t pthread_self(void);
 
 static inline void NORETURN pthread_exit(void *ret)
 {
-- 
2.23.0.248.g3a9dd8fb08

Re: [PATCH 1/2] mingw: apply array.cocci rule

[Johannes Schindelin]

Hi Denton,

On Wed, 4 Sep 2019, Denton Liu wrote:

> After running Coccinelle on all sources inside compat/ that were created
> by us[1], it was found that compat/mingw.c violated an array.cocci rule
> in two places and, thus, a patch was generated. Apply this patch so that
> all compat/ sources created by us follows all cocci rules.
>
> [1]: Do not run Coccinelle on files that are taken from some upstream
> because in case we need to pull updates from them, we would like to have
> diverged as little as possible in order to make merging updates simpler.
>
> The following sources were determined to have been taken from some
> upstream:
>
> * compat/regex/
> * compat/inet_ntop.c
> * compat/inet_pton.c
> * compat/nedmalloc/
> * compat/obstack.{c,h}
> * compat/poll/
>
> Signed-off-by: Denton Liu <liu.denton@gmail.com>
> ---
>  compat/mingw.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/compat/mingw.c b/compat/mingw.c
> index 738f0a826a..a3b1e9e3bb 100644
> --- a/compat/mingw.c
> +++ b/compat/mingw.c
> @@ -1265,7 +1265,7 @@ static wchar_t *make_environment_block(char **deltaenv)
>  		}
>
>  		ALLOC_ARRAY(result, size);
> -		memcpy(result, wenv, size * sizeof(*wenv));
> +		COPY_ARRAY(result, wenv, size);
>  		FreeEnvironmentStringsW(wenv);
>  		return result;
>  	}
> @@ -1309,7 +1309,7 @@ static wchar_t *make_environment_block(char **deltaenv)
>  			continue;
>
>  		size = wcslen(array[i]) + 1;
> -		memcpy(p, array[i], size * sizeof(*p));
> +		COPY_ARRAY(p, array[i], size);

ACK!

Thanks,
Dscho

>  		p += size;
>  	}
>  	*p = L'\0';
> --
> 2.23.0.248.g3a9dd8fb08
>
>

Re: [PATCH 2/2] compat/*.[ch]: remove extern from function declarations using spatch

[Johannes Schindelin]

Hi Denton,

On Wed, 4 Sep 2019, Denton Liu wrote:

> In 554544276a (*.[ch]: remove extern from function declarations using
> spatch, 2019-04-29), we removed externs from function declarations using
> spatch but we intentionally excluded files under compat/ since some are
> directly copied from an upstream and we should avoid churning them so
> that manually merging future updates will be simpler.
>
> In the last commit, we determined the files which taken from an upstream
> so we can exclude them and run spatch on the remainder.
>
> This was the Coccinelle patch used:
>
> 	@@
> 	type T;
> 	identifier f;
> 	@@
> 	- extern
> 	  T f(...);
>
> and it was run with:
>
> 	$ git ls-files compat/\*\*.{c,h} |
> 		xargs spatch --sp-file contrib/coccinelle/noextern.cocci --in-place
> 	$ git checkout -- \
> 		compat/regex/ \
> 		compat/inet_ntop.c \
> 		compat/inet_pton.c \
> 		compat/nedmalloc/ \
> 		compat/obstack.{c,h} \
> 		compat/poll/
>
> Coccinelle has some trouble dealing with `__attribute__` and varargs so
> we ran the following to ensure that no remaining changes were left
> behind:
>
> 	$ git ls-files compat/\*\*.{c,h} |
> 		xargs sed -i'' -e 's/^\(\s*\)extern \([^(]*([^*]\)/\1\2/'
> 	$ git checkout -- \
> 		compat/regex/ \
> 		compat/inet_ntop.c \
> 		compat/inet_pton.c \
> 		compat/nedmalloc/ \
> 		compat/obstack.{c,h} \
> 		compat/poll/

I wonder whether we want to make this part of the (slightly misnamed)
"Static Analysis" job in our CI.

>
> Signed-off-by: Denton Liu <liu.denton@gmail.com>
> ---
>  compat/mingw.h         | 6 +++---
>  compat/win32/pthread.h | 6 +++---
>  2 files changed, 6 insertions(+), 6 deletions(-)
>
> diff --git a/compat/mingw.h b/compat/mingw.h
> index a03e40e6e2..9ad204c57c 100644
> --- a/compat/mingw.h
> +++ b/compat/mingw.h
> @@ -11,7 +11,7 @@ typedef _sigset_t sigset_t;
>  #undef _POSIX_THREAD_SAFE_FUNCTIONS
>  #endif
>
> -extern int mingw_core_config(const char *var, const char *value, void *cb);
> +int mingw_core_config(const char *var, const char *value, void *cb);
>  #define platform_core_config mingw_core_config
>
>  /*
> @@ -443,7 +443,7 @@ static inline void convert_slashes(char *path)
>  			*path = '/';
>  }
>  #define PATH_SEP ';'
> -extern char *mingw_query_user_email(void);
> +char *mingw_query_user_email(void);
>  #define query_user_email mingw_query_user_email
>  #if !defined(__MINGW64_VERSION_MAJOR) && (!defined(_MSC_VER) || _MSC_VER < 1800)
>  #define PRIuMAX "I64u"
> @@ -580,4 +580,4 @@ int main(int argc, const char **argv);
>  /*
>   * Used by Pthread API implementation for Windows
>   */
> -extern int err_win_to_posix(DWORD winerr);
> +int err_win_to_posix(DWORD winerr);
> diff --git a/compat/win32/pthread.h b/compat/win32/pthread.h
> index c6cb8dd219..f1cfe73de9 100644
> --- a/compat/win32/pthread.h
> +++ b/compat/win32/pthread.h
> @@ -50,7 +50,7 @@ typedef struct {
>  	DWORD tid;
>  } pthread_t;
>
> -extern int pthread_create(pthread_t *thread, const void *unused,
> +int pthread_create(pthread_t *thread, const void *unused,
>  			  void *(*start_routine)(void*), void *arg);
>
>  /*
> @@ -59,10 +59,10 @@ extern int pthread_create(pthread_t *thread, const void *unused,
>   */
>  #define pthread_join(a, b) win32_pthread_join(&(a), (b))
>
> -extern int win32_pthread_join(pthread_t *thread, void **value_ptr);
> +int win32_pthread_join(pthread_t *thread, void **value_ptr);
>
>  #define pthread_equal(t1, t2) ((t1).tid == (t2).tid)
> -extern pthread_t pthread_self(void);
> +pthread_t pthread_self(void);

ACK!

Thanks,
Dscho

>
>  static inline void NORETURN pthread_exit(void *ret)
>  {
> --
> 2.23.0.248.g3a9dd8fb08
>
>

Re: [PATCH 1/2] mingw: apply array.cocci rule

[Junio C Hamano]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> Hi Denton,
>
> On Wed, 4 Sep 2019, Denton Liu wrote:
>
>> After running Coccinelle on all sources inside compat/ that were created
>> by us[1], it was found that compat/mingw.c violated an array.cocci rule
>> in two places and, thus, a patch was generated. Apply this patch so that
>> all compat/ sources created by us follows all cocci rules.
>>
>> [1]: Do not run Coccinelle on files that are taken from some upstream
>> because in case we need to pull updates from them, we would like to have
>> diverged as little as possible in order to make merging updates simpler.
>>
>> The following sources were determined to have been taken from some
>> upstream:
>>
>> * compat/regex/
>> * compat/inet_ntop.c
>> * compat/inet_pton.c
>> * compat/nedmalloc/
>> * compat/obstack.{c,h}
>> * compat/poll/
>>
>> Signed-off-by: Denton Liu <liu.denton@gmail.com>
>> ---
>>  compat/mingw.c | 4 ++--
>>  1 file changed, 2 insertions(+), 2 deletions(-)
>>
>> diff --git a/compat/mingw.c b/compat/mingw.c
>> index 738f0a826a..a3b1e9e3bb 100644
>> --- a/compat/mingw.c
>> +++ b/compat/mingw.c
>> @@ -1265,7 +1265,7 @@ static wchar_t *make_environment_block(char **deltaenv)
>>  		}
>>
>>  		ALLOC_ARRAY(result, size);
>> -		memcpy(result, wenv, size * sizeof(*wenv));
>> +		COPY_ARRAY(result, wenv, size);
>>  		FreeEnvironmentStringsW(wenv);
>>  		return result;
>>  	}
>> @@ -1309,7 +1309,7 @@ static wchar_t *make_environment_block(char **deltaenv)
>>  			continue;
>>
>>  		size = wcslen(array[i]) + 1;
>> -		memcpy(p, array[i], size * sizeof(*p));
>> +		COPY_ARRAY(p, array[i], size);
>
> ACK!
>
> Thanks,
> Dscho

Yup, looks good.  Thanks both.

Re: [PATCH 2/2] compat/*.[ch]: remove extern from function declarations using spatch

[Denton Liu]

Hi Dscho,

On Wed, Sep 04, 2019 at 11:43:06PM +0200, Johannes Schindelin wrote:
> Hi Denton,
> 
> On Wed, 4 Sep 2019, Denton Liu wrote:
> 
> > In 554544276a (*.[ch]: remove extern from function declarations using
> > spatch, 2019-04-29), we removed externs from function declarations using
> > spatch but we intentionally excluded files under compat/ since some are
> > directly copied from an upstream and we should avoid churning them so
> > that manually merging future updates will be simpler.
> >
> > In the last commit, we determined the files which taken from an upstream
> > so we can exclude them and run spatch on the remainder.
> >
> > This was the Coccinelle patch used:
> >
> > 	@@
> > 	type T;
> > 	identifier f;
> > 	@@
> > 	- extern
> > 	  T f(...);
> >
> > and it was run with:
> >
> > 	$ git ls-files compat/\*\*.{c,h} |
> > 		xargs spatch --sp-file contrib/coccinelle/noextern.cocci --in-place
> > 	$ git checkout -- \
> > 		compat/regex/ \
> > 		compat/inet_ntop.c \
> > 		compat/inet_pton.c \
> > 		compat/nedmalloc/ \
> > 		compat/obstack.{c,h} \
> > 		compat/poll/
> >
> > Coccinelle has some trouble dealing with `__attribute__` and varargs so
> > we ran the following to ensure that no remaining changes were left
> > behind:
> >
> > 	$ git ls-files compat/\*\*.{c,h} |
> > 		xargs sed -i'' -e 's/^\(\s*\)extern \([^(]*([^*]\)/\1\2/'
> > 	$ git checkout -- \
> > 		compat/regex/ \
> > 		compat/inet_ntop.c \
> > 		compat/inet_pton.c \
> > 		compat/nedmalloc/ \
> > 		compat/obstack.{c,h} \
> > 		compat/poll/
> 
> I wonder whether we want to make this part of the (slightly misnamed)
> "Static Analysis" job in our CI.

Do you mean running cocci on all of our source files as opposed to just
the files we compile? These two patches are part of an experimental (and
unsubmitted) patchset that does exactly that. Seeing that there's
interest, I'll try to send it in soon.

Thanks,

Denton

> 
> >
> > Signed-off-by: Denton Liu <liu.denton@gmail.com>
> > ---
> >  compat/mingw.h         | 6 +++---
> >  compat/win32/pthread.h | 6 +++---
> >  2 files changed, 6 insertions(+), 6 deletions(-)
> >
> > diff --git a/compat/mingw.h b/compat/mingw.h
> > index a03e40e6e2..9ad204c57c 100644
> > --- a/compat/mingw.h
> > +++ b/compat/mingw.h
> > @@ -11,7 +11,7 @@ typedef _sigset_t sigset_t;
> >  #undef _POSIX_THREAD_SAFE_FUNCTIONS
> >  #endif
> >
> > -extern int mingw_core_config(const char *var, const char *value, void *cb);
> > +int mingw_core_config(const char *var, const char *value, void *cb);
> >  #define platform_core_config mingw_core_config
> >
> >  /*
> > @@ -443,7 +443,7 @@ static inline void convert_slashes(char *path)
> >  			*path = '/';
> >  }
> >  #define PATH_SEP ';'
> > -extern char *mingw_query_user_email(void);
> > +char *mingw_query_user_email(void);
> >  #define query_user_email mingw_query_user_email
> >  #if !defined(__MINGW64_VERSION_MAJOR) && (!defined(_MSC_VER) || _MSC_VER < 1800)
> >  #define PRIuMAX "I64u"
> > @@ -580,4 +580,4 @@ int main(int argc, const char **argv);
> >  /*
> >   * Used by Pthread API implementation for Windows
> >   */
> > -extern int err_win_to_posix(DWORD winerr);
> > +int err_win_to_posix(DWORD winerr);
> > diff --git a/compat/win32/pthread.h b/compat/win32/pthread.h
> > index c6cb8dd219..f1cfe73de9 100644
> > --- a/compat/win32/pthread.h
> > +++ b/compat/win32/pthread.h
> > @@ -50,7 +50,7 @@ typedef struct {
> >  	DWORD tid;
> >  } pthread_t;
> >
> > -extern int pthread_create(pthread_t *thread, const void *unused,
> > +int pthread_create(pthread_t *thread, const void *unused,
> >  			  void *(*start_routine)(void*), void *arg);
> >
> >  /*
> > @@ -59,10 +59,10 @@ extern int pthread_create(pthread_t *thread, const void *unused,
> >   */
> >  #define pthread_join(a, b) win32_pthread_join(&(a), (b))
> >
> > -extern int win32_pthread_join(pthread_t *thread, void **value_ptr);
> > +int win32_pthread_join(pthread_t *thread, void **value_ptr);
> >
> >  #define pthread_equal(t1, t2) ((t1).tid == (t2).tid)
> > -extern pthread_t pthread_self(void);
> > +pthread_t pthread_self(void);
> 
> ACK!
> 
> Thanks,
> Dscho
> 
> >
> >  static inline void NORETURN pthread_exit(void *ret)
> >  {
> > --
> > 2.23.0.248.g3a9dd8fb08
> >
> >

Re: [PATCH 2/2] compat/*.[ch]: remove extern from function declarations using spatch

[Johannes Schindelin]

Hi Denton,

On Wed, 4 Sep 2019, Denton Liu wrote:

> On Wed, Sep 04, 2019 at 11:43:06PM +0200, Johannes Schindelin wrote:
> >
> > On Wed, 4 Sep 2019, Denton Liu wrote:
> >
> > > In 554544276a (*.[ch]: remove extern from function declarations using
> > > spatch, 2019-04-29), we removed externs from function declarations using
> > > spatch but we intentionally excluded files under compat/ since some are
> > > directly copied from an upstream and we should avoid churning them so
> > > that manually merging future updates will be simpler.
> > >
> > > In the last commit, we determined the files which taken from an upstream
> > > so we can exclude them and run spatch on the remainder.
> > >
> > > This was the Coccinelle patch used:
> > >
> > > 	@@
> > > 	type T;
> > > 	identifier f;
> > > 	@@
> > > 	- extern
> > > 	  T f(...);
> > >
> > > and it was run with:
> > >
> > > 	$ git ls-files compat/\*\*.{c,h} |
> > > 		xargs spatch --sp-file contrib/coccinelle/noextern.cocci --in-place
> > > 	$ git checkout -- \
> > > 		compat/regex/ \
> > > 		compat/inet_ntop.c \
> > > 		compat/inet_pton.c \
> > > 		compat/nedmalloc/ \
> > > 		compat/obstack.{c,h} \
> > > 		compat/poll/
> > >
> > > Coccinelle has some trouble dealing with `__attribute__` and varargs so
> > > we ran the following to ensure that no remaining changes were left
> > > behind:
> > >
> > > 	$ git ls-files compat/\*\*.{c,h} |
> > > 		xargs sed -i'' -e 's/^\(\s*\)extern \([^(]*([^*]\)/\1\2/'
> > > 	$ git checkout -- \
> > > 		compat/regex/ \
> > > 		compat/inet_ntop.c \
> > > 		compat/inet_pton.c \
> > > 		compat/nedmalloc/ \
> > > 		compat/obstack.{c,h} \
> > > 		compat/poll/
> >
> > I wonder whether we want to make this part of the (slightly misnamed)
> > "Static Analysis" job in our CI.
>
> Do you mean running cocci on all of our source files as opposed to just
> the files we compile? These two patches are part of an experimental (and
> unsubmitted) patchset that does exactly that. Seeing that there's
> interest, I'll try to send it in soon.

I look forward to it!
Dscho