From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: ** X-Spam-ASN: AS6130 216.105.38.0/24 X-Spam-Status: No, score=2.8 required=3.0 tests=BAYES_00,BODY_8BITS, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,HTML_MESSAGE,MAILING_LIST_MULTI, MSGID_FROM_MTA_HEADER,RCVD_IN_BL_SPAMCOP_NET,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H2,RCVD_IN_SBL_CSS,SPF_HELO_PASS,SPF_PASS shortcircuit=no autolearn=no autolearn_force=no version=3.4.2 Received: from lists.sourceforge.net (lists.sourceforge.net [216.105.38.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id CCCE61F8C8 for ; Thu, 7 Oct 2021 02:24:52 +0000 (UTC) Received: from [127.0.0.1] (helo=sfs-ml-4.v29.lw.sourceforge.com) by sfs-ml-4.v29.lw.sourceforge.com with esmtp (Exim 4.90_1) (envelope-from ) id 1mYJ5R-0006T0-FV; Thu, 07 Oct 2021 02:24:45 +0000 Received: from [172.30.20.202] (helo=mx.sourceforge.net) by sfs-ml-4.v29.lw.sourceforge.com with esmtps (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mYJ5O-0006St-W9 for sox-devel@lists.sourceforge.net; Thu, 07 Oct 2021 02:24:43 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sourceforge.net; s=x; h=MIME-Version:Content-Type:Subject:References: In-Reply-To:Message-ID:To:From:Date:Sender:Reply-To:Cc: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=z9rA5xy9/i42lREBgWpkf9Kp8sPj2Q6zJC1VNyIZaWE=; b=OTa7X/6wLqlv2MAlvLI43B3gM mKFI2akkKaCq4c74hAzWPKy4gqFD1XdkVGL9ig3Y0+ak2KoF66uUq/Dg1TKZT9h+0jjFlrsv2rj3x EOo0zr0SMh33KaPhkCp6Z3IbXgZLK+0FuhawRzVn0HpdWC1gmBfTDRAsv9kNUs+7amWmk=; DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sf.net; s=x ; h=MIME-Version:Content-Type:Subject:References:In-Reply-To:Message-ID:To: From:Date:Sender:Reply-To:Cc:Content-Transfer-Encoding:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=z9rA5xy9/i42lREBgWpkf9Kp8sPj2Q6zJC1VNyIZaWE=; b=HF3IHPZJypY6Hvo8CCjpUuehqJ 7zcQ0z2IvXX4onrLlK0OjaXuAunbMgPlaL57SbBU9tJkuSMdj4YfhxM2l65VupZ+J3kefhT0PYYUb nSiUt6+FIKIsE6pZgr9k4vR8k92Ysg9MmUDudi2c5VXK11KSnoVC02ce69gQsH744I9c=; Received: from mail-os2jpn01olkn0162.outbound.protection.outlook.com ([104.47.92.162] helo=JPN01-OS2-obe.outbound.protection.outlook.com) by sfi-mx-2.v28.lw.sourceforge.com with esmtps (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.92.3) id 1mYJ5J-0008V2-Rv for sox-devel@lists.sourceforge.net; Thu, 07 Oct 2021 02:24:42 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=X7MUD7CiBCYPkVhLAAGgYI6cQnuEKADWzzaa6bLCOTXOensITQ2dwtk+Ef/zhP0fIO4gGsi+HTmpZQxEaY2leuJSm6zick+IihrpP1ySuH2ERg/4dVRK8AlyQL45VFYzDkrXPKWhqAcTlpWU9KMYBBsK/lm8a2ZeNzIfJsIP+mg7OVMcB7iIGWg+T4IBPDotP/h0uPLSCxKdhNAhf5NB+JQ6Ufn6wGijsbKwAoEV3Q7+6cOD8rUN7T4YM1Z6e+iGrjd+KSn103xVE0sl3BaIu2ZnpTWNTlyjuJpL2j7xv1ZTRoSAYrwfbtOQ76ndlAT6yGVrdgTWL6Tv5c7XAxbtHg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=z9rA5xy9/i42lREBgWpkf9Kp8sPj2Q6zJC1VNyIZaWE=; b=Ey9xVVm0nJ7wdxvKwYeczOQsrnPGLTVbzxsC6YANxL9R1qgU9ogNlLUlGdkF3Z7XczrmCZAW+sWKk3sgx4exIoZdeFvW49KcJ0/rOvKp0uau8EBrW0urxsnPq94fkRtJN9Lp0JdXTkpOT7Rvwef6+9E6yZuapJ+rjt3wUyXDAGtqSAkHUZZ/LnRMNc3kN9nO4HVUnpfJ1TfmHt4MlWmZKxl3RYQHFq5kwI60oESDPYyahwUc4fZhiWT89c0cLdBvYyiteoMCAhtALutSHjfifCH48eg4hAD/i0AgLdh5vufivQyZ0IDwPndSlhbZqdxU6qxgBpY68/z3VB0/bz3GHQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outlook.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=z9rA5xy9/i42lREBgWpkf9Kp8sPj2Q6zJC1VNyIZaWE=; b=memi1pMSXavn1il3G/LRUvz21GMDAZq0utQSj+dTjrlaOdf4mg41yeex6RntgQ8lnXVEe5xWKAVS6be8Dpx7H50ITrNmymzgf3d2Y54Xl9+iHmmbpzx+GMZ9bHBnaKCepvGpxBrMFGKnKvbI6rQzTfjhFy/H1sTY0YYr27xJAFqtbg3jKcUz8duXDBtwK3YSDY1/o+vlxk4PwDpD1kT3zzRDkDOThRPQO9BTpS3Ve5P0475ibNvXRV6SrLvdZ6br3WDhjOhObfxdKuZ/wtx98NTsTB2Dhe1srG6a+3YlqtdYk/fHaoLRDgGulpLo9VX5FOfiPoL5zXabXsAwRCPoCg== Received: from OS3PR01MB5976.jpnprd01.prod.outlook.com (2603:1096:604:d8::13) by OSBPR01MB4376.jpnprd01.prod.outlook.com (2603:1096:604:76::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4587.18; Thu, 7 Oct 2021 02:24:29 +0000 Received: from OS3PR01MB5976.jpnprd01.prod.outlook.com ([fe80::e14f:40a0:5df6:f6c6]) by OS3PR01MB5976.jpnprd01.prod.outlook.com ([fe80::e14f:40a0:5df6:f6c6%8]) with mapi id 15.20.4587.018; Thu, 7 Oct 2021 02:24:29 +0000 Date: Thu, 7 Oct 2021 10:24:21 +0800 From: Sun Zhenliang To: sox-devel@lists.sourceforge.net Message-ID: In-Reply-To: References: X-Readdle-Message-ID: 9ec77263-b06a-4326-bb65-9b7f973b1811@Spark Content-Type: multipart/mixed; boundary="615e5a5a_542289ec_7b22" X-TMN: [cVcSc56p40mT+XsB9HXFJw3P8c0zjKppEa4U6wx2qlE=] X-ClientProxiedBy: HK2PR02CA0217.apcprd02.prod.outlook.com (2603:1096:201:20::29) To OS3PR01MB5976.jpnprd01.prod.outlook.com (2603:1096:604:d8::13) X-Microsoft-Original-Message-ID: <9ec77263-b06a-4326-bb65-9b7f973b1811@Spark> MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from [127.0.0.1] (220.246.124.189) by HK2PR02CA0217.apcprd02.prod.outlook.com (2603:1096:201:20::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4566.14 via Frontend Transport; Thu, 7 Oct 2021 02:24:29 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: a261d4ce-3abf-499f-c576-08d989399757 X-MS-TrafficTypeDiagnostic: OSBPR01MB4376: X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Xii79SDkwkO82q4h1QxEdQZGB5NMM5XoKDH3tmx5aWdun3TBJFLJthnlCuQSUT6ECOf2SHTTppmKeOBPio9dYDlxA2WsQZ3rK9CyDO7O1QBoo23s1HRdVSoj56jMXH4uvu7cfDgouU0Wph8u7/nqO4z7lCjeX40f61oagcS6CnrNi7Q736pRWKyh6P+lmcvitUz1Z9LSmQrnM9uaG/vT5x6zouKO0p24dY21DW5BEP1jdtKBhFOxfyDxwEO8L54V+szvRBtuRXRuSvfTz8JOnLaXpuaoXHz/P52a16i1jPJCmDTZ1gzFBSHqVWCsxoUf3LQ+Bs8htQMwl2W0zNLxgI5y2vnNGOZOqLjsy7ObVk4Q33X+yhCyKW+YumsTF5jq9poXw93CBL9oc742mRlGU3JrTJfGI9EPkugjWpNnYP7moaYOLokOBON2ev8GJsW7/O2vNGBoDkG+O3Bs95QGprQSTSFOsUQc9Eajxujg1YRpUoETzXZsai197z5sz0FPXlAxx/79Vt5Eus9nE0Hmtg== X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: hfVncOcCDeDzV+3pgNCbroKfVTi7JRqiw5k1Uy6SzfJpfOTmVuuh6EgjpsnLN90pM9cskXA8Wv6h73JOeR+RWCD1XojgkEWbA1uAr2JnZarWMseil48xK2N7BdkeqEsycfDZrRfjZLO3BC56O0YUNw== X-OriginatorOrg: outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: a261d4ce-3abf-499f-c576-08d989399757 X-MS-Exchange-CrossTenant-AuthSource: OS3PR01MB5976.jpnprd01.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Oct 2021 02:24:29.8074 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: OSBPR01MB4376 X-Headers-End: 1mYJ5J-0008V2-Rv Subject: Re: [PATCH] formats: disallow seeking in dynamic memory buffers X-BeenThere: sox-devel@lists.sourceforge.net X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: sox-devel@lists.sourceforge.net Errors-To: sox-devel-bounces@lists.sourceforge.net --615e5a5a_542289ec_7b22 Content-Type: multipart/alternative; boundary="615e5a5a_4b588f54_7b22" --615e5a5a_4b588f54_7b22 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline =E5=9C=A8 2021=E5=B9=B410=E6=9C=887=E6=97=A5 +0800 05:35=EF=BC=8CSven Neu= mann via SoX-devel =EF=BC=8C=E5=86=99=E9= =81=93=EF=BC=9A > Hi, > > in one of our internal applications we are using SoX (the 14.4.2+git201= 90427 version from Ubuntu) to convert from a variety of audio formats to = the WAV file format. We observed that the tests for the conversion occasi= onally failed and over the last days I found time to dig deeper into this= . > > We are using sox=5Fopen=5Fmemstream=5Fwrite() to write to a dynamically= allocated in-memory stream. In our tests sometimes the size of the resul= ting WAV buffer would have the expected size, sometimes it would be 44 by= tes, the size of the WAV header. Valgrind told me that the behavior of is= =5Fseekable() in formats.c depends on uninitialized memory. In your git r= epository I found a fix for this: > > commit bb38934e11035c8fab141f70dabda3afdd17da36 > Author: Mans Rullgard > Date: Tue Aug 4 17:19:49 2020 +0100 > > format: improve is=5Fseekable() test > > Streams opened with fmemopen() do not have an underlying file descripto= r, > so the fstat() will fail, and a random result is returned. > > A simpler method that works regardless of file type is to call fseek() > and check if it reports success. > > Suggested by Stefan Sauer . > > > Now with this fix applied valgrind was happy, however now our conversio= n from MP3 to WAV would always result in only 44 bytes, as read from the = buffer=5Fsize=5Fptr location passed to sox=5Fopen=5Fmemstream=5Fwrite(). = It turns out that with above change the undefined behavior is fixed for s= treams created with open=5Fmemstream() and is=5Fseekable() will now relia= bly returns sox=5Ftrue for such streams. This allows the WAV writer code = to do an fseek() to the start of the stream followed by a write of the WA= V header with correct length information. However such a seek followed by= a write causes the dynamically allocated memory stream to be truncated. = Thus after calling sox=5Fclose() the size reported for the stream will be= 44 bytes, that's not what we want. Unfortunately we can not simply fix t= his by reporting the full buffer size as the buffer will actually have be= en truncated, and a trailing null byte is appended after the WAV header. = It looks like we can indeed not seek and fix data in a dynamically alloca= ted stream. Thus I am attaching a patch that changes the code in formats.= c to set ft->seekable to false for streams opened with open=5Fmemstream()= . With this change applied on top of the improvement for the is=5Fseekabl= e() test, our tests pass reliably and valgrind seems happy as well. > > I am attaching the patch here, please consider it for inclusion. I am a= lso attaching a simple test application that writes to a stream, seeks to= the front and performs another write. The output of this program illustr= ates that the buffer is truncated: > > buf =3D =60hello', size =3D 5 > buf =3D =60hello, world', size =3D 12 > buf =3D =60heyho', size =3D 5 Bug mentioned=C2=A0https://sourceforge.net/p/sox/mailman/message/37325794= /. It is possible to fix the whole data. =46rom my test to open=5Fmemstream(= ), the dynamically allocated memory would not be free before the ft->fp was clos= ed. The data written is still there if it=E2=80=99s not covered by other writ= e opt. In this WAV issue, the seek opt is just in sox=5Fclose() to rewrite heade= r, which is the end of transcoding and will not cover the data area. We can just f= tell() to get the end of file before the fseek and seek back to the end after re= writing. Output with ftell opt: buf =3D =60hello', size =3D 5 buf =3D =60hello, world', size =3D 12 buf =3D =60heyho, world', size =3D 12 As for other formats, the situation may be different from WAV and needs s= pecific analysis. > > > Regards, > Sven > > > > > =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F > SoX-devel mailing list > SoX-devel=40lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/sox-devel --615e5a5a_4b588f54_7b22 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline
=E5=9C=A8 2021=E5=B9=B410=E6=9C=887=E6=97=A5 +0800 05:35= =EF=BC=8CSven Neumann via SoX-devel <sox-devel@lists.sourceforge.net>= =EF=BC=8C=E5=86=99=E9=81=93=EF=BC=9A
Hi,
in one of our internal applications we are using SoX (the 14.4.2+git2019042= 7 version from Ubuntu) to convert from a variety of audio formats to the WA= V file format. We observed that the tests for the conversion occasionally f= ailed and over the last days I found time to dig deeper into this.

We are using sox_open_memstream_write() to write to a dynamically allocated= in-memory stream. In our tests sometimes the size of the resulting WAV buf= fer would have the expected size, sometimes it would be 44 bytes, the size = of the WAV header. Valgrind told me that the behavior of is_seekable() in f= ormats.c depends on uninitialized memory. In your git repository I found a = fix for this:

commit bb38934e11035c8fab141f70dabda3afdd17da36
Author: Mans Rullgard <mans@mansr.com>
Date: Tue Aug 4 17:19:49 2020 +0100

format: improve is_seekable() test

Streams opened with fmemopen() do not have an underlying file descriptor, so the fstat() will fail, and a random result is returned.

A simpler method that works regardless of file type is to call fseek()
and check if it reports success.

Suggested by Stefan Sauer <ensonic@google.com>.


Now with this fix applied valgrind was happy, however now our conversion fr= om MP3 to WAV would always result in only 44 bytes, as read from the buffer= _size_ptr location passed to sox_open_memstream_write(). It turns out that = with above change the undefined behavior is fixed for streams created with = open_memstream() and is_seekable() will now reliably returns sox_true for s= uch streams. This allows the WAV writer code to do an fseek() to the start = of the stream followed by a write of the WAV header with correct length inf= ormation. However such a seek followed by a write causes the dynamically al= located memory stream to be truncated. Thus after calling sox_close() the s= ize reported for the stream will be 44 bytes, that's not what we want. Unfo= rtunately we can not simply fix this by reporting the full buffer size as t= he buffer will actually have been truncated, and a trailing null byte is ap= pended after the WAV header. It looks like we can indeed not seek and fix d= ata in a dynamically allocated stream. Thus I am attaching a patch that cha= nges the code in formats.c to set ft->seekable to false for streams open= ed with open_memstream(). With this change applied on top of the improvemen= t for the is_seekable() test, our tests pass reliably and valgrind seems ha= ppy as well.

I am attaching the patch here, please consider it for inclusion. I am also = attaching a simple test application that writes to a stream, seeks to the f= ront and performs another write. The output of this program illustrates tha= t the buffer is truncated:

buf =3D `hello', size =3D 5
buf =3D `hello, world', size =3D 12
buf =3D `heyho', size =3D 5
Bug mentioned https://sourceforge.net/p/s= ox/mailman/message/37325794/.

It is possible to fix the whole data. From my test to open_memstream(), the=  
dynamically allocated memory would not be free before the ft->fp was clo= sed. 
The data written is still there if it=E2=80=99s not covered by other write = opt. 

In this WAV issue, the seek opt is just in sox_close() to rewrite header, w= hich 
is the end of transcoding and will not cover the data area. We can just fte= ll() 
to get the end of file before the fseek and seek back to the end after rewr= iting.

Output with ftell opt:
buf =3D `hello', size =3D 5
buf =3D `hello, world', size =3D 12
buf =3D `heyho, world', size =3D 12

As for other formats, the situation may be different from WAV and needs spe= cific 
analysis.



Regards,
Sven




_______________________________________________
SoX-devel mailing list
SoX-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sox-devel
--615e5a5a_4b588f54_7b22-- --615e5a5a_542289ec_7b22 Content-Type: application/octet-stream Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="test.c" I2luY2x1ZGUgPHN0ZGlvLmg+CgppbnQKbWFpbiAodm9pZCkKewogIGNoYXIgKmJwOwogIHNpemVf dCBzaXplOwogIEZJTEUgKnN0cmVhbTsKCiAgaW50IGVuZDsKCiAgc3RyZWFtID0gb3Blbl9tZW1z dHJlYW0gKCZicCwgJnNpemUpOwogIGZwcmludGYgKHN0cmVhbSwgImhlbGxvIik7CiAgZmZsdXNo IChzdHJlYW0pOwogIHByaW50ZiAoImJ1ZiA9IGAlcycsIHNpemUgPSAlbGRcbiIsIGJwLCBzaXpl KTsKICBmcHJpbnRmIChzdHJlYW0sICIsIHdvcmxkIik7CiAgZmZsdXNoIChzdHJlYW0pOwogIHBy aW50ZiAoImJ1ZiA9IGAlcycsIHNpemUgPSAlbGRcbiIsIGJwLCBzaXplKTsKCiAgZW5kID0gZnRl bGwoc3RyZWFtKTsKCiAgZnNlZWsgKHN0cmVhbSwgMCwgU0VFS19TRVQpOwogIGZwcmludGYgKHN0 cmVhbSwgImhleWhvIik7CgogIGZzZWVrKHN0cmVhbSwgZW5kLCBTRUVLX1NFVCk7CgogIGZjbG9z ZSAoc3RyZWFtKTsKICBwcmludGYgKCJidWYgPSBgJXMnLCBzaXplID0gJWxkXG4iLCBicCwgc2l6 ZSk7CgogIHJldHVybiAwOwp9Cgo= --615e5a5a_542289ec_7b22 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline --615e5a5a_542289ec_7b22 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ SoX-devel mailing list SoX-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sox-devel --615e5a5a_542289ec_7b22--