From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS6130 216.105.38.0/24 X-Spam-Status: No, score=-2.7 required=3.0 tests=AWL,BAYES_00,DKIM_ADSP_ALL, DKIM_INVALID,DKIM_SIGNED,HTML_MESSAGE,HTTPS_HTTP_MISMATCH, MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS, SPF_PASS shortcircuit=no autolearn=no autolearn_force=no version=3.4.2 Received: from lists.sourceforge.net (lists.sourceforge.net [216.105.38.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id 869B61F8C8 for ; Thu, 7 Oct 2021 14:28:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.sourceforge.net; s=beta; h=Cc:Reply-To:From:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:Subject:MIME-Version: Content-Type:In-Reply-To:References:Message-ID:Date:To:Sender: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=OaMIBthaW6jEKcJcwUxqNOF4C/b5SIPq0pUzmgpf/M8=; b=AOl2rrPRus40NIyq9xXKrw3e2 5mRKd5EIbG3jFdrK6lIZOC7bhzOCGGIXd4jSS9Nyey+k41cFvg2dOfFh3vOqCfRSQ8GjXoc4AemAL 0jv0MJDtO5vD81KKHFOSaI+VmcQqJZM3YPZMvYvP7a0we2IWd22qj+4W6guVeCjWvhsf8=; Received: from [127.0.0.1] (helo=sfs-ml-4.v29.lw.sourceforge.com) by sfs-ml-4.v29.lw.sourceforge.com with esmtp (Exim 4.90_1) (envelope-from ) id 1mYUNK-00005o-EX; Thu, 07 Oct 2021 14:27:58 +0000 Received: from [172.30.20.202] (helo=mx.sourceforge.net) by sfs-ml-4.v29.lw.sourceforge.com with esmtps (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mYUNJ-00005i-Gf for sox-devel@lists.sourceforge.net; Thu, 07 Oct 2021 14:27:57 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sourceforge.net; s=x; h=MIME-Version:Content-Type:In-Reply-To:References: Message-ID:Date:Subject:To:From:Sender:Reply-To:Cc:Content-Transfer-Encoding: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=wrmGrYyKY9/nDTfuUtwKPCV/tDCDr6qruhLmDS3mtL4=; b=Dj/a4A6UiOct7OJCIt0KcBPhpX OcEUPthzospTcsCQfA1He/UOEdoZdeX+HxIvdjloJJJKjLHpY3hGHIxXWynMoEAjF8FCBXVZPVDIU ZQDGv1eT9KWdzT49bUmh7rzosLPauBJpebGEG1xgOov1CETsMHrrL+XryxAoa/6z2PrU=; DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sf.net; s=x ; h=MIME-Version:Content-Type:In-Reply-To:References:Message-ID:Date:Subject: To:From:Sender:Reply-To:Cc:Content-Transfer-Encoding:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=wrmGrYyKY9/nDTfuUtwKPCV/tDCDr6qruhLmDS3mtL4=; b=O1CgGncwTaGdZv+mBAGGxVwwpI kvI2vnlWzCiueFGC89gfp1iR5qzuaZ5Yo1kQ5ABAaS8Mi4pqAkQUfU1ANMsH54ixZ2EG9socRCfnY MGZOONF1HkpI/o8t0/6IP+x+429h92nuzYE1hqjhLl1iuiGEFxWphHm6mwOzL4Tmv0hw=; Received: from mx0a-001a0901.pphosted.com ([67.231.144.179] helo=mx0b-001a0901.pphosted.com) by sfi-mx-2.v28.lw.sourceforge.com with esmtps (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.92.3) id 1mYUNI-000216-4I for sox-devel@lists.sourceforge.net; Thu, 07 Oct 2021 14:27:57 +0000 Received: from pps.filterd (m0074903.ppops.net [127.0.0.1]) by mx0a-001a0901.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 197B9CjG009072 for ; Thu, 7 Oct 2021 09:20:34 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=logmein.com; h=from : to : subject : date : message-id : references : in-reply-to : content-type : mime-version; s=email; bh=wrmGrYyKY9/nDTfuUtwKPCV/tDCDr6qruhLmDS3mtL4=; b=diPxEtB4H7oPfHgQbmiXELs1IP4AmYfYiV1XnsEJCzgV4XdKtGySWbKD3Va1VAg08gkJ SuWsh3f01KIpDg2lUVW93J9yGjloDlfpJaXXeekQPVHjGfx/gL7q8+HUwAZkZPxHs1zF aGc4kUU/Ozykt5vlHP7rdn3H9zUaz8/4Ae8= Received: from nam10-bn7-obe.outbound.protection.outlook.com (mail-bn7nam10lp2108.outbound.protection.outlook.com [104.47.70.108]) by mx0a-001a0901.pphosted.com with ESMTP id 3bhrktt2dg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 07 Oct 2021 09:20:33 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=UELF/WJOcIn05JfIsm3kprRrM8ML3KHerKFJXrr8Ngtpz4FJvCFRIl0YkvshWMZEiKsj4sRRQkWNIytRSl0obDVcAXH9T7m9lGIQOAh5WNjf7MxzeSD8RQoh+YJQbnPAoHj4skpd2nTpQjalINn+ACGZ7DjBxKP4SnXUf1/HlhL42Fhi36D+DgCZdhzQI0i/E6pFqz+wJ5aRinnlCk4tdkrzJ1K6m9/onLcupztOXLIMSfmzlpWG7HuJk0lRTBXv3mZWkOXD5mRr3DjDB9UyxuPaSSwgGOA4XGJ9NySq9tQClqEM9ttGK8FlQ5ra4HPRL1dFujMuse6EgGnmNCVWeQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=wrmGrYyKY9/nDTfuUtwKPCV/tDCDr6qruhLmDS3mtL4=; b=WkY1hboMdaavmLGeLlN0MYE48ISWDWUzREeO71TdrbsILsKXfRszhuzRaHKKtT9lq8ZunAT57AM5xndd4RvJGa+0tfZI9zMRQjZMdNBpQc/lV7hQEloPCyg1lU1PAJLtXroCQsobhQyO0cnBEAlU46R4QxHKVe/2bzGemXkGchxkLXLst48kFcVZqFZOdd02T7uHoT15BdjBLGAlDH9U7FBTTPruIpKpw/OvzLLH9T9bmNfX/w1jjh48/Rn8rBVTC2lo5lxyglK7HjKFDAvlQNYZTFebXwb5rjZsoy0b+qIxqmtxwCwrGcmRU3mcac0C/ovQSfFfvKP8vDJBKjgzZA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=logmein.com; dmarc=pass action=none header.from=logmein.com; dkim=pass header.d=logmein.com; arc=none Received: from CO1PR15MB4890.namprd15.prod.outlook.com (2603:10b6:303:e1::8) by MW4PR15MB4764.namprd15.prod.outlook.com (2603:10b6:303:10d::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4587.19; Thu, 7 Oct 2021 13:20:30 +0000 Received: from CO1PR15MB4890.namprd15.prod.outlook.com ([fe80::40ad:43c3:ed52:168c]) by CO1PR15MB4890.namprd15.prod.outlook.com ([fe80::40ad:43c3:ed52:168c%9]) with mapi id 15.20.4587.020; Thu, 7 Oct 2021 13:20:30 +0000 To: "sox-devel@lists.sourceforge.net" Thread-Topic: [SoX-devel] .Re: [PATCH] formats: disallow seeking in dynamic memory buffers Thread-Index: AQHXu3YZ1hKNxIal+USyiR4kefI/WKvHhEdM Date: Thu, 7 Oct 2021 13:20:30 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-GB, en-US Content-Language: en-GB X-MS-Has-Attach: yes X-MS-TNEF-Correlator: suggested_attachment_session_id: db0b18fc-b0d3-65cc-2730-82467a51d8b8 authentication-results: lists.sourceforge.net; dkim=none (message not signed) header.d=none; lists.sourceforge.net; dmarc=none action=none header.from=logmein.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: e6bad0f5-1f68-4eb8-0e17-08d989953c6d x-ms-traffictypediagnostic: MW4PR15MB4764: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: sEMT5Mu7ZxlbzOoVJEhTHtr+gs8iYxAAOEwrgDuS53itw2Z8Pc7RHivd6KY2IxEXIUkuN9t7OA7xdnwabslZMVo0Hp+aH+HdpHtnbCW+OOPSOg1ocr7vuK1C4GVKaZS7LXwRTNI3uKIk6Gj+9R48W3CdqKQH5VOVN3JLLjluirxLQpP9qpArSe1+iA1uAf0RHbGkSc6zw/6LpfYTOA4EcgM3pMCitp0YKQcKyxEzxs8qsYqfxtBV8gZPqSfyCp+qhwuVNG6DnNA/9nl8wYnFHYjZ8ep+f7x/wqsCxahAuG4oV2TQDLe71DeDJv4u75UcShHfmCf0Yx3TfdYQgJ0pGbJSC0nkcm2cSzErSCYPuSLLIKuNM13Ge9Hom/Rqsd7nhL5z8jNgDw35LT+joY9oaR8WxZBOm0/+WnGxx/6WMsiPJVetrmb2qhKNQEcoYjiTvacg3EXqJW1Hm8fs2es+gNqQRNyjomOs+utSKvpItilSFVF5zRrZnNkU0MX734m5oZRN6NjVZEDa+0vU/4pKpzRYE5f3GOr5CJqEE16XHJuQw4Lvpc2xsRXvhdySBlQb7HRXvTRHOW5eHxZ5blAgh2/QSbBmgvJaJcA39gXHdKjesm+Y70eZQc/4DvEApbTESDPfBYrO3/aOi+6cBGdHWPQ4hPkl+pRJFeLihhX+9pBg4r8YoZ8uU/mdDTR81/NPcckN/vkNdWgCbUoytfvdj0K4wq7y7bVllZJb7jXTVFX3BFs3om4ysZMJ6sA5tXrTl6XWe3TRuFVLsylXWJocIC2qBZmM5s2e9VMFyDtlYaA= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CO1PR15MB4890.namprd15.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(9686003)(5660300002)(45080400002)(83380400001)(508600001)(8676002)(55016002)(66946007)(76116006)(66556008)(66446008)(966005)(33656002)(52536014)(6916009)(38070700005)(66476007)(64756008)(316002)(2906002)(53546011)(6506007)(8936002)(19627405001)(186003)(91956017)(26005)(86362001)(7696005)(99936003)(122000001)(38100700002)(71200400001)(166002); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-2022-jp?B?ZktBcTNJY282b0MyWmZNMjhYYTdxcjY5UlRSR0pYWjBCSnVTMXhLbkgy?= =?iso-2022-jp?B?MUFBVHZ0dmd3WkxhaXFCZkx0VXVhem4wdVkrVWdyZmtEZ0pYS05yenNS?= =?iso-2022-jp?B?bUlhcW1XWVdUMEJCOHMxalhlem55SFpVRU9DOGF1YTV4S1hBN1MyMi9Q?= =?iso-2022-jp?B?WXZ3aTFiVDF4aDBXWVN3a3FSU0xwUUJmKzJyMEhOdi95T2hpeFZyT3NX?= =?iso-2022-jp?B?MXhvYkI4VGdHRGdMU0I5NkdzTHVmSG05QWJTM1pSV09TUmVYbXVMMGFM?= =?iso-2022-jp?B?T2xKQXAwRytPdTJWM2RhaHhEM2FHdG9MQU9lUi9Fd1dwVnJtem1Sbmg1?= =?iso-2022-jp?B?T1pqY3Bka1ZETkU1VHhUWjNNVHNZVEJBYkpITGVJNkQ1Z2cvQ0tNa3E3?= =?iso-2022-jp?B?Unoyb1RVR2tXdzVibGJ4UzRaaDhDRjE0ajdXOUhoNzdEMnlhc0loNDRL?= =?iso-2022-jp?B?NWU0NmVkSVdlTWszbzY5RXlDdzJ1b0VNL1lFM25KMEVkM2RoellnN0Jl?= =?iso-2022-jp?B?dFpteUJTNnJmeWkwNWZTWkZvWHVvZXZVQkxEVzBLNXlZZW9SNUFxamFW?= =?iso-2022-jp?B?V3ZtL0FrTE52bzk2eWs3SWowRC9CQk1ENk1rbkF5WDRDSEFDUUhTQUND?= =?iso-2022-jp?B?YmxKS281MFZxZ09lbEpSKzlNa0pCQnpvTkJVY2F4TGdLdU55S2tGOVBN?= =?iso-2022-jp?B?Ry8zZlRrRjBISWdGMGhPZ3QwVFdmb1VvTmpub2VhWnIxOGg3Q2Mvelpr?= =?iso-2022-jp?B?enNOeXdib3YyTWk2MkdaOCtkek5CbHBjL050c0kwWWZwQXIwQ2swRFlC?= =?iso-2022-jp?B?SjdWUFFQNUk5eEd3dGxNRSthaW9vSmwyb2hBb2pMM29UbGtyQ0dPR3ZL?= =?iso-2022-jp?B?TW9zVEIvb0Znczd6RkhZZlorSGNTZ1lIZ1JpdnhqTkp2QVVCWkJxWmR4?= =?iso-2022-jp?B?cWxIQjFHMEhVNmpJNnNnbFdDdVcvVmMrQXlpMXlrOTh4a1Fmd1VVVTZF?= =?iso-2022-jp?B?RkNqaGtCbm9pNXhRellVM2dlS1lMN3l4ZWdMQTVjUnVVZ1ErY1VCUjgx?= =?iso-2022-jp?B?YXNKTHFWZFJkdHRaMTAwdjdmTXFWT25YdGdqeXI3aVJFMGtlcmxFcFRW?= =?iso-2022-jp?B?TzRyblJyT0tPc2RCd0RFMUl2RFV5aUJHRmJXRUhyOXU0Mk9nTk9SeVRT?= =?iso-2022-jp?B?VXViSVJjM2pqdDZ2c0xJRHBmeGJIS1B3NGQ3VGRmbTNiTDF1eDFrVW1p?= =?iso-2022-jp?B?QXB1QitrOGlQWmFTL0l1OUVkZlFoV085VEQwYTg1bHNQZ0EyMVBNdVpx?= =?iso-2022-jp?B?YjF2RWg2OTRYOXZ1b3hMUUEzNU0rcUNiWVZvcURPVzlQOWljT2d5RlJ5?= =?iso-2022-jp?B?UlE5MVUxMHg1RG1lblVqek05VWZtM1RxSUk4TWQzSFFPOTJxakc5em5D?= =?iso-2022-jp?B?VmlXUXJLbHdZbGNxK3dBclZScTFlNmZUeTI0dWh6NXdaeHFZWmlBYng5?= =?iso-2022-jp?B?Y3cwYUhxMWc2UHJTN0FFcTV6UGhiQ0F3dW1JcGE2N29jSG9NQ2c1TXVi?= =?iso-2022-jp?B?TkhzOElSN1lnb0pEQkRWa1o4aG9OSmFTbVBBbXVJa0xBMTMxaGRGeGtq?= =?iso-2022-jp?B?UFlNVitqdE4rd0hjam1sWGEvTWN6N0l5NWpCamNSQWFySUtwRlVCOHFJ?= =?iso-2022-jp?B?bzQ2c1pRMW1TKzVpeGZoM2RqTktjRm1wY3FYeDVqd05KaCs1ZmFKUVhv?= =?iso-2022-jp?B?VGN5eUZOMjA5bWIrdkc1WTNxc1dSY2FjSkNhdWlmMTduNjZKU0FrVm10?= =?iso-2022-jp?B?NkdiYkxKMzBXOTlXaTh6TGptbG9JT1VkczJSTHBHWTJNamV3cy9wRktE?= =?iso-2022-jp?B?OEM5ekR6ZG1zL2Z3OHpqOUQvVWo0PQ==?= x-ms-exchange-transport-forked: True Content-Type: multipart/mixed; boundary="_004_CO1PR15MB4890D41F45E417C614B82ED98CB19CO1PR15MB4890namp_" MIME-Version: 1.0 X-OriginatorOrg: logmein.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: CO1PR15MB4890.namprd15.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: e6bad0f5-1f68-4eb8-0e17-08d989953c6d X-MS-Exchange-CrossTenant-originalarrivaltime: 07 Oct 2021 13:20:30.5253 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84c4e5b0-26a0-4dac-b686-301d76713569 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: /4F6PSJyBVb7elhXyHvwLdQ4ajTXkZ/wT2ecYPnnM3zs+CTVOs5xDrF65THIGUNIx406dY/9R8pIVtCVmPuhjEz+x9bXOC6XknadJs3pHEE= X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR15MB4764 X-Proofpoint-GUID: gSRju3x68Dyw8Yqv9VxdlRe2pnLodSiK X-Proofpoint-ORIG-GUID: gSRju3x68Dyw8Yqv9VxdlRe2pnLodSiK X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.182.1,Aquarius:18.0.790,Hydra:6.0.391,FMLib:17.0.607.475 definitions=2021-10-07_01,2021-10-07_02,2020-04-07_01 X-Proofpoint-Spam-Reason: safe X-Headers-End: 1mYUNI-000216-4I Subject: Re: .Re: [PATCH] formats: disallow seeking in dynamic memory buffers X-BeenThere: sox-devel@lists.sourceforge.net X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Sven Neumann via SoX-devel Reply-To: sox-devel@lists.sourceforge.net Cc: Sven Neumann Errors-To: sox-devel-bounces@lists.sourceforge.net --_004_CO1PR15MB4890D41F45E417C614B82ED98CB19CO1PR15MB4890namp_ Content-Type: multipart/alternative; boundary="_000_CO1PR15MB4890D41F45E417C614B82ED98CB19CO1PR15MB4890namp_" --_000_CO1PR15MB4890D41F45E417C614B82ED98CB19CO1PR15MB4890namp_ Content-Type: text/plain; charset="iso-2022-jp" Content-Transfer-Encoding: quoted-printable Hi, thanks for the feedback. Attached you find a patch for the seeking that occ= urs when the WAV header is rewritten. This fixes the reported problem for m= e and it has seen some testing, mostly with dynamic memory streams though. I should be able to find time to look at other places that might need a sim= ilar fix. Regards, Sven ________________________________ From: Sun Zhenliang Sent: 07 October 2021 14:22 To: sox-devel@lists.sourceforge.net Subject: Re: [SoX-devel] .Re: [PATCH] formats: disallow seeking in dynamic = memory buffers =1B$B:_=1B(B 2021=1B$BG/=1B(B10=1B$B7n=1B(B7=1B$BF|=1B(B +0800 14:18=1B$B!$= =1B(BSven Neumann via SoX-devel =1B$B!$. This is exactly the problem I am trying to solve. I am somewhat surprised that we can actually recover the whole buffer after= a seek and write because the memory stream maintains a null byte after the= data. I was assuming that this means that we can not simply recover by see= king back to the end of the buffer because one byte would have been overwri= tten by that terminator byte. I tried what you suggested (code attached), a= nd it seems that the terminating null byte is not moved on a seek operation= , but only when the file descriptor is closed. Of course it would be desirable to have a proper WAV header even for stream= s opened with sox_open_memstream_write(). I can prepare a fix that introduc= es a seek back, but I guess I would have to do something similar for all fo= rmats that seek in the output buffer. Before I start to work on that I woul= d like to get approval from the maintainer that this is the right approach. Yes, I guess there are lots of works to do related to this seek opt. Let m= e know if you need any help. Regards, SunZhenliang Regards, Sven From: Sun Zhenliang Sent: 07 October 2021 04:24 To: sox-devel@lists.sourceforge.net Subject: Re: [SoX-devel] [PATCH] formats: disallow seeking in dynamic memor= y buffers =1B$B:_=1B(B 2021=1B$BG/=1B(B10=1B$B7n=1B(B7=1B$BF|=1B(B +0800 05:35=1B$B!$= =1B(BSven Neumann via SoX-devel =1B$B!$ Date: Tue Aug 4 17:19:49 2020 +0100 format: improve is_seekable() test Streams opened with fmemopen() do not have an underlying file descriptor, so the fstat() will fail, and a random result is returned. A simpler method that works regardless of file type is to call fseek() and check if it reports success. Suggested by Stefan Sauer . Now with this fix applied valgrind was happy, however now our conversion fr= om MP3 to WAV would always result in only 44 bytes, as read from the buffer= _size_ptr location passed to sox_open_memstream_write(). It turns out that = with above change the undefined behavior is fixed for streams created with = open_memstream() and is_seekable() will now reliably returns sox_true for s= uch streams. This allows the WAV writer code to do an fseek() to the start = of the stream followed by a write of the WAV header with correct length inf= ormation. However such a seek followed by a write causes the dynamically al= located memory stream to be truncated. Thus after calling sox_close() the s= ize reported for the stream will be 44 bytes, that's not what we want. Unfo= rtunately we can not simply fix this by reporting the full buffer size as t= he buffer will actually have been truncated, and a trailing null byte is ap= pended after the WAV header. It looks like we can indeed not seek and fix d= ata in a dynamically allocated stream. Thus I am attaching a patch that cha= nges the code in formats.c to set ft->seekable to false for streams opened = with open_memstream(). With this change applied on top of the improvement f= or the is_seekable() test, our tests pass reliably and valgrind seems happy= as well. I am attaching the patch here, please consider it for inclusion. I am also = attaching a simple test application that writes to a stream, seeks to the f= ront and performs another write. The output of this program illustrates tha= t the buffer is truncated: buf =3D `hello', size =3D 5 buf =3D `hello, world', size =3D 12 buf =3D `heyho', size =3D 5 Bug mentioned https://sourceforge.net/p/sox/mailman/message/37325794/. It is possible to fix the whole data. From my test to open_memstream(), the dynamically allocated memory would not be free before the ft->fp was closed= . The data written is still there if it=1B$B!G=1B(Bs not covered by other wri= te opt. In this WAV issue, the seek opt is just in sox_close() to rewrite header, w= hich is the end of transcoding and will not cover the data area. We can just fte= ll() to get the end of file before the fseek and seek back to the end after rewr= iting. Output with ftell opt: buf =3D `hello', size =3D 5 buf =3D `hello, world', size =3D 12 buf =3D `heyho, world', size =3D 12 As for other formats, the situation may be different from WAV and needs spe= cific analysis. Regards, Sven _______________________________________________ SoX-devel mailing list SoX-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sox-devel _______________________________________________ SoX-devel mailing list SoX-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sox-devel --_000_CO1PR15MB4890D41F45E417C614B82ED98CB19CO1PR15MB4890namp_ Content-Type: text/html; charset="iso-2022-jp" Content-Transfer-Encoding: quoted-printable
Hi,

thanks for the feedback. Attached you find a patch for the seeking that occ= urs when the WAV header is rewritten. This fixes the reported problem for m= e and it has seen some testing, mostly with dynamic memory streams though.<= /div>

I should be able to find time to look at other places that might need a sim= ilar fix.


Regards,
Sven



From: Sun Zhenliang <his= unzhenliang@outlook.com>
Sent: 07 October 2021 14:22
To: sox-devel@lists.sourceforge.net <sox-devel@lists.sourceforge.= net>
Subject: Re: [SoX-devel] .Re: [PATCH] formats: disallow seeking in d= ynamic memory buffers
 
=1B$B:_=1B(B 2021=1B$BG/=1B(B10=1B$B7n=1B(B7=1B$BF|=1B(B = +0800 14:18=1B$B!$=1B(BSven Neumann via SoX-devel <sox-devel@lists.sourc= eforge.net>=1B$B!$
Hi,

thanks for pointing me to https://sourcefor= ge.net/p/sox/mailman/message/37325794/. This is exactly the problem I am trying to solve.

I am somewhat surprised that we can actually recover the whole buffer = after a seek and write because the memory stream maintains a null byte afte= r the data. I was assuming that this means that we can not simply recover b= y seeking back to the end of the buffer because one byte would have been overwritten by that terminator byte. I tr= ied what you suggested (code attached), and it seems that the terminating n= ull byte is not moved on a seek operation, but only when the file descripto= r is closed.

Of course it would be desirable to have a proper WAV header even for stream= s opened with sox_open_memstream_write(). I can prepare a fix that introduc= es a seek back, but I guess I would have to do something similar for all fo= rmats that seek in the output buffer. Before I start to work on that I would like to get approval from the maint= ainer that this is the right approach.
Yes, I guess there are lots of works to do related to thi= s seek opt.  Let me know if you need any help. 

Regards,
SunZhenliang


Regards,
Sven


From: Sun Zhenliang <hisunzhenliang@outlook.com>
Sent: 07 October 2021 04:24
To: sox-devel@lists.sourceforge.net <sox-devel@lists.sourceforge.net>=
Subject: Re: [SoX-devel] [PATCH] formats: disallow seeking in dynamic memor= y buffers
 
=1B$B:_=1B(B 2021=1B$BG/=1B(B10=1B$B7n=1B(B7=1B$BF|=1B(B +0800 05:35=1B$B!$= =1B(BSven Neumann via SoX-devel <sox-devel@lists.sourceforge.net>=1B$= B!$ Hi,

in one of our internal applications we are using SoX (the 14.4.2+git2019042= 7 version from Ubuntu) to convert from a variety of audio formats to the WA= V file format. We observed that the tests for the conversion occasionally f= ailed and over the last days I found time to dig deeper into this.

We are using sox_open_memstream_write() to write to a dynamically allocated= in-memory stream. In our tests sometimes the size of the resulting WAV buf= fer would have the expected size, sometimes it would be 44 bytes, the size = of the WAV header. Valgrind told me that the behavior of is_seekable() in formats.c depends on uninitialize= d memory. In your git repository I found a fix for this:

commit bb38934e11035c8fab141f70dabda3afdd17da36
Author: Mans Rullgard <mans@mansr.com>
Date: Tue Aug 4 17:19:49 2020 +0100

format: improve is_seekable() test

Streams opened with fmemopen() do not have an underlying file descriptor, so the fstat() will fail, and a random result is returned.

A simpler method that works regardless of file type is to call fseek()
and check if it reports success.

Suggested by Stefan Sauer <ensonic@google.com>.


Now with this fix applied valgrind was happy, however now our conversion fr= om MP3 to WAV would always result in only 44 bytes, as read from the buffer= _size_ptr location passed to sox_open_memstream_write(). It turns out that = with above change the undefined behavior is fixed for streams created with open_memstream() and is_seekabl= e() will now reliably returns sox_true for such streams. This allows the WA= V writer code to do an fseek() to the start of the stream followed by a wri= te of the WAV header with correct length information. However such a seek followed by a write causes the dyn= amically allocated memory stream to be truncated. Thus after calling sox_cl= ose() the size reported for the stream will be 44 bytes, that's not what we= want. Unfortunately we can not simply fix this by reporting the full buffer size as the buffer will actua= lly have been truncated, and a trailing null byte is appended after the WAV= header. It looks like we can indeed not seek and fix data in a dynamically= allocated stream. Thus I am attaching a patch that changes the code in formats.c to set ft->seekable to false= for streams opened with open_memstream(). With this change applied on top = of the improvement for the is_seekable() test, our tests pass reliably and = valgrind seems happy as well.

I am attaching the patch here, please consider it for inclusion. I am also = attaching a simple test application that writes to a stream, seeks to the f= ront and performs another write. The output of this program illustrates tha= t the buffer is truncated:

buf =3D `hello', size =3D 5
buf =3D `hello, world', size =3D 12
buf =3D `heyho', size =3D 5
Bug mentioned https://sourceforge.net/p/sox= /mailman/message/37325794/.

It is possible to fix the whole data. From my test to open_memstream(), the=  
dynamically allocated memory would not be free before the ft->fp was clo= sed. 
The data written is still there if it=1B$B!G=1B(Bs not covered by other wri= te opt. 

In this WAV issue, the seek opt is just in sox_close() to rewrite header, w= hich 
is the end of transcoding and will not cover the data area. We can just fte= ll() 
to get the end of file before the fseek and seek back to the end after rewr= iting.

Output with ftell opt:
buf =3D `hello', size =3D 5
buf =3D `hello, world', size =3D 12
buf =3D `heyho, world', size =3D 12

As for other formats, the situation may be different from WAV and needs spe= cific 
analysis.



Regards,
Sven




_______________________________________________
SoX-devel mailing list
SoX-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sox-= devel

_______________________________________________
SoX-devel mailing list
SoX-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sox-= devel
--_000_CO1PR15MB4890D41F45E417C614B82ED98CB19CO1PR15MB4890namp_-- --_004_CO1PR15MB4890D41F45E417C614B82ED98CB19CO1PR15MB4890namp_ Content-Type: text/x-patch; name="0001-wav-seek-back-to-end-of-file-after-rewriting-the-hea.patch" Content-Description: 0001-wav-seek-back-to-end-of-file-after-rewriting-the-hea.patch Content-Disposition: attachment; filename="0001-wav-seek-back-to-end-of-file-after-rewriting-the-hea.patch"; size=1554; creation-date="Thu, 07 Oct 2021 13:19:10 GMT"; modification-date="Thu, 07 Oct 2021 13:19:55 GMT" Content-Transfer-Encoding: base64 RnJvbSBmOGZjZmU5YzlhNDhhMjc3NDA3ZTUzNGEzYjVkM2NmMTUyYWMxYjllIE1vbiBTZXAgMTcg MDA6MDA6MDAgMjAwMQpGcm9tOiBTdmVuIE5ldW1hbm4gPHN2ZW4ubmV1bWFubkBsb2dtZWluLmNv bT4KRGF0ZTogVGh1LCA3IE9jdCAyMDIxIDA4OjE4OjAwICswMjAwClN1YmplY3Q6IFtQQVRDSF0g d2F2OiBzZWVrIGJhY2sgdG8gZW5kIG9mIGZpbGUgYWZ0ZXIgcmV3cml0aW5nIHRoZSBoZWFkZXIK ClRoaXMgZml4ZXMgYmVoYXZpb3IgZm9yIG91dHB1dCBzdHJlYW1zIGNyZWF0ZWQgd2l0aApzb3hf b3Blbl9tZW1zdHJlYW1fd3JpdGUoKS4gV2l0aG91dCBzZWVraW5nIGJhY2sgdGhlIGJ1ZmZlciB3 b3VsZApiZSB0cnVuY2F0ZWQgdG8ganVzdCB0aGUgaGVhZGVyLgotLS0KIHNyYy93YXYuYyB8IDE4 ICsrKysrKysrKysrKysrKy0tLQogMSBmaWxlIGNoYW5nZWQsIDE1IGluc2VydGlvbnMoKyksIDMg ZGVsZXRpb25zKC0pCgpkaWZmIC0tZ2l0IGEvc3JjL3dhdi5jIGIvc3JjL3dhdi5jCmluZGV4IDNm NmJlYjQ1Li5jMzFhYjlmOCAxMDA2NDQKLS0tIGEvc3JjL3dhdi5jCisrKyBiL3NyYy93YXYuYwpA QCAtMTU3MCwxMyArMTU3MCwyNSBAQCBzdGF0aWMgaW50IHN0b3B3cml0ZShzb3hfZm9ybWF0X3Qg KiBmdCkKICAgICAgICAgaWYgKCFmdC0+c2Vla2FibGUpCiAgICAgICAgICAgcmV0dXJuIFNPWF9F T0Y7CiAKKyAgICAgICAgb2ZmX3QgZW5kID0gbHN4X3RlbGwoZnQpOwogICAgICAgICBpZiAobHN4 X3NlZWtpKGZ0LCAob2ZmX3QpMCwgU0VFS19TRVQpICE9IDApCiAgICAgICAgIHsKLSAgICAgICAg ICAgICAgICBsc3hfZmFpbF9lcnJubyhmdCxTT1hfRU9GLCJDYW4ndCByZXdpbmQgb3V0cHV0IGZp bGUgdG8gcmV3cml0ZSAud2F2IGhlYWRlci4iKTsKLSAgICAgICAgICAgICAgICByZXR1cm4gU09Y X0VPRjsKKyAgICAgICAgICAgIGxzeF9mYWlsX2Vycm5vKGZ0LCBTT1hfRU9GLCAiQ2FuJ3QgcmV3 aW5kIG91dHB1dCBmaWxlIHRvIHJld3JpdGUgLndhdiBoZWFkZXIuIik7CisgICAgICAgICAgICBy ZXR1cm4gU09YX0VPRjsKKyAgICAgICAgfQorCisgICAgICAgIGludCByZXN1bHQgPSB3YXZ3cml0 ZWhkcihmdCwgMSk7CisgICAgICAgIGlmIChyZXN1bHQgIT0gU09YX1NVQ0NFU1MpCisgICAgICAg ICAgICByZXR1cm4gcmVzdWx0OworCisgICAgICAgIC8qIFNlZWsgYmFjayB0byBlbmQgb2Ygc3Ry ZWFtIGFzIGEgZHluYW1pYyBtZW1vcnkgc3RyZWFtcyB3b3VsZCB0cnVuY2F0ZSBvdGhlcndpc2Uu ICovCisgICAgICAgIGlmIChsc3hfc2Vla2koZnQsIGVuZCwgU0VFS19TRVQpICE9IDApCisgICAg ICAgIHsKKyAgICAgICAgICAgIGxzeF9mYWlsX2Vycm5vKGZ0LCBTT1hfRU9GLCAiQ2FuJ3Qgc2Vl ayB0byBlbmQgb2Ygb3V0cHV0IGZpbGUgYWZ0ZXIgcmV3cml0aW5nIC53YXYgaGVhZGVyLiIpOwor ICAgICAgICAgICAgcmV0dXJuIFNPWF9FT0Y7CiAgICAgICAgIH0KIAotICAgICAgICByZXR1cm4g KHdhdndyaXRlaGRyKGZ0LCAxKSk7CisgICAgICAgIHJldHVybiBTT1hfU1VDQ0VTUzsKIH0KIAog LyoKLS0gCjIuMjUuMQoK --_004_CO1PR15MB4890D41F45E417C614B82ED98CB19CO1PR15MB4890namp_ Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline --_004_CO1PR15MB4890D41F45E417C614B82ED98CB19CO1PR15MB4890namp_ Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ SoX-devel mailing list SoX-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sox-devel --_004_CO1PR15MB4890D41F45E417C614B82ED98CB19CO1PR15MB4890namp_--