Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Telecommunications Industry Association TR41.3.3-11-05-05 Document Cover Sheet Project Number PN-3-4705.120-RV1 Document Title TIA-920.120-A Speakerphone Subjective Tests Proposal Source Microsoft Corporation Contact Name: Hong Sodoma Phone: 425-707-5380 Complete Address: One Microsoft Way, Fax: 425-936-7329 Redmond, WA 98052 Email: [email protected] Distribution Intended Purpose of Document (Select one) TR-41.3.3 X For Incorporation Into TIA Publication For Information Other (describe) - The document to which this cover statement is attached is submitted to a Formulating Group or sub-element thereof of the Telecommunications Industry Association (TIA) in accordance with the provisions of Sections 6.4.1–6.4.6 inclusive of the TIA Engineering Manual dated March 2005, all of which provisions are hereby incorporated by reference. Abstract This document proposes a set of subjective tests to help validate audio quality for speakerphone devices. Due to sophisticated DSP algorithms, today’s speakerphone performance is hard to measure with objective measurements only. It is often needed to evaluate the quality subjectively and in practical scenarios. This proposal provides some guidelines for such subjective evaluations. We propose to add such recommendations to TIA-920.120-A to help designers and test labs evaluate audio quality of the speakerphones under normal usage conditions. (Submitted for discussion on 8/11/2011) Subjective tests All devices shall be evaluated subjectively in a regular VoIP calls. The goal is to identify any obvious problems that may have been missed in the objective tests. The test should be run under normal usage condition and some stress conditions. If the device itself does not support VoIP calls and requires a softphone client, the tests below should be run when the device is connected to a softphone client that it is intended to be used with. For simplicity, we use “VoIP device” to refer to both scenarios. Page 1 of 6 Telecommunications Industry Association TR41.3.3-11-05-05 Normal usage conditions: Room reverberation time: 0.5 to 0.75 seconds (Averaged over 100 to 7000Hz) Room ambient noise level: 43-45dBA Human talker speech level: 60 to 70dB SPL measured at 0.5 meters from the mouth. Examples of stress conditions: Room reverberation time: 0.5 to 0.75 seconds (averaged over 100 to 7000Hz) Room ambient noise level: 50-52dBA Human talker speech level: 55-60dB or 80-85dB SPL measured at 0.5 meters from the mouth. Acoustic echo cancellation (single-talk) Purpose This test is to make sure that there are no echo leaks or distortion in speech for this VoIP device because of any problems which couldn’t be detected by the objective tests outlined in the previous sections. Requirement There shall not be any echo leaks during single-talk operation. There shall not be any signal distortion of the near-end speech signal. Test Procedure This test requires two people. The test should be run under normal usage conditions defined above. 1. Review the DUT user manual for the usage scenarios that are supported. Person 1 shall place the product in a real meeting room/office as instructed in the user manual. If there isn’t any specification then similar positioning shall be used as described in Figs. 5 and 6 of IEEE Std. 1329-1999. 2. Person 2 shall be located in a quiet office environment with a high-quality reference headset. 3. Set up a VoIP phone call between the two ends. 4. Ask Person 1 to set the render volume to the preferred listening level 5. Start the test by having Person 1 and 2 to take turns speaking. 6. Person 2 shall listen to whether his own voice or a distorted version of his own voice is echoing back. In case of echo leaks or any leaks of echo perceived as distorted noise is observed by Person 2, then the DUT fails this requirement. Additionally, Person 2 shall pay attention to the voice quality of Person 1’s voice. 7. Repeat steps 5 and 6 for the maximum render volume supported by the device. 8. Repeat steps 4 through 6 with Person 1 moving away from the DUT to the border of the maximum supported microphone pick-up range. Also, the render volume shall be adjusted such that at that range the render volume is acceptable for a normal voice conversation. 9. Repeat step 8 for the maximum render volume supported by the device. Acoustic echo cancellation (double-talk) Purpose This test is to make sure that it is easy for the near-end user and the far-end user to interject while the other end is talking. Also it should make sure that there are no echo leaks in double-talk situations. Page 2 of 6 Telecommunications Industry Association TR41.3.3-11-05-05 Requirement There shall not be any echo leaks during double-talk operation. Slight signal distortion of the near-end speech signal is acceptable. The user of the DUT shall be able to interrupt the far-end participants and also the far-end participants shall be able to interrupt the near-end user who is using the DUT. Test Procedure This test requires two people. The test should be run under normal usage conditions defined above. 1. Review the DUT user manual for the usage scenarios that are supported. Person 1 shall place the product in a normal meeting room/office as instructed in the user manual. If there isn’t any specification, then similar positioning shall be used as described in Figs. 5 and 6 of IEEE Std. 1329-1999. 2. Person 2 shall be located in a quiet office environment with a high-quality reference headset. 3. Set up a VoIP call. 4. Ask Person 1 to set the render volume to the preferred listening level 5. Start the test by taking turns to speak for 30 seconds. Gradually try more and more double-talk and interrupting each other. In the extreme case of consistent double-talk, one person may count numbers and the other person may recite alphabets or read continuously. 6. Person 2 shall listen to whether his own voice or a distorted version of his own voice is echoing back. In case of echo leaks or any leaks of echo perceived as distorted noise is observed by Person 2, then the DUT fails this requirement. 7. Person 2 shall evaluate if it was possible for him/her to interrupt Person 1. If it was not possible to interrupt easily then the DUT fails this requirement. 8. Person 1 shall evaluate if it was possible for him/her to interrupt Person 2. If it was not possible to interrupt easily then the DUT fails this requirement. Test 7 and 8 should include different periods of silence prior to interruptions. The interrupting participant should try interruptions after short periods of silence (1-2 seconds) and after longer periods of silence (~10 seconds). 9. Repeat steps 5 through 8 for the maximum render volume supported by the device. 10. Repeat steps 5 through 8 with Person 1 moving away from the DUT to the border of the maximum supported microphone pick-up range. Also, the render volume shall be adjusted such that at that range the render volume is acceptable for a normal voice conversation. 11. Repeat step 10 for the maximum render volume supported by the speakerphone device Send loudness Purpose This requirement should ensure that the near-end user’s voice has sufficient loudness when being sent to the far end. This test should also reveal any problems in the interaction of any built-in gain control algorithms in devices with softphone algorithms. Requirement The near-end speaker’s voice shall be loud enough for the far-end user. In case of a speakerphone, this requirement shall be met when the near-end user moves to the border of the supported microphone pick-up range. Test Procedure This test requires two people. The test should be run under normal usage conditions defined above. Page 3 of 6 Telecommunications Industry Association 1 2 3 4 5 6 TR41.3.3-11-05-05 Refer to the DUT user manual for the recommended usage scenarios. Person 1 shall place the product in a normal meeting room/office as instructed in the user manual. If there isn’t a specification then similar positioning shall be used as described in Figs. 5 and 6 of IEEE Std. 1329-1999. Person 2 shall be in a quiet office environment with a high-quality reference speakerphone that has passed TIA-920.120A. Note that for this test no headset shall be used as usually it will require more effort for Person 2 to listen to Person 1 on a speakerphone. The render volume on the speakerphone shall be maximized. Set up a VoIP call. Start the test by Person 1 speaking into the DUT at the recommended usage position. Person 2 shall see if he can easily understand Person 1. If Person 1 sounds too soft, the DUT fails this requirement. Repeat step 5 with person 1 moving away from the DUT to the border of the maximum supported microphone pick-up range. The maximum supported microphone pick-up rage is typically recommended by the device manufacturer. Send distortion Purpose This requirement helps ensure that when the near end user is speaking in a normal or slightly elevated voice, the far end listener does not perceive distortions (possibly due to microphone clipping). Requirement When the near-end is talking in a normal voice (about 65dBA measured at 0.5meters from the near end speaker), the far-end should not hear clipping distortions. This should be true even when the near end slight elevates his voice, (about 76dB SPL measured at 0.5 meters from the near end speaker). Test Procedure This test requires two people. 1 Refer to the DUT user manual for the recommended usage scenarios. Person 1 shall place the product in a normal meeting room/office as instructed in the user manual. If there isn’t a specification then similar positioning shall be used as described in Figs. 5 and 6 of IEEE Std. 1329-1999. 2 Person 2 shall be in a quiet office environment with a high-quality reference headset. 3 Set up a VoIP call. 4 Start the test by Person 1 speaking into the DUT at the recommended usage position at normal voice. 5 Person 2 shall not hear any clipping distortions. 6 Person 1 can then speak in louder voice as specified in requirement. Persona 2 shall not hear clipping distortions. Page 4 of 6 Telecommunications Industry Association TR41.3.3-11-05-05 Send noise Purpose This requirement should ensure that there isn’t any excessive noise transmitted from the DUT to the far-end participants. Also this test will make sure that there aren’t any annoying noise level changes. This is often referred to as noise pumping. Requirement The near-end speaker’s voice shall be transmitted without any excessive background noise. If the near-end user is silent then there shall not be any objectionable noise be sent by the DUT. This test should be run in both normal and stress conditions. Test procedure 1. 2. 3. 4. 5. Refer to the DUT user manual for the recommended usage scenarios. Person 1 shall place the product in a normal meeting room/office as instructed in the user manual. If there isn’t a specification then similar positioning shall be used as described in Figs. 5 and 6 of IEEE Std. 1329-1999. Person 2 shall be located in a quiet office environment with a high-quality reference headset. Set up a VoIP call. Start the test by Person 1 speaking into the DUT at the recommended usage position. Person 1 shall also have silence periods of 10 seconds or more in between his speech. Person 2 shall listen to whether he or she can notice any excessive noise. Also Person 2 shall listen for any noticeable noise level change. Page 5 of 6 Telecommunications Industry Association Page 6 of 6 TR41.3.3-11-05-05