Download Subjective tests - Telecommunications Industry Association

Telecommunications Industry Association
TR41.3.3-11-05-05
Document Cover Sheet
Project Number
PN-3-4705.120-RV1
Document Title
TIA-920.120-A Speakerphone Subjective Tests Proposal
Source
Microsoft Corporation
Contact
Name:
Hong Sodoma
Phone:
425-707-5380
Complete
Address:
One Microsoft Way,
Fax:
425-936-7329
Redmond, WA 98052
Email:
[email protected]
Distribution
Intended Purpose
of Document
(Select one)
TR-41.3.3
X
For Incorporation Into TIA Publication
For Information
Other (describe) -
The document to which this cover statement is attached is submitted to a Formulating Group or
sub-element thereof of the Telecommunications Industry Association (TIA) in accordance with the
provisions of Sections 6.4.1–6.4.6 inclusive of the TIA Engineering Manual dated March 2005, all of
which provisions are hereby incorporated by reference.
Abstract
This document proposes a set of subjective tests to help validate audio quality for speakerphone
devices. Due to sophisticated DSP algorithms, today’s speakerphone performance is hard to
measure with objective measurements only. It is often needed to evaluate the quality subjectively
and in practical scenarios. This proposal provides some guidelines for such subjective
evaluations. We propose to add such recommendations to TIA-920.120-A to help designers and
test labs evaluate audio quality of the speakerphones under normal usage conditions.
(Submitted for discussion on 8/11/2011)
Subjective tests
All devices shall be evaluated subjectively in a regular VoIP calls. The goal is to identify any obvious
problems that may have been missed in the objective tests. The test should be run under normal usage
condition and some stress conditions. If the device itself does not support VoIP calls and requires a
softphone client, the tests below should be run when the device is connected to a softphone client that it is
intended to be used with. For simplicity, we use “VoIP device” to refer to both scenarios.
Page 1 of 6
Telecommunications Industry Association
TR41.3.3-11-05-05
Normal usage conditions:
 Room reverberation time: 0.5 to 0.75 seconds (Averaged over 100 to 7000Hz)
 Room ambient noise level: 43-45dBA
 Human talker speech level: 60 to 70dB SPL measured at 0.5 meters from the mouth.
Examples of stress conditions:
 Room reverberation time: 0.5 to 0.75 seconds (averaged over 100 to 7000Hz)
 Room ambient noise level: 50-52dBA
 Human talker speech level: 55-60dB or 80-85dB SPL measured at 0.5 meters from the mouth.
Acoustic echo cancellation (single-talk)
Purpose
This test is to make sure that there are no echo leaks or distortion in speech for this VoIP device because of
any problems which couldn’t be detected by the objective tests outlined in the previous sections.
Requirement
There shall not be any echo leaks during single-talk operation. There shall not be any signal distortion of
the near-end speech signal.
Test Procedure
This test requires two people. The test should be run under normal usage conditions defined above.
1. Review the DUT user manual for the usage scenarios that are supported. Person 1 shall place the
product in a real meeting room/office as instructed in the user manual. If there isn’t any
specification then similar positioning shall be used as described in Figs. 5 and 6 of IEEE Std.
1329-1999.
2. Person 2 shall be located in a quiet office environment with a high-quality reference headset.
3. Set up a VoIP phone call between the two ends.
4. Ask Person 1 to set the render volume to the preferred listening level
5. Start the test by having Person 1 and 2 to take turns speaking.
6. Person 2 shall listen to whether his own voice or a distorted version of his own voice is echoing
back. In case of echo leaks or any leaks of echo perceived as distorted noise is observed by Person
2, then the DUT fails this requirement. Additionally, Person 2 shall pay attention to the voice
quality of Person 1’s voice.
7. Repeat steps 5 and 6 for the maximum render volume supported by the device.
8. Repeat steps 4 through 6 with Person 1 moving away from the DUT to the border of the maximum
supported microphone pick-up range. Also, the render volume shall be adjusted such that at that
range the render volume is acceptable for a normal voice conversation.
9. Repeat step 8 for the maximum render volume supported by the device.
Acoustic echo cancellation (double-talk)
Purpose
This test is to make sure that it is easy for the near-end user and the far-end user to interject while the other
end is talking. Also it should make sure that there are no echo leaks in double-talk situations.
Page 2 of 6
Telecommunications Industry Association
TR41.3.3-11-05-05
Requirement
There shall not be any echo leaks during double-talk operation. Slight signal distortion of the near-end
speech signal is acceptable. The user of the DUT shall be able to interrupt the far-end participants and also
the far-end participants shall be able to interrupt the near-end user who is using the DUT.
Test Procedure
This test requires two people. The test should be run under normal usage conditions defined above.
1. Review the DUT user manual for the usage scenarios that are supported. Person 1 shall place the
product in a normal meeting room/office as instructed in the user manual. If there isn’t any
specification, then similar positioning shall be used as described in Figs. 5 and 6 of IEEE Std.
1329-1999.
2. Person 2 shall be located in a quiet office environment with a high-quality reference headset.
3. Set up a VoIP call.
4. Ask Person 1 to set the render volume to the preferred listening level
5. Start the test by taking turns to speak for 30 seconds. Gradually try more and more double-talk and
interrupting each other. In the extreme case of consistent double-talk, one person may count
numbers and the other person may recite alphabets or read continuously.
6. Person 2 shall listen to whether his own voice or a distorted version of his own voice is echoing
back. In case of echo leaks or any leaks of echo perceived as distorted noise is observed by Person
2, then the DUT fails this requirement.
7. Person 2 shall evaluate if it was possible for him/her to interrupt Person 1. If it was not possible to
interrupt easily then the DUT fails this requirement.
8. Person 1 shall evaluate if it was possible for him/her to interrupt Person 2. If it was not possible to
interrupt easily then the DUT fails this requirement.
Test 7 and 8 should include different periods of silence prior to interruptions. The interrupting
participant should try interruptions after short periods of silence (1-2 seconds) and after longer
periods of silence (~10 seconds).
9. Repeat steps 5 through 8 for the maximum render volume supported by the device.
10. Repeat steps 5 through 8 with Person 1 moving away from the DUT to the border of the maximum
supported microphone pick-up range. Also, the render volume shall be adjusted such that at that
range the render volume is acceptable for a normal voice conversation.
11. Repeat step 10 for the maximum render volume supported by the speakerphone device
Send loudness
Purpose
This requirement should ensure that the near-end user’s voice has sufficient loudness when being sent to
the far end. This test should also reveal any problems in the interaction of any built-in gain control
algorithms in devices with softphone algorithms.
Requirement
The near-end speaker’s voice shall be loud enough for the far-end user. In case of a speakerphone, this
requirement shall be met when the near-end user moves to the border of the supported microphone pick-up
range.
Test Procedure
This test requires two people. The test should be run under normal usage conditions defined above.
Page 3 of 6
Telecommunications Industry Association
1
2
3
4
5
6
TR41.3.3-11-05-05
Refer to the DUT user manual for the recommended usage scenarios. Person 1 shall place the
product in a normal meeting room/office as instructed in the user manual. If there isn’t a
specification then similar positioning shall be used as described in Figs. 5 and 6 of IEEE Std.
1329-1999.
Person 2 shall be in a quiet office environment with a high-quality reference speakerphone that
has passed TIA-920.120A. Note that for this test no headset shall be used as usually it will require
more effort for Person 2 to listen to Person 1 on a speakerphone. The render volume on the
speakerphone shall be maximized.
Set up a VoIP call.
Start the test by Person 1 speaking into the DUT at the recommended usage position.
Person 2 shall see if he can easily understand Person 1. If Person 1 sounds too soft, the DUT fails
this requirement.
Repeat step 5 with person 1 moving away from the DUT to the border of the maximum supported
microphone pick-up range. The maximum supported microphone pick-up rage is typically
recommended by the device manufacturer.
Send distortion
Purpose
This requirement helps ensure that when the near end user is speaking in a normal or slightly elevated
voice, the far end listener does not perceive distortions (possibly due to microphone clipping).
Requirement
When the near-end is talking in a normal voice (about 65dBA measured at 0.5meters from the near end
speaker), the far-end should not hear clipping distortions. This should be true even when the near end slight
elevates his voice, (about 76dB SPL measured at 0.5 meters from the near end speaker).
Test Procedure
This test requires two people.
1 Refer to the DUT user manual for the recommended usage scenarios. Person 1 shall place the
product in a normal meeting room/office as instructed in the user manual. If there isn’t a
specification then similar positioning shall be used as described in Figs. 5 and 6 of IEEE Std.
1329-1999.
2 Person 2 shall be in a quiet office environment with a high-quality reference headset.
3 Set up a VoIP call.
4 Start the test by Person 1 speaking into the DUT at the recommended usage position at normal
voice.
5 Person 2 shall not hear any clipping distortions.
6 Person 1 can then speak in louder voice as specified in requirement. Persona 2 shall not hear
clipping distortions.
Page 4 of 6
Telecommunications Industry Association
TR41.3.3-11-05-05
Send noise
Purpose
This requirement should ensure that there isn’t any excessive noise transmitted from the DUT to the far-end
participants. Also this test will make sure that there aren’t any annoying noise level changes. This is often
referred to as noise pumping.
Requirement
The near-end speaker’s voice shall be transmitted without any excessive background noise. If the near-end
user is silent then there shall not be any objectionable noise be sent by the DUT.
This test should be run in both normal and stress conditions.
Test procedure
1.
2.
3.
4.
5.
Refer to the DUT user manual for the recommended usage scenarios. Person 1 shall place the
product in a normal meeting room/office as instructed in the user manual. If there isn’t a
specification then similar positioning shall be used as described in Figs. 5 and 6 of IEEE Std.
1329-1999.
Person 2 shall be located in a quiet office environment with a high-quality reference headset.
Set up a VoIP call.
Start the test by Person 1 speaking into the DUT at the recommended usage position. Person 1
shall also have silence periods of 10 seconds or more in between his speech.
Person 2 shall listen to whether he or she can notice any excessive noise. Also Person 2 shall listen
for any noticeable noise level change.
Page 5 of 6
Telecommunications Industry Association
Page 6 of 6
TR41.3.3-11-05-05