Parimala Shankaraiah, the curious tester, wrote a blog entry on hiring testers. In it, she references a questionnaire for interviewing new testers. Taking a closer look into it, I felt the urge to answer the questionnaire. Follow me along while I visit the 25 questions there, though I’m sure it’ll take me more than the 1.5 hours indicated there, so I’m sure to get some penalty points deducted.

The introduction is crisp. But there is a serious flaw in the description. The scores are listed up as followed:

• Score less than 50% – Fail
• Score 50% to 65% – Trainee Tester
• Score 65% to 80% – Tester
• Score more than 80% – Senior Tester

Any tester aware of boundary analysis will mark this as ambiguous. Is a score of 50% now a failure or a trainee tester? What about 65% and 80%? Read A Practitioner’s Guide to Software Test Design for more on the topic.

So, now after mentioning this underlying flaw of the questionnaire, let’s take a look on the questions and answer them.

1. What statement do you consider to be most important and why?
a) Testing has the primary intent of showing the system meets the users needs.
b) Testing has the primary intent of finding faults

c) The primary intent of testing is to inform a business decision. That said, testing is gathering information in order to make an informed decision. This may consist of showing that the system meets the users needs, or to find important faults. So, in a way, a) and b) are both correct and wrong. The proper answer here is “it depends”. It depends on the mission of the testing activities for the project at hand.

2. You have run all your tests and they all pass. Is this good news or bad news?

This depends. It’s good news for me if I may now start to explore the product since the scripted tests are finally over. On the other hand if I am out of energy to think about a new test to run while all my previous tests have passed, this may be a bad sign if I’m exhausted by the testing activities of the past overtime period.

3. What would you do if you were asked to test a system which is unfamiliar to you has out-of-date or inadequate documentation?

I would ask the customer, the developer of the system as well as anyone that worked on the project about more information about the product. Based on the answers to these questions, I would try to define my mission for the testing activities I’m asked to do, and clarify this mission with the one who asked me to test that system.

4. In running a test you find the actual result does not match the expected result – what would you do?

I take a closer look on my oracle and the system I test. In case my oracle fails in this particular situation, I change my expectations; in case the system is wrong, I file a bug report – either using the bug tracking system or consulting the developer directly, depending on how critical the defect is.

5. Do you consider positive or negative testing to be most important or trying to break the system – and why?

The answer to this question heavily depends on the circumstances of the testing activities, the project at hand, and the mission of the tester on the project. At times it’s more important to have a passing verification check, at times it’s critical to have a robust system and showing this by more negative tests. There is no context-free “most important” to be answered in general. Applying Michael Bolton’s Relativity Rule (X is X to some person (at some time)), in every project I have to find an answer to the question “if it’s most important, to whom is it most important right now?”. Depending on the answer to that question, I can derive the approach taking.

6. How would you define a good test?

Good for whom or for what? For me, a good test consists of a sapient thinking up a critical condition in the program that might harm the stakeholder of the project or the program at some time. This might include today’s situation as well as some (limited) future or past situations.

7. You have been assigned to test the new Triangle Determination Application (see screen shot below).

As you can see the screen consists of three text fields and a single button. The user is expected to enter an integer value into each of the three text fields. Upon hitting the OK button the program will print a message in a separate dialog box stating whether the triangle is scalene (all sides are different lengths), isosceles (two sides are the same length), or equilateral (all three sides are the same length).

Write a set of test cases (i.e. specific sets of data) that you feel would adequately test this program. Write the tests so that someone other than you can run them.

I have troubles understanding your definition of test case here. Can you help us derive a common understanding of this term, so that I can provide you what you asked for, please? In addition, what does the term “integer” mean to you. Who’s the user going to exercise the application?

8. In testing the above application you identify what you believe to be a fault – instead of printing the message concerning the type of triangle in a separate dialog box the application is printing the message in the space between the 3 text fields and the OK button. What should your next step be (answer and state why)?
a) Continue testing to the end of the script, and then report the bug.
b) Stop testing, report the bug immediately, then continue alternative scripts
c) Stop testing, report the bug and await a fix.
d) Continue testing and report the bug later, along with those found in other scripts

As I’m working from a test charter, I write down a note about the behavior, and continue to explore the application. So, none of the above holds.

9. You have raised a fault, but Development are unable to reproduce it. What should your next step be? (Give answer and state why)
a) Let development sign off the bug as not reproducible.
b) Sign off the bug yourself as not reproducible.
c) Tell development the bug definitely exists and you will not pass it unless fixed.
d) Re-test and upon confirmation provide more detailed information to Development, talking them through each stage if necessary.

Again, it’s e) I ask my superior if bug pin-pointing is within the scope of my testing activities and help to clarify my mission. Considering the severity of the bug and the project at hand, I might then take one of the given answers depending on what serves the project.

10. Scenario:
You have two sets of tests to run on the new version of the software.
Test Set 1: a test set to provide confidence that software has not regressed from the previous version.
Test Set 2: a detailed test set to investigate potential faults in the new release of software.
Having run test set 1 you discover a number of faults in the new version of software – what do you do?

I report my findings to the project stakeholders in order to issue an informed decision about the further actions of my testing and the project at large.

11. Draw and explain the ‘V’ Model and how testing fits into the Development Lifecycle. Indicate on the model where you would design your tests.

The V-Model is Analyze – Design – Implement – Look for whom to blame. Testing starts right at the beginning of the project when the requirements are trawled and analyzed and continuous throughout the whole project. Therefore I would design my tests throughout the whole project lifetime.

12. Describe the stages of testing and what the objectives are at each stage.

There are no stages in testing. Different activities need to be used during different times of the project. Especially as the work regresses back in the V-Model lifetime of the project, previously conducted testing activities may become vital tot he project again. In general the objective of any testing activity is to help project stakeholders make well-informed decisions about the further course of the project.

13. Explain what you understand by the terms:
Regression Testing and Re-Testing

Regression Testing is a sapient activity evaluating the product for flaws in previously existing business value. As this activity is boring, it’s better to convert is to regression checking. Re-Testing is the act of re-visiting previously found bugs, and evaluating the product with a bugfix whether the bug was fixed or not.

14. Scenario:
You have planned to run 600 tests on your own. Each test will take approximately 10 minutes to run. Your manager has told you that you must complete these tests within one week. What would you do?

I feel tempted to run as many tests as possible from my previously determined 600 tests. I resist this temptation, approach my manager, and negotiate to follow a session-based chartered approach to testing. After having compiled my 600 tests into test charters for the next week, I start to work in time-boxed sessions on these and report my findings back to my manager.

15. Do you consider testing tools to be valuable during the testing process – why/why not?

This depends on the tool. There are time-consuming tools in use in our industry today as well as time-saving tools. I prefer the latter one, while I might end up being forced to use the former ones. If this is the case, I try to make use of the time-consuming tools as lean as possible, so that I may spent more and more time testing, rather than debugging the testing tool. This should also explain the whys and why-nots.

16. List 3 test tool categories and describe what each can do.

Test automation tools help to automate checking related activities. Bug tracking system help to track issues and defects over the project lifetime. Most commercial test documentation tools help some tool vendor to earn some money, while most of the time MS Word and Notepad served the purpose of documenting testing outcomes for me in the past.

17. Name 2 standards that refer to testing

Sorry, I don’t care about testing standards as I find them thought-limiting.

18. How would you test these requirements:
a) The system must be user-friendly
b) The system must be easy to install
c) The following response times are to be achieved with the new system:

• Initial loading of the web application must be achieved within 3 seconds
• Updating of the information on the web page must be no more than 5 seconds

I’d consult a user survey for the first two points, and use a performance testing approach after clarifying the performance metrics in terms of how often this should be met.

19. Why do you consider testing to be necessary?

Requirements solve the problem of knowing what to build. Testing solves the problem of answering the question whether the problem of the stakeholder or user is indeed solved by the implemented software system.

20. A hotel telephone system can perform 3 functions:

• Call another hotel room by entering a room number (201 to 500)
• Call an external line by entering a 9, followed by the number
• Call various hotel services
  • 0 = Operator
  • 7 = Room Service
  • 8 = Reception

Write a set of test cases to adequately test this telephone system

Despite the discussion on the test cases that we had earlier, what do you consider adequately here? What’s the cirticality of the project? May any humans be injured, if they can’t call emergency calls by the system? How many simultaneous users should the system be able to serve? What various users will use the system? How does a telephone in your business look like? How many phone calls my the receptionist serve at once? In addition to the answers to these question, please assign me a project stakeholder who can work with me to clarify any further open questions that will arise.

21. Describe what you understand about the term “Static Testing” and list 3 static testing techniques.

Static testing is the static analysis of some source code. Code reviews, inspections, and walkthroughs are techniques used within static testing.

22. How would you prioritise your tests (list 5)?

In general, I would start to explore the product, take a look over my session notes and think about automating some of the tedious tasks. The priority for the types of tests executed are derived based upon the project at hand. This may include security testing, failure tests, and the like. Depending on the mission of the testing activities my priorities are derived. There’s no all fits answer here.

23. Scenario:
You are testing 2 programs and have 3 weeks to test them both. Having run all of your tests on both programs you finish testing within 2 weeks. You need to decide which of the 2 programs you would re-visit and run further tests against. Choose which program you would re-test (can choose only one!) – and state you reasons:

Program A
Programmer: A
Complexity Level: 2
Lines of Code: 2000
Number of tests: 100
Number of bugs found: 10
(1 high severity, 3 medium & 6 low)

Program B
Programmer: B
Complexity Level: 2
Lines of Code: 2000
Number of tests: 100
Number of bugs found: 50
(10 high severity, 25 medium & 15 low)

It depends. In the first program I might have had a bad week, so re-visiting this product may be the best thing to do. In the second product there may be further bugs lurking, but follow-up testing might not add any additional information gathered from the product. Depending on the particular programmer, I negotiate my preferences with both the project manager and the programmers to make an informed decision for the underlying projects.

24. An ATM has been specified to work in the following way:
Enter a card and if the card is invalid reject the card and exit system. If it is a valid card then enter a PIN number. Check to see if the PIN is invalid – if it is then display a message ‘invalid pin number, please re-enter’. If 3 attempts are made with an invalid pin then the machine keeps the card. If it is a valid PIN then the user can select one of the following transactions:

• Cash Withdrawal without receipt
• Cash Withdrawal with receipt
• Balance Enquiry
• Statement request
• Cancel

What tests would you produce to test this application? State any assumptions when testing

What’s my mission here? Shall I test the hardware or the software? Test cases might be either distinguished by a test charter or by a decision table from my perspective. I negotiate the approach taken derived from the project’s needs.

25. The following is an extract from a fault log, write down any potential problems or omissions with this: [left out intentionally]

There is no triaging information mentioned in the bug report. This means that the programmer and the tester are left on their own to fight out who is right and who is wrong. The tester seems to be emotionally attached to her report. The project manager needs to steer this situation and enable successful communication among team members in this case. A meeting with the product responsible and the programmer and tester may help clarify the situation here.

The bug report does not mention a summary and reproduction steps. The tester should mention the used oracle, i.e. referring to the document or the user expectations. The tester should mentioned what he tried to do and how he tried it in a crisp way, so that the bug report is easily readable, digestable and easy to triage.