Testing
The QA Role - What Is It Really?
When creating business cards, we have the freedom to choose any title we feel like reflects us best. Officially, many of us are “Quality Analysts”, or QAs. However, as we like to push back against the traditional “Quality Assurance” label, many of us picked alternative titles to make this point. Some of the titles are: Quality Alchemist, Question Authority or one we like in particular, Question Asker, because asking questions is a great way to achieve our goals. Yes, we’re Quality Advocates, building a culture of quality in the team and Quality Analysts, helping to build the right product in the right way. Asking questions is such an important part of our work, we decided to write an article about how everyone, regardless of role, can benefit from some well placed questions.
Despite your role - be a Question Asker
If we are trying to not only develop the product the right way, but also build the right product, the ability to ask questions is one of the most valuable skills we can all bring to the team. Questions help us learn, align and collaborate better. Questioning helps us to voice risks, share knowledge, and in general create better products.
Questions are a sign of a healthy work environment
In order to be able to ask questions, we have to establish safety within the team. Safety is required in order to answer these questions truthfully. If the opinions/concerns are silenced, asking questions may be a challenge. Also, in general, we can aim to learn how to ask questions so that even the roughest environments find them useful.
Questions aren’t checklists
Checklists often appear in ceremonies and documents like Definitions of Done, but the questions we will talk about in this article aren’t a checklist. They are meant to start discussions, get people thinking and lead to everyone learning more about the system we’re building. Socratic questioning involves asking questions to improve understanding and that’s how these questions should be used.
1. “What is our motivation for working on this?”
Peter Drucker said “there is nothing worse than doing the wrong thing well”. It’s important to make sure we’re building the right thing. What problem are we solving here? What value will this provide the user? How will this help us work faster? If we can’t answer any of those questions, we might need to take a step back and ask if this is really the best place to spend our time. Another side of this question is thinking about what happens if we don’t build this - what kind of loss will that be? If we are fine not building it, should we build it in the first place?
2. “How are we going to test this?”
Put the topic of testing upfront. Will we be able to validate that this works? What sort of tests will be written to build confidence in this feature? How will this change the shape of our Test Pyramid? Will the tests fit into our existing suites or will we need to build new test sets and pipeline steps? We tend to shy away from detailed test plans, but having this conversation sets us up to do effective Test Driven Development. This is also a great time to talk about edge cases and maybe even write an exploratory testing charter.
3. “What happens if this part breaks?”
We spend a lot of our time thinking about how to make things work, so this question helps us think about how things break. Draw out an architecture diagram and ask what happens if one of those services is unreachable. How about two of them? What happens if this configuration is set wrong? What happens if we send data with this field missing? What will the impact to the user be? This requires technical understanding to reason about failure modes and how likely they are and can be a great way to start analysing risks and focussing our efforts. If one part of our system could cause catastrophic failures if it goes down, maybe we need to think about a simpler design.
4. “How will we know if this fails?”
If we’re building something, we obviously plan on it working, but what happens if something goes wrong? Modern applications are complex systems with lots of moving parts, where failure of a computer you didn’t know existed can render your application unusable, so monitoring for failures is a key part of application design. If something causes our feature to stop working, how will we know? Will a dashboard clearly show something is wrong? Will someone be alerted? Do we need to change how we emit log messages to make failures detectable and observable? Developing with observability in mind helps us to question what signal we will get if something goes wrong. As in the question above, if we can’t detect/visualize failure easily, that might be a signal that we need to rethink the design.
5. “How does this change our attack surface?”
As we deal with valuable assets, like personal data or credit card details, we need to keep them both available for use and protected, and as we build out our systems, we need to think about how these assets are exposed to potential attackers. Could an anonymous user get sensitive data from this API? Are we encrypting this data in flight? Are we comfortable storing valuable data in multiple places? Security needs to be built in at every stage of development and this question can help us think about how we can do that.
6. “How confident are we that we can finish this story?”
We often discuss dependencies when analysing stories as a missing dependency can grind progress to a halt just as quickly as it started. Do we have dependencies that could stop us finishing this work? Maybe another team needs to deliver a feature before we can finish our work. How about our team, do we have enough knowledge within the team to get this done? This question can be difficult to answer with certainty, after all, we often don’t know everything we need until we start working on it, but asking the question gets the team thinking and helps create a safe space where team members can raise their concerns.
7. “When will we know if this is a success?”
So the tests are passing and we’ve deployed to production, but is this a success? Are our users actually using the new feature? Are they getting value from it? When and how will we know? Agile puts a lot of emphasis on fast feedback cycles, and this question helps us think about how we’re actually going to get feedback on our work and how long it will take. Going deeper, what does it actually mean for this story to be successful? If we are unable to define and measure the success measures of a story, this might be a good time to go back to the first question and ask what our motivation for working on this is.
For all of these questions, the discussion and follow up questions matter a lot more than just the question. It should be the start of a conversation, not the end of one.
These questions are just some of many examples of what you could ask your team. Some will make sense in your context, some won’t. In our teams we use questions as conversation starters and evolve them based on the needs of the team.
Question asking helps align the teams, voice over the concerns, evaluate the risks, and build quality into our products. Encourage questioning, and, be a question asker yourself, too.
Despite your role - be a Question Asker
If we are trying to not only develop the product the right way, but also build the right product, the ability to ask questions is one of the most valuable skills we can all bring to the team. Questions help us learn, align and collaborate better. Questioning helps us to voice risks, share knowledge, and in general create better products.
Questions are a sign of a healthy work environment
In order to be able to ask questions, we have to establish safety within the team. Safety is required in order to answer these questions truthfully. If the opinions/concerns are silenced, asking questions may be a challenge. Also, in general, we can aim to learn how to ask questions so that even the roughest environments find them useful.
Questions aren’t checklists
Checklists often appear in ceremonies and documents like Definitions of Done, but the questions we will talk about in this article aren’t a checklist. They are meant to start discussions, get people thinking and lead to everyone learning more about the system we’re building. Socratic questioning involves asking questions to improve understanding and that’s how these questions should be used.
7 helpful questions to ask your development team
Most of these questions we’d recommend to ask in the story refinement sessions, or story kick-off (it’s a ritual where business analyst, programmers, quality analyst align on the expectations on the story’s implementation to start or “kick-off” the development of it), but as well throughout the development process to get the team on board.1. “What is our motivation for working on this?”
Peter Drucker said “there is nothing worse than doing the wrong thing well”. It’s important to make sure we’re building the right thing. What problem are we solving here? What value will this provide the user? How will this help us work faster? If we can’t answer any of those questions, we might need to take a step back and ask if this is really the best place to spend our time. Another side of this question is thinking about what happens if we don’t build this - what kind of loss will that be? If we are fine not building it, should we build it in the first place?
2. “How are we going to test this?”
Put the topic of testing upfront. Will we be able to validate that this works? What sort of tests will be written to build confidence in this feature? How will this change the shape of our Test Pyramid? Will the tests fit into our existing suites or will we need to build new test sets and pipeline steps? We tend to shy away from detailed test plans, but having this conversation sets us up to do effective Test Driven Development. This is also a great time to talk about edge cases and maybe even write an exploratory testing charter.
3. “What happens if this part breaks?”
We spend a lot of our time thinking about how to make things work, so this question helps us think about how things break. Draw out an architecture diagram and ask what happens if one of those services is unreachable. How about two of them? What happens if this configuration is set wrong? What happens if we send data with this field missing? What will the impact to the user be? This requires technical understanding to reason about failure modes and how likely they are and can be a great way to start analysing risks and focussing our efforts. If one part of our system could cause catastrophic failures if it goes down, maybe we need to think about a simpler design.
4. “How will we know if this fails?”
If we’re building something, we obviously plan on it working, but what happens if something goes wrong? Modern applications are complex systems with lots of moving parts, where failure of a computer you didn’t know existed can render your application unusable, so monitoring for failures is a key part of application design. If something causes our feature to stop working, how will we know? Will a dashboard clearly show something is wrong? Will someone be alerted? Do we need to change how we emit log messages to make failures detectable and observable? Developing with observability in mind helps us to question what signal we will get if something goes wrong. As in the question above, if we can’t detect/visualize failure easily, that might be a signal that we need to rethink the design.
5. “How does this change our attack surface?”
As we deal with valuable assets, like personal data or credit card details, we need to keep them both available for use and protected, and as we build out our systems, we need to think about how these assets are exposed to potential attackers. Could an anonymous user get sensitive data from this API? Are we encrypting this data in flight? Are we comfortable storing valuable data in multiple places? Security needs to be built in at every stage of development and this question can help us think about how we can do that.
6. “How confident are we that we can finish this story?”
We often discuss dependencies when analysing stories as a missing dependency can grind progress to a halt just as quickly as it started. Do we have dependencies that could stop us finishing this work? Maybe another team needs to deliver a feature before we can finish our work. How about our team, do we have enough knowledge within the team to get this done? This question can be difficult to answer with certainty, after all, we often don’t know everything we need until we start working on it, but asking the question gets the team thinking and helps create a safe space where team members can raise their concerns.
7. “When will we know if this is a success?”
So the tests are passing and we’ve deployed to production, but is this a success? Are our users actually using the new feature? Are they getting value from it? When and how will we know? Agile puts a lot of emphasis on fast feedback cycles, and this question helps us think about how we’re actually going to get feedback on our work and how long it will take. Going deeper, what does it actually mean for this story to be successful? If we are unable to define and measure the success measures of a story, this might be a good time to go back to the first question and ask what our motivation for working on this is.
For all of these questions, the discussion and follow up questions matter a lot more than just the question. It should be the start of a conversation, not the end of one.
These questions are just some of many examples of what you could ask your team. Some will make sense in your context, some won’t. In our teams we use questions as conversation starters and evolve them based on the needs of the team.
Question asking helps align the teams, voice over the concerns, evaluate the risks, and build quality into our products. Encourage questioning, and, be a question asker yourself, too.
Disclaimer: The statements and opinions expressed in this article are those of the author(s) and do not necessarily reflect the positions of Thoughtworks.
