A challenge for many projects, especially in research, is describing the quality criteria and distinguishing it from scope. A really big clue that you're talking about quality is when those subjective words start happening - better, faster, smaller, less, more, cheaper.
This is the second of a short series of posts about quality management, to include some of my take-aways from teaching as part of the Mohawk College and CARA Research Administration Certificate program, as well as experiences in my own work related to quality.
Be clear about the project and objectives: As discussed in “know thy project,” we need to be clear about what the project is and then set appropriate quality criteria for that. If we consider the preparation and submission of a research proposal as our project, the goal is to submit a complete and high-quality proposal by the deadline. Getting funded is NOT the goal of the proposal project. Getting funded is the strategic question (the problem we’re trying to solve is that we need funding for the research) and the applying for funding is one of the ways we attempt to address that. In measuring the quality of the proposal work, we might consider helpful reviewers who comment on the likely fundability (a quality attribute) of the proposal. We may ask an AI tool to evaluate the readability of the proposal. We can assess the proposal’s alignment with the competition’s assessment criteria and make our own assessment as to how well the proposal aligns with that. But once the proposal is submitted, that’s the end of the proposal project.
If we look across multiple proposals, and so consider our proposal development process, we can review our overall success rate in funding as a measure of the quality of all proposals or our proposal process, but that is not a measurement of the quality of the proposal submission process itself unless we identify a specific element of that process that is contributing to the (good or poor) success rate.
Project vs product: In research (like in most projects) we can have a high quality of performance and still have a poor quality of result. This can be related to the point above – being clear about what our project is – but also includes considering what we are measuring in quality control and ensuring that those elements relate to the quality we want.
For example, a clinical trial can be about measuring the effectiveness of a treatment or intervention. Ultimately, whether or not the treatment is implemented and used can be affected by factors other than effectiveness (ex. cost, reliability, ease of use, comprehension, demand, competition). The clinical trial can be successful, with sufficient participants and demonstration of safety and efficacy, but the treatment may not be used and so is not successful. This is not a failure of the clinical trial project, but a failure of the larger project of implementation of the treatment.
Quality standards: speaking of clinical trials, a student recently asked about identifying the "objective standard" for quality in such projects. For these projects, quality might be defined as, "as good as it needs to be to get regulatory approval for the product." Therefore, the objective standard is whatever the regulator requires. As well, quality can be very closely related to scope, as the targeted number of participants completing the trial (the scope) reflects the power (quality) of the study to demonstrate safety or efficacy. Often with clinical trials, there is also an element of grade, where the aim is to show that the new intervention is "better than" the current one; part of the quality standard here would be, "how much better?"
Checklists: These are excellent control tools for both inspection and prevention, depending on when and how they are implemented. A checklist that is used each time a process is followed is an example of prevention – ensuring no required element or step is missed before proceeding to the next. Prevention checklists usually require sign-off of the checklist before proceeding and become a part of project documentation. A checklist used to review work already completed, such as confirming the quality of a product, would be an inspection approach. (see also The Checklist Manifesto, by Atul Gawande.)
Document and data control: Document control is a great (and often overlooked) tool for quality control by prevention - as in, preventing errors from entering the system. Documents can be accessible – so people can get the information they need – but still be controlled to prevent unauthorized changes. Documents that lots of people access but only a few should edit can be "read only" and password protected to prevent someone making an undocumented change that cannot be undone later. For control documents such as expense logs and participant registries, making archive copies at regular intervals can also preserve the integrity of project information.
Similarly, data management principles such as a FAIR allow for “open” science and data while still ensuring sufficient control of the data to maintain its integrity and quality. Establishing what the controls will be for data can ensure maximum openness and sharing with the necessary protections for confidentiality, intellectual property, and ownership.
Publications and citations: These can be measures of quality if those are defined as quality criteria and metrics within the project. In some studies or trials, a peer-reviewed publication is a desirable requirement but not a mandatory one (especially since acceptance of a publication is beyond the control of the project), so may not be an appropriate measure of success. Similarly, social media connects can be a measure of the quality of outreach or knowledge mobilization.
More recent consideration of citations as metrics in research and science question the “gold standard” nature of these. There are many factors that contribute to citation rates that have nothing to do with quality. This article (Dag et al, 2019) says: “Research quality is a multidimensional concept, where plausibility/soundness, originality, scientific value, and societal value commonly are perceived as key characteristics.” Other issues have emerged with peer-reviewed publications in recent years, including lack of reproducibility, conflicts of interest, equality of accessibility, and the mutability of electronic publications (including the difficulties of managing amendments and retractions). There is also the issue of representation in the publication space, as described in this piece on Citation Justice.
Similarly, the social media spaces can be challenging as measurable indications of quality. The popularity and proliferation of an online post often has little to do with how good, real, or true it is. A tweet may get a lot of views, likes, and traffic, but if the replies and reposts are negative, or a post goes viral for all the wrong reasons, then the social media measurements would not indicate quality but rather the opposite.
As discussed before, focusing on good enough doesn’t mean “not excellent”. In any project, we should strive for the necessary level of quality, and curb any tendency to try for perfection. I know I'd rather be on a good-enough successful project than a perfect failure.
What are your experiences with managing project quality? What about balancing quality and scope? Please comment below or email me at firstname.lastname@example.org with your questions and feedback.
Interested in more on this or other topics? Check out my upcoming webinars and presentations at www.lyricmgmt.com. Follow me on Twitter and LinkedIn for the latest on these and other topics. You can also complete this brief survey to join my mailing list to receive a monthly newsletter with blog posts and webinar schedules, and to be entered in a monthly draw for a prize.
Who is Robyn?
My career as a research project manager is rewarding, dynamic, challenging, and fun. I'm looking forward to sharing my knowledge and experience in communication, organization, and common sense approaches in research management and leadership, and to enabling others to learn and grow in this exciting career.