Supporting practitioners in developing effective decision support tools for natural resource managers

EXPLORING THE DECISION SUPPORT TOOL LANDSCAPE

To better understand the process and prevalence of participatory DST development, we operationalized the conceptual framework described above via an assessment of peer-reviewed literature. Specifically, we reviewed scholarly manuscripts published between 2008 and 2018 that addressed DSTs in the context of US agriculture and forestry. First, we developed a list of search terms to identify DSTs designed for the agricultural (including livestock and grazing land) and forestry sectors. We used the Web of Science search engine because it is sufficiently comprehensive of the topics of interest and has the machine-readable functionality to export and analyze search results. Given the diversity of fields that use DSTs (e.g., health, manufacturing, etc.), the majority of articles discussed topics outside our areas of interest or were removed due to our exclusion criteria. We ultimately included 23 DST papers relevant to our geographic and topical focus (see supplemental table 1). It should be noted that this review is not intended as a comprehensive treatment of the literature. We utilized the conceptual framework described in the previous section to guide our coding protocol, describing each paper’s methodology as well as how the authors addressed or failed to address key aspects of our conceptual framework (stakeholder assessment and engagement, problem identification, design and deployment, and evaluation).

How Are Stakeholders Being Identified and Their Needs Assessed? We analyzed the selected papers to determine if DSTs were requested by potential end-users, and if the authors described the network of stakeholders who were users or potential users of their tool. Nearly half of the papers used the problem that their tool is designed to fix as the justification for the tool, rather than explicitly describing the stakeholder demand for the tool. While this does not exclude the possibility that end users contributed to problem identification, it was not transparently obvious that this was the case. In contrast, several publications describing tools designed to support fire management decision making, primarily for the US Forest Service (Calkin et al. 2011a, 2011b; Drury et al. 2016; Ryan and Opperman 2013; Thompson et. al. 2015), were often explicit in their description of the need for the tools as articulated by the end user. Perhaps because many forest management DST developers were agency employees themselves, they were able to articulate the end-users’ needs more effectively than others.

All but one paper indicated the audience for whom their DST was intended, and just over two-thirds described stakeholder engagement processes at some point in the development process. The other articles were coded within the continuum of engagement, from no-engagement to contractual, consultant, collaborative, and collegial (see description above and reference to Meadow et al. [2015]). Only one article was coded as “contractual,” around a quarter were coded as “consultant” and “collaborative,” respectively, and three were coded as “collegial.” The articles that did not describe stakeholder engagement were categorized as “unclear” by our team, owing to the lack of available information, and these articles represent 30% of all the articles.

There was a great deal of inconsistency in how authors across all articles described their DST development and stakeholder engagement methods. In many cases, authors implied that they had engaged stakeholders, but did not provide further information (Calkin et al. 2011a, 2011b; Hunt et al. 2016; Ryan and Opperman 2013; Thompson et al. 2015). In other cases, more details were provided. For instance, Breuer et al. (2008) and Templeton et al. (2014) both describe the suite of AgroClimate Tools developed by the University of Florida. In these papers, the authors clearly described multiple modes of iterative engagement with target end-users in the development of the tools, including methods such as Sondeo surveys, focus groups, and regional workshops with relevant stakeholders who were connected to the University of Florida Cooperative Extension Service.

How Are Problems Identified? All articles we analyzed identified a main problem that their DST was designed to address. Tools were designed to address different types of natural resource problems, from mitigating dairy waste in Florida (see DynoFlo in Cabrera et al. [2008]) to reducing fungicide applications in strawberry (Fragaria × ananassa) production (see Strawberry Advisory System in Pavan et al. [2011]). Many of these tools were related to farming and ranching and were designed to deal with the types of complex decisions faced by producers (e.g., what crop should be planted under certain weather conditions or when to apply manure to reduce runoff risk). The majority of tools related to forestry were designed for forest industry professionals and wildfire and fuels managers who work with, or in partnership with, the US Forest Service.

In assessing how the problem was identified, we explored whether authors articulated how they defined the network of relevant stakeholders and whether they described the stakeholder needs or perspectives relative to the problem. Most papers clearly identified the potential stakeholders that might find value in using their tools. Most articulated how stakeholders understood the problem or associated problems that a tool might help them address. This was done either through formal assessment or through a review of the general background on the problem. However, 56% of the papers did not describe their methods for assessing whether stakeholders were necessarily requesting a DST to aid in their management of said problem.

How Are Decision Support Tools Designed and Deployed? Through analyzing how the authors describe beta testing, we sought to understand whether there was an iterative or regular engagement with users throughout the design and deployment process. Sixteen out of the 23 papers described some process for engaging stakeholders in this way. Again, this was not described with equal clarity or detail across papers. The process for prototyping and refining DSTs varied, from statements such as “Eighteen extension agents, researchers, consultants, and farmers provided feedback about the decisions support tool that utilize such forecasts during focus groups” (Templeton et al. 2014) to the relatively vague description in Easton et al. that simply says, “each of the tools described here was developed in response to specific users’ needs” (2017). The latter statement implied relevance to both problem identification and prototyping. Given this variability, it was not always possible to assess how engaged stakeholders were in the design, deployment, and subsequent improvements made to the tools themselves by reviewing a manuscript alone.

How Are Decision Support Tools Evaluated? An evaluation process was described in 52% of the articles, but in only 30% of articles was this evaluation considered purposeful (i.e., authors articulated a clear reasoning for why and how they implemented an evaluation). The methods for evaluation included surveys (22%), focus groups (9%), workshops and meetings (9%), and interviews (4%). In several cases, the evaluation was informal, or the methodology was unclear. Many authors suggested that their tools are critical for addressing a specific problem and were well-designed to help end-users improve their decision-making, while providing little evidence of what evaluation methods supported that conclusion. For example, in Calkin et al., authors state that “WFDSS has provided valuable real-time decision support to improve strategic decision making and communication by fire managers…and the development and application of WFDSS has helped the US Forest Service establish commitment to efficient and effective fire management with a strong focus on wildfire cost containment during a period of unprecedented fire activity” (2011a). While we have no reason to dispute this statement, the reader is provided little evidence for how authors arrived at this conclusion.

We also assessed whether authors evaluated the usefulness, usability, and usage of their tool. When evaluation was discussed in the manuscripts, authors described evaluation of terms of usefulness (39%) followed by usage (26%), and usability (17%). Few articles described more than one of these modes of evaluation. One exception was Jones et al. (2010) in which the authors describe a survey conducted to assess users’ assessment of usefulness and usability of their Decision Aid System as well as database tracking of tool usage. In most cases, however, authors focused on one aspect of evaluating a tool. For instance, in Pavan et al. (2011), the authors assessed the usefulness of the Strawberry Advisory System by working closely with three large commercial strawberry farms in Florida to provide iterative input on the development of the tool. For the purposes of this paper, we did not seek to evaluate the methodological rigor of a particular evaluation method (e.g., the use of a survey versus focus group) but rather sought to note whether or how the methods of evaluation were described.