Modelling methodology for assessing the impact of new technology on complex sociotechnical systems

Abstract

Developing complex sociotechnical systems often involves integrating new technology into existing systems by applying systems engineering processes. This requires an understanding of the problem space and the possible impact of the new technology. Systems engineering uses modelling to explore the structural, functional, and operational elements of the problem and solution space (Hitchins 2008). Historically, systems engineering has however struggled with complex sociotechnical systems projects, as it cannot cope with the dynamic behaviour of complex sociotechnical systems. The hypothesis of this thesis is that addressing the contribution of humans performing work in a complex, constrained and dynamic environment using modelling will result in a better understanding in the analysis phase; it should also lead to improved requirements, designs, selection of technologies, and implementation strategies, enabling sociotechnical systems to cope with complex operating environments. A sociotechnical system consists of humans applying technology to perform work through processes within a social structure (organisation) aimed at achieving a defined objective (Bostrom & Heinen 1977, Walker et al. 2009). Work can become complex due to non-linear and dynamic interaction among the people themselves, among people and technology, as well as among people and the environment. Complexity may lead to “wicked and messy” problems, as many unintended or unpredicted consequences may be experienced. The new technology may also lead to new task possibilities that evolve user requirements (Carroll & Rosson 1992). Systems engineering, as developed in the 1950s, forms the basis of developing systems, including sociotechnical systems. Classic systems engineering processes assume that problems can be isolated and decomposed, making the development of complex sociotechnical systems difficult. One way to improve the success of systems engineering is to ensure that the problem to be solved is properly understood. Analysis of the problem and solution space involves capturing and modelling the knowledge and mental models of the stakeholders, to support understanding the system’s requirements. A good description of the problem situation through a model is the first step towards designing and developing a solution. The aim of this study is to develop and demonstrate a modelling methodology for complex sociotechnical systems, in support of the systems engineering process. The two approaches used in the modelling methodology are cognitive work analysis and system dynamics. Cognitive work analysis is a framework for analysing the way people perform work in an organisation, while taking the environmental constraints into consideration. The outputs of cognitive work analysis are constructs or models that capture the structure of the problem. Functions provided by different technological elements are linked to the functional requirements of the system, to achieve its purpose (Lintern 2012). However, cognitive work analysis is limited in investigating the dynamic effect of decisions and policies on the system (Cummings 2006). The dynamic behaviour of complex sociotechnical systems can be analysed using system dynamics, which uses the structure of the system in simulation. System dynamics analyse the effect of feedback and delays on operating the system, as a result of decisions based on policies (Sterman 2000). The design science research framework, which also supports the research design of this thesis, is used to implement the modelling and structure the methodology. Design science research aims at creating technology for a human purpose, unlike the natural sciences, which are geared towards attempting to understand and define reality (March & Smith 1995). The proposed methodology is demonstrated in a case study using modelling and analysis of the impact of a new collaboration technology on command and control systems. Command and control is a good example of a complex sociotechnical system, as humans use technology to assemble and analyse information for situation assessment in support of planning operations (Walker et al. 2009). These systems are also used to control the successful implementation of plans in constrained and variable operating environments. The modelling methodology is demonstrated by modelling and assessing the effect of a new command and control technology for border safeguarding operations, anti-poaching operations and community policing forums. The new technology to be implemented in these complex sociotechnical systems is called “Cmore”. It is a web-based collaboration system that uses smartphones to capture information and track users. Even though the three demonstrations constitute similar systems, the different contextual situations result in diverse behaviour and issues to be investigated. The demonstrations centre on the functions of situation awareness and decision support. The different output models for the command and control systems are used in system dynamics simulations to assess the effect of new technology on the operating and effectiveness of a system. The case studies demonstrated that the modelling methodology support learning about the implementation of a new technology in various complex sociotechnical systems. The developed models and constructs also supported developing evaluation templates during the planning of experiments through identifying key issues. The system dynamics simulations used parametric inputs to investigate the behaviour of the system. In most cases, the simulation outputs identified interesting and counter-intuitive behaviour for deeper assessment. The community policing forum case study also gathered qualitative empirical evidence on the system's behaviour, during a field experiment. The outcomes are compared with the models and simulation outputs to improve the system behavioural models. The learning and improved understanding of the complex sociotechnical system behaviour gained through the modelling methodology, demonstrated its utility.