Tools for data collection

• Tracking data on the staff employed by the RI (scientists, technicians, managers, visiting students, trainees, etc.)
• Tracking the number of students/scientific staff/non-scientific staff trained by the RI
• Tracking the number of training programmes/ conferences/seminars organised by the RI
• Tracking information on wages, positions and career paths through specific tools or longitudinal surveys
• Statistics on long-life salary by sectors of employment (e.g. Payscale, Glassdoor, Eurostat databases, Eurofound surveys)
• Survey to identify the potential of the RI in terms of employment/usage opportunity for scientific users based on activities, working conditions and other relevant variables
• Tracking the number of publications in peer-reviewed journals or other scientific contributions (e.g. proceedings, papers, book chapters, books and technical reports) authored by RI’s staff and directly related to the RI activities
• Tracking number of citations on existing databases (Web of Science, Scopus, PubMed, arXiv, INSPIRE, etc.) and impact factors based on word search
• Tracking the cost of a scientist to produce a publication or organise/attend a conference
• Tracking number of successful funding applications of trainees
• Tracking the type, extent and quality of collaborations with other RIs and world leading teams by the number of joint projects, and volume, duration and outputs of visits, etc.
• Survey to identify the prestige of the RI as a training facility along a timeline by measuring the outputs of training programmes, satisfaction of people trained, perceptions about the quality of trainings, etc.

Approaches for data analysis

• Socio-economic assessment based on impact multipliers: One example of the application of this methodology is the investment of the RI in highly-skilled people with higher salaries that stimulates consumption in the local economy; another example is the organisation of training programmes and the multiplier effect of those attendees.
• Cost-benefit analysis: This methodology can be applied, for example, by valuing the expected increase in earnings by people trained at the RI during their working lifetime compared to a suitable control group. Another example is the assessment of the stock of knowledge output created by the RI in the form of publications, which can be calculated by computing the marginal cost of producing them. The ‘willingness to pay’ approach or the ‘marginal cost of production’ can be employed to value scientific conferences, meetings and other scientific events.
• Approaches based on multiple criteria: Application of the approach included monitoring RI employees or training achievements based on a combination of indicators.
• Theory-based approaches: These can be used to better map out the logic of the highly interlinked and cumulative effects that arise in terms of skills creation, networking benefits and relationship capital that RI activities promote. The approach is useful to explain under what conditions and in which circumstances certain benefits materialise, providing better understanding on the mechanisms that generate each impact.
• Case studies and narratives: Combining qualitative and quantitative evidence, this approach can be used to acquire a deep understanding of the benefits accruing to a sample of people trained by the RI, for example in terms of skills acquired, competences applied in the future work and their impact on career development.  

Tools for data collection

• Tracking the number/volume/nature of contracts with suppliers
• Tracking number of software/instruments/applications/other technologies developed by the RI
• Tracking the number of patents developed (e.g. PATSTAT) and licences awarded
• Tracking the number of start-ups and spin-offs facilitated by the RI
• Analysis of market/industrial statistics regarding sales, developments and revenues of suppliers
• Analysis of balance sheets of suppliers (e.g. Orbis database)
• Systematic surveys of supplier and user firms/industries to track the impacts (i.e. innovations in products/processes, reputation effects, etc.) of their collaboration with the RI
• Systematic surveys of start-ups and spin-offs to better understand the contribution of the RI to their growth
• Tracking the number of downloads of open data and the access of unique users (IP addresses)
• Tracking access and diverse usage of information provided by the RI through specific informatic tools
• Periodic surveys/studies to assess expenditure of employees and visitors (e.g. costs in reaching RIs, including travel, accommodation, any other cost, and motivation, if its RI related, etc.)
• Qualitative analysis of the RI strategy (e.g. managerial decisions, implementation of specific programmes) for innovation and technological diffusion from RIs

Approaches for data analysis

• Input-output models: These can be used to calculate direct and indirect economic impacts on the supply chain derived from RI investment. Average economic impacts can be calculated at sectorial, local and national levels depending on the coverage of the input-output tables. For instance, it is feasible to calculate the multipliers for contracting local and regional suppliers or the investments needed to generate spin-offs.
• Methodologies grounded in the knowledge production-function approach: These can be used to estimate both private and social return on investment in research, mainly in terms of changes in macro-economic variable (GDP) or micro-economic outputs, e.g. positive effect of university research on firms’ performance, including profitability, sales, cost savings, development of new technologies and patents.
• Cost-benefit analysis: This methodology can be use to measure the benefits (improved profit) to firms/suppliers by collaborating with the RI. The incremental profit gains can be obtained by comparing the results of firms entering into relationship with the RI with those of a suitable control group. The CBA is not recommended to capture wider economic impacts, i.e. on the supply chain or at national or local levels. These wider impacts can be valued by input-output models.
• Approaches based on multiple criteria: These can be used, for instance, to measure the changes in the RI’s production capacity by combining diverse indicators. Similarly, studying such changes over time is a way to understand changes in economic and innovation performance.
• Theory-based approaches: These can be applied to understand in finer detail the logic of causation between the investment and software/tools/instruments developed by the RI. It could show the patterns of innovation and explain under which circumstances and conditions RI activities are more likely to lead to certain innovations.
• Case studies and narratives: These can be used to understand the benefits accruing to a sample of suppliers from RI procurement activities (e.g. acquisition of technical know-how, development of new products, reputational effects). This approach may be of interest to the nodes of distributed RIs and can be eventually used in cross-case studies to compare factors that generate impact in different contexts.

Tools for data collection

• Tracking the number of RI visitors by type (country of origin, age, etc.) and by reasons of visit (training, exhibition, promotional events, etc.)
• Tracking the number of social media (e.g. Twitter, LinkedIn) followers and interactions
• Tracking the number of visitors to the RI’s webpage using web analytics (e.g. number of unique visits, average duration of sessions, etc.)
• Tracking the number and content of RI-related news items in the press and online media (e.g. through the use of Meltwater media monitoring products)
• Surveys of users to assess their willingness to pay for a solution (e.g. treatment of cancer, waste management) provided by the RI
• Surveys of various user communities on how well RI data is used
• Surveys of citizens to assess their willingness to pay for or support RI activities
• Tracking the number of participants in (high-level) scientific events organised by an RI
• Content analysis about educational policies/practices regarding inclusion of relevant RI-related topics in academic curricula, through revision of policy documents or data collected ad hoc through surveys
• Qualitative data collection (e.g. semi-structured interviews) to identify specific contributions of the RI to gender balance, social inclusion, environmental issues both internally within the organisation and externally throughout society

Approaches for data analysis

• Cost-benefit analysis: This methodology can be used to estimate the social value of different cultural and outreach effects on society, and solutions addressing societal challenges can be estimated by applying the concept of people’s ‘willingness to pay’ for such activities (e.g. how much are people willing to pay to visit an RI, or assigning a value of time they spend on the RI’s website, or more broadly the value attributed to cancer treatment services provided by the RI in the health sector, etc.).
• Approaches based on multiple criteria: These can be used to monitor public events organised by the RI and the sharing/use of data provided by virtual RIs based on a set of indicators (mostly quantitative aggregated indicators).
• Theory-based approaches: These can be used, for instance, to identify more clearly the key causal links between RI communication and awareness-raising activities and public engagement with science, student career choice and the contribution to a more scientifically literate society. Equally, tracing more concrete impact pathways, it can help to establish a better understanding how RIs contribute to cultural impacts such as the shift in the way knowledge is created and disseminated, how digitalisation of cultural artefacts impacts the sense of identity, integration, etc.
• Case studies and narratives: This approach can help to build a deeper understanding of the benefits accruing to a sample of users from the solution provided by the RI (e.g. development of a new vaccine, provision of curated data on socio-demographics across all European countries, etc.). Case studies can also help to shed light how RIs contribute to less frequent and less investigated areas, such as environmental impacts or those on social inclusion.

Tools for data collection

• Tracking of RI’s presence in relevant thematic committees/networks, including qualitative analysis of specific contributions to the definition of scientific norms and standardisation
• Tracking of RI’s contribution to relevant discussions with policy makers (e.g. participation in policy forums, minuted meeting conclusions, interviews, etc.)
• Tracking the number and content of contracts with specific policy stakeholders
• Tracking tool with detailed information about data transfer/provision in support of policy making
• Periodic perception studies about trust in science (mostly through surveys to local communities or relevant stakeholders)

Approaches for data analysis

• Cost-benefit analysis: This methodology enables an assessment of the social value of the RI by estimating the willingness of taxpayers to contribute financially to the research activity conducted at the RI.
• Approaches based on multiple criteria: By combining diverse indicators, these approaches help to understand how well different RI activities influence decision-making processes.
• Theory-based approach: This can be used to map out the various communication channels and avenues RIs use to influence policy decisions and to generate more clear links; how these activities lead to specific policy outcomes (e.g. standards, regulatory frameworks, etc.) and how their implementation manifests in wider impacts.
• Case studies and narratives: These can be used, for example, to provide a detailed picture on the role of the RI in science diplomacy through narrating the approach to building international partnerships and promoting science as a priority in the international arena.