MINISTRY OF EDUCATION
RUSSIAN FEDERATION
ALTAI STATE UNIVERSITY
Faculty of Economics
Department of Crisis Management, Business Assessment and Innovation
METHOD OF EXPERT ASSESSMENTS
(course work)
Completed by a student
3 courses, group 277
Strekalova S.B.
Job protected
Barnaul – 1999
Introduction 3
Chapter 1. EXPERTISE IN MANAGEMENT 5
1.1. The role of experts in management 5
1.2. Expert assessment method 7
1.3. Organization of expert assessment 9
1.4. Selection of experts 9
1.5. Expert survey 10
Chapter 2. FORMALIZATION OF INFORMATION
AND SCALES OF COMPARISONS 12
Chapter 3. PROCESSING EXPERT ASSESSMENTS 16
3.1. Processing tasks 16
3.2. Group assessment of objects 17
3.3. Assessing the consistency of expert opinions 22
3.4. Processing pairwise object comparisons 25
3.5. Determining the relationship between rankings 27
Conclusion 31
References 32
INTRODUCTION
The modern economy places new, higher demands on management. Issues of improving management methods are now becoming very important, since it is in this area that there are even greater reserves for increasing the efficiency of the national economy.
A significant factor in increasing the scientific level of management is the use of mathematical methods and models when preparing decisions. However, complete mathematical formalization of technical and economic problems is often not feasible due to their qualitative novelty and complexity. In this regard, expert methods are increasingly being used, which are understood as a set of logical and mathematical-statistical methods and procedures aimed at obtaining from specialists the information necessary for preparing and selecting rational decisions.
Expert methods are now used in situations where the choice, justification and assessment of the consequences of decisions cannot be made on the basis of accurate calculations. Such situations often arise when developing modern problems of managing social production and, especially, when forecasting and long-term planning. In recent years, expert assessments have found wide application in socio-political and scientific-technical forecasting, in planning the national economy, industries, associations, in the development of major scientific, technical, economic and social programs, and in solving individual management problems.
As social production develops, not only the complexity of management increases, but also the requirements for the quality of decisions made. In order to increase the validity of decisions and take into account the numerous factors that influence their results, a comprehensive analysis is required, based both on calculations and on reasoned judgments of managers and specialists familiar with the state of affairs and development prospects in various areas of practical activity. The use of expert methods ensures the active and targeted participation of specialists at all stages of decision-making, which can significantly improve their quality and efficiency.
The purpose of our work is to study the method of expert assessments - one of the most important stages in making competent management decisions.
1) studying the role of expertise in management;
2) consideration of the procedure for organizing expert assessment;
3) study of the types of scales and the order of their use;
4) detailed consideration of the final stage of expert assessment – processing of expert assessments.
The abstract consists of an introduction, three chapters, a conclusion and a list of references.
The first chapter discusses the need for expertise in management, considers the method of expert assessments, and the stages of organizing expert assessment.
The second chapter is devoted to the consideration of comparison scales; the characteristics of each type of scale and the procedure for their use in formalizing information are given.
The third chapter discusses the processing of expert assessments: processing tasks, group assessment of objects, assessing the consistency of expert opinions, processing paired comparisons of objects and determining the relationship of rankings.
Since the purpose of this work is to consider expert assessment in a theoretical aspect, practical application is not considered.
In conclusion, the role of the expert assessment method in management decision-making is considered.
Chapter 1. EXPERTISE IN MANAGEMENT
1.1. The role of experts in management
Modern society is developing under the ever-increasing influence of the scientific and technological revolution, which causes fundamental changes in production, profound changes in the structure and economy of the national economy. The ongoing scientific and technological revolution in its influence goes far beyond the sphere of material production, capturing all aspects of the life of society, predetermining most decisions aimed at its rational economic and social development.
The history of the development of science, technology and production shows that simultaneously with the consistent replacement of human functions with machine functions, its role in the field of management increases. The continuous increase in the volume of expenditures on the development of science, on the creation of new technology and the improvement of production significantly increases the importance of decisions made at all levels of national economic management. The future of science. Technology and economics largely depend on the quality and timeliness of these decisions, and objective trends in scientific and technological progress can accelerate or slow down under their influence.
Optimization methods based on the use of formal, most often mathematical models, which save time and money when solving many practical problems, are now gaining particular importance in management. Building models helps to bring complex and sometimes uncertain factors associated with a decision-making problem into a logically coherent scheme and determine what data is needed to evaluate and select alternatives.
In the management process, there is a natural desire to find a solution that is objectively the best of all possible. Mathematical programming is now widely used as an optimization tool. Successes in the application of mathematical programming to the solution of various kinds of economic, scientific, technical and military problems have given rise to methodological views according to which a fundamental solution to control problems is possible only when all its aspects are displayed in a system of interconnected mathematical models.
However, the formalization of technical, economic and management decisions is complicated by a number of features of the current stage of scientific and technological progress. The life of society is so complex that it is difficult to count on the emergence of models that would fully reflect the nature and quantitative relationships of socio-economic processes. Real reality is always more complex than the most subtle mathematical models, and its development often outstrips formal knowledge. Management problems require the participation of people as an integral element of the solution. And finally, the management process itself always involves an orientation not only to numerical data, but also to ordinary common sense. The use of mathematical programming and computer technology allows decisions to be made based on more complete and reliable information. But there is no doubt that, under any conditions, choosing a rational solution requires something more than a good mathematical model.
When making decisions, we usually assume that the information used to support them is valid and reliable. But for many economic, scientific and technical problems, which are qualitatively new and non-repetitive in nature, this assumption is either obviously not realized, or it cannot be proven at the time of decision-making.
The availability of information and the correctness of its use largely determine the optimality of the chosen solution. In addition to data consisting of numerical statistical quantities, information includes other quantities that cannot be directly measured, such as assumptions about possible decisions and their results. Practice shows that the main difficulties that arise when searching for and choosing business solutions are primarily due to the insufficiently high quality and incompleteness of the available information.
The main difficulties associated with information that arise when developing complex decisions can be divided into the following groups.
Firstly, the initial statistical information is often not reliable enough.
Secondly, some of the information is qualitative in nature and cannot be quantified. Thus, it is impossible to accurately calculate the degree of influence of social and political factors on the implementation of plans, evaluate the economic effect of future inventions, etc. But, since these factors and phenomena have a significant impact on the results of decisions, they cannot be ignored.
Thirdly, in the process of preparing decisions, situations often arise when, in principle, it is possible to obtain the necessary information, but at the time of making a decision it is not available, since this is associated with a large investment of time or money.
Fourthly, there is a large group of factors that may affect the implementation of a decision in the future, but they cannot be accurately predicted.
Fifthly, one of the most significant difficulties in choosing solutions is that any scientific or technical idea contains the potential for various schemes for its implementation, and any economic action can lead to multiple outcomes. The problem of choosing the best solution option can also arise because there are usually resource limitations, and therefore, the adoption of one option is always associated with the rejection of other solutions.
Sixthly, when choosing the best solution, we are often faced with the ambiguity of a generalized criterion, on the basis of which we can compare possible outcomes. The polysemy, multidimensionality, and qualitative differences of indicators are a serious obstacle to obtaining a generalized assessment of the relative effectiveness, importance, value, or usefulness of each of the possible solutions.
In this regard, one of the main features of solving complex problems is that the use of calculations here is always intertwined with the use of judgments of managers, scientists, and specialists. These judgments make it possible to at least partially compensate for the lack of information, make fuller use of individual and collective experience, and take into account specialists’ assumptions about the future states of objects. The pattern of development of science and technology is that new knowledge and scientific and technical information accumulate over a long period of time. Often this accumulation occurs in a hidden form in the minds of scientists and developers. They, like no one else, are able to assess the prospects of the field in which they work and foresee the characteristics of those systems in the creation of which they are directly involved.
Experience shows that the use of unsystematized judgments of individual specialists in solving many complex scientific and technical problems is not effective enough due to the variety of relationships between the main elements of such problems and the impossibility of covering them all. When using traditional procedures for preparing decisions, it is often not possible to consider a wide range of factors and take into account the entire range of alternative ways to solve problems.
All this forces us to resort to recruiting groups of specialists representing various fields of knowledge as experts. The use of group expertise allows not only to consider many aspects and factors, but also to combine various approaches with the help of which the manager finds the best solution.
The essence of the expert assessment method is that experts carry out an intuitive-logical analysis of a problem with a quantitative assessment of judgments and formal processing of the results. The generalized expert opinion obtained as a result of processing is accepted as a solution to the problem. The integrated use of intuition (unconscious thinking), logical thinking and quantitative assessments with their formal processing allows us to obtain an effective solution to the problem.
When fulfilling their role in the management process, experts perform two main functions: they form objects (alternative situations, goals, decisions, etc.) and measure their characteristics (probabilities of events occurring, coefficients of significance of goals, preferences for solutions, etc.) . The formation of objects is carried out by experts based on logical thinking and intuition. In this case, the knowledge and experience of the expert play a big role. Measuring the characteristics of objects requires experts to know the measurement theory.
The characteristic features of the expert assessment method as a scientific tool for solving complex non-formalizable problems are, firstly, the scientifically based organization of all stages of the examination, ensuring the greatest efficiency of work at each stage, and, secondly, the use of quantitative methods both in organizing the examination and and when assessing expert judgments and formal group processing of results. These two features distinguish the method of expert assessments from the usual long-known examination, widely used in various spheres of human activity.
Expert collective assessments were widely used on a national scale to solve complex problems of managing the national economy already in the first years of Soviet power. In 1918, a Council of Experts was created under the Supreme Council of the National Economy, whose task was to solve the most complex problems of reorganizing the country's national economy. When drawing up five-year plans for the development of the country's national economy, expert assessments of a wide range of specialists were systematically used.
Currently, in our country and abroad, the method of expert assessments is widely used to solve important problems of various nature. In various industries, associations and enterprises there are permanent or temporary expert commissions that formulate decisions on various complex non-formalized problems.
The entire set of poorly formalized problems can be divided into two classes. The first class includes problems for which there is sufficient information potential to successfully solve these problems. The main difficulties in solving first-class problems during expert assessment lie in realizing the existing information potential by selecting experts, constructing rational survey procedures and applying optimal methods for processing its results. In this case, the survey and processing methods are based on the use of the principle of a “good” meter. This principle means that the following hypotheses are satisfied:
1) the expert is a repository of a large amount of rationally processed information, and therefore he can be considered as a high-quality source of information;
2) the group opinion of experts is close to the true solution to the problem.
If these hypotheses are correct, then the results of measurement theory and mathematical statistics can be used to construct survey procedures and processing algorithms.
The second class includes problems in relation to which the information potential of knowledge is insufficient to ensure confidence in the validity of the specified hypotheses. When solving problems from this class, experts can no longer be considered “good measurers.” Therefore, it is necessary to process the examination results very carefully. The use of averaging methods that are valid for “good meters” in this case can lead to large errors. For example, the opinion of one expert, which is very different from the opinions of other experts, may turn out to be correct. In this regard, for problems of the second class, qualitative processing should mainly be used.
The scope of application of the expert assessment method is very wide. We list typical problems solved by the method of expert assessments:
1) compiling a list of possible events in various areas over a certain period of time;
2) determination of the most probable time intervals for the occurrence of a set of events;
3) determination of management goals and objectives, ordering them by degree of importance;
4) identification of alternatives (options for solving the problem with an assessment of their preferences;
5) alternative distribution of resources for solving problems with an assessment of their preference;
6) alternative options for making decisions in a certain situation with an assessment of their preference.
To solve the listed typical problems, various types of expert assessment methods are currently used. The main types include: questionnaires and interviews; brainstorm; discussion; meeting; operational game; scenario.
Each of these types of expert assessment has its own advantages and disadvantages, which determine the rational scope of application. In many cases, the greatest effect is achieved by the integrated use of several types of examination.
The questionnaire and scenario require individual work by an expert. Interviewing can be carried out either individually or with a group of experts. Other types of examination require the collective participation of experts in the work. Regardless of individual or group participation of experts in the work, it is advisable to obtain information from many experts. This makes it possible to obtain, based on data processing, more reliable results, as well as new information about the dependence of phenomena, events, facts, and expert judgments, which is not explicitly contained in the statements of experts.
When using the method of expert assessments, problems arise. The main ones are: selection of experts, conducting a survey of experts, processing survey results, organizing examination procedures.
1.3. Organization of expert assessment
The first stage of organizing work on the use of expert assessment is the preparation and publication of a guidance document that formulates the purpose of the work and the main provisions for its implementation. This document should reflect the following issues: statement of the experimental problem; goals of the experiment; justification for the need for the experiment; turnaround time; tasks and composition of the management group; responsibilities and rights of the group; financial and material support for work.
An examination manager is appointed to prepare this document, as well as to supervise the entire work. He is entrusted with the formation of a management group and responsibility for organizing its work.
After formation, the management group carries out work on selecting an expert group in approximately the following sequence: understanding the problem being solved; determining the range of areas of activity related to the problem; determination of the proportion of experts in each area of activity; determining the number of experts in the group; drawing up a preliminary list of experts taking into account their location; analysis of the qualities of experts and clarification of the list of experts in the group; obtaining consent from experts to participate in the work; compilation of the final list of the expert group.
In parallel with the process of forming a group of experts, the management group is developing the organization and methodology for conducting expert interviews. In this case, the following issues are resolved: place and time of the survey; number and objectives of survey rounds; survey form; the procedure for recording and collecting survey results; composition of necessary documents.
The next stage of the management group’s work is to determine the organization and methodology for processing survey data. At this stage, it is necessary to determine the tasks and timing of processing, processing procedures and algorithms, forces and means for carrying out processing.
In the process of directly conducting a survey of experts and processing its results, the management group carries out a set of works in accordance with the developed plan, adjusting it as necessary in terms of content, timing and provision of resources.
The last stage of work for the management group is to formalize the results of the work. At this stage, the results of the expert assessment are analyzed; compilation of a report; discussion and approval of the results; submission of work results for approval; familiarization with the results of examination of organizations and individuals.
1.4. Selection of experts
To implement the expert assessment procedure, it is necessary to form a group of experts. The general requirement when forming a group of experts is to effectively solve the problem of examination. The effectiveness of solving the problem is determined by the characteristics of the reliability of the examination and the costs of it.
The reliability of expert assessment can only be determined on the basis of a practical solution to the problem and analysis of its results. The use of experts is precisely due to the fact that there are no other ways to obtain information. Therefore, assessment of the reliability of the examination can be carried out, as a rule, only based on a posteriori (post-experimental) data. If the examination is carried out systematically with approximately the same composition of experts, then it becomes possible to accumulate statistical data on the reliability of the work of a group of experts and obtain a stable numerical assessment of reliability. This assessment can be used as a priori data on the reliability of the panel of experts for subsequent examinations.
The reliability of group expert assessment depends on the total number of experts in the group, the proportion of different specialists in the group, and the characteristics of the experts.
Determining the nature of the dependence of reliability on the listed factors is another problem in the expert selection procedure.
A difficult problem in the selection procedure is the formation of a system of expert characteristics that significantly influence the course and results of the examination. These characteristics should describe the specific properties of the specialist and the possible relationships between people that influence the examination. An important requirement for the characteristics of an expert is the measurability of these characteristics.
Another problem is the organization of the procedure for selecting experts, i.e. determination of a clear sequence of work performed in the process of selecting experts and the necessary resources for their implementation.
The maximum number of experts in a group is checked for financial resource limitations. Having determined the relationship between reliability, the number of experts and payment costs, the management group presents this information to management and formulates possible decision alternatives. Such alternatives may be either reducing the reliability of the expert assessment results to a level that ensures compliance with the limitation on the costs of paying experts, or maintaining the original requirement for the reliability of the examination and increasing the costs of paying experts.
The next stage of the expert selection process is the compilation of a preliminary list of experts. When compiling this list, an analysis of the qualities of experts is carried out. In addition to taking into account the qualities of experts, their location and the possibility of participation of selected specialists in the examination are determined. When assessing qualities, the opinion of people who know candidate experts well is taken into account.
After compiling a list of experts, letters are sent to them inviting them to participate in the examination. The letters explain the purpose of the examination, its timing, procedure, scope of work and terms of remuneration. The letters are accompanied by expert data forms and self-assessment of competence. After receiving the experts' responses, the management team compiles a final list of experts.
After compiling and approving the list, a message is sent to experts about their inclusion in the expert group. If the expert assessment is carried out using a questionnaire method, then simultaneously with notification of inclusion in the expert group, all experts are sent a questionnaire with the necessary instructions for filling them out. The notification to experts about their inclusion in the examination ends the work of selecting experts.
The survey is the main stage of collaboration between the management group and experts. The main content of the survey is:
Statement of the problem and presentation of questions to experts;
Information support for the work of experts;
Development of judgments, assessments, proposals by experts;
Collection of results of experts' work.
There are three types of problems that are solved during the survey process:
Qualitative or quantitative assessment of specified objects;
Construction of new facilities;
Construction and evaluation of new objects.
In collective examination, the following main types of survey are used: discussion, questioning and interviewing, the method of collective generation of ideas, or brainstorming.
Questionnaires can be conducted with or without feedback. When surveying with feedback, the survey of experts is carried out in several stages, bringing to the attention of the experts some of the results of the survey at the previous stage, including the assessments of individual experts and their arguments.
The main thing in organizing a survey is to ensure maximum information and maximum creative activity and independence of the expert. It is necessary to strive to bring to each expert, as far as possible, all the information related to the phenomenon being analyzed, which is available to both experts and survey organizers, without at the same time depriving the expert of creative independence and activity.
However, the expert's ability to process information is limited. As a result, the expert may make a decision without using all the information at his disposal. In addition, new information is perceived by a person with a certain internal resistance and does not immediately affect already established subjective assessments. The attitude towards new information is more favorable, and its perception and use is more complete if it is presented in an intelligible, bright and compact form.
From these psychological characteristics follows the need to provide experts with opportunities to record incoming information by keeping records, using technical means, as well as the need to pre-process information and present it to experts in the most perceivable form.
It is necessary to emphasize the contradictory nature of the exchange of information between experts, since obtaining such information carries the danger of losing creative independence in the construction of a model of an object by an expert. It is impossible to fully resolve this contradiction, and during each examination, its organizers must find a reasonable compromise, first of all, by choosing the type of survey, the form and degree of communication between experts.
Each type of survey has its own advantages and disadvantages in building the exchange of information between experts and in organizing their independent creativity. The choice of one type of survey or another is determined by many factors, the main ones being:
The purpose and objectives of the examination;
The essence and complexity of the problem being analyzed;
Completeness and reliability of source information;
The required volume and reliability of information obtained as a result of the survey;
Time allocated for the survey and examination in general;
Acceptable cost of the survey and examination in general;
Number of experts and management team members, their characteristics.
Questioning is the most effective and most common type of survey, because it allows the best combination of information support for experts with their independent creativity.
Chapter 2. FORMALIZATION OF INFORMATION AND SCALES OF COMPARISONS
Rational use of information received from experts is possible provided that it is formed into a form convenient for further analysis aimed at preparing and making decisions.
The possibilities of formalizing information depend on the specific features of the object under study, the reliability and completeness of the available data, and the level of decision-making. The form of presentation of expert data also depends on the adopted criterion, the choice of which, in turn, is significantly influenced by the specifics of the problem under study.
The formalization of information received from experts should be aimed at preparing solutions to such technical, economic and business problems that cannot be fully described mathematically, since they are “weakly structured”, i.e. contain uncertainties associated not only with measurement, but also with the very nature of the goals being studied, the means of achieving them and external conditions.
When analyzing prospects, it is necessary not only to present in the form of indirect estimates part of the information that cannot be quantified, and not only to express with the help of such estimates quantifiable information about which there is not sufficiently reliable data at the time of preparing the decision. The most important thing is to formalize this information in such a way as to help the decision maker choose from a variety of actions one or more that are most preferable with respect to some criterion.
If an expert is able to compare and evaluate possible options for action, assigning a certain number to each of them, then he has a certain system of preferences. Depending on the scale on which these preferences can be specified, expert assessments contain a greater or lesser amount of information and have a different ability to be formalized.
The objects or phenomena under study can be identified or distinguished on the basis of characteristics or factors. A factor is a set consisting of at least two elements reflecting different levels of some quantities to be considered. The level of some factors can be expressed quantitatively (in rubles, percentages, kilograms, etc.) - such factors are called quantitative. The level of others cannot be expressed using numbers; they are called qualitative.
Factors are conventionally divided into discrete and continuous. Factors with a certain, usually small, number of levels are discrete. Factors whose levels are considered to form a continuous set are called continuous. Depending on the goals and capabilities of the analysis, the same factors can be treated either as discrete or continuous.
Let's consider the basic logical axioms that are used in expert methods when formalizing information using various scales.
Using nominal scales the objects under study can be identified and distinguished on the basis of three axioms of identification:
1) i either there is j, or there is no j ;
2) if i There is j, That j There is i ;
3) if i There is j And j There is k, That i There is k .
Factors in this case act as associative indicators that have information that can be formalized in the form of binary assessments of two levels: 1 (identical) or 0 (different).
In cases where the objects under study can be arranged in a certain sequence as a result of comparison, taking into account any significant factor(s), ordinal scales allowing to establish equivalence or dominance.
Suppose that it is necessary to arrange in a certain sequence n objects according to any factor (criterion). Let's represent this ordering in the form of a matrix where i, j = 1,2,…, n .
Quantities establish relationships between objects and can be defined as follows:
Let us establish the basic axioms necessary to comply with the ordering conditions. The ratio means that i preferable j, must be asymmetric, i.e., if then and transitive, i.e., if then
The ratio means that i And j are equivalent is called an equivalence relation. This ratio should be
reflexive, i.e.
symmetrical, i.e., if then
transitive, i.e., if and then
In addition, these two relations must be compatible, i.e., if and then and also, if and then
And finally, the ordering must be coherent, i.e. for any i and j or or or
The use of ordinal scales allows one to distinguish between objects in cases where the factor (criterion) is not specified explicitly, i.e. when we do not know the comparison sign, but we can partially or completely order objects based on the system of preferences that the expert has.
Any set A we will call it ordered if for any two of its elements X And Y it has been established that either X preceded Y, or Y preceded X. Sometimes it is not possible to establish strict precedence for all elements of a set, but it is possible to perform “group” ordering, when subsets of equivalent elements are ordered. Next, we can set the task of comparing and ordering these subsets.
The use of ordinal scales makes it possible to transform the estimates obtained from experts corresponding to all monotonically increasing functions. So, for example, positive estimates can either be replaced by their squares, or logarithms, or any other monotonically increasing function.
To formalize assessments received from experts, they often use interval scales. When such scales are used for these purposes, almost all conventional statistical measures can be taken. The exceptions are those measures that require knowledge of the “true” zero point of the scale, which is introduced here conditionally.
Interval scales suggest the possibility of transforming ratings obtained on one scale into ratings on another scale using the equation
The differences between values on the interval scale become measures on the ratio scale, i.e. on a regular numerical scale, because as a result of subtraction you can get rid of the constant term b .
In a number of cases, when formalizing expert assessments, the property of additivity, which is inherent only in the ratio scale, is used. The presence of additivity is expressed by the following axioms:
1) if j = a And i> 0, then i + j > a ;
2) i + j = j + i ;
3) if i = a And j = b, That i + j = a + b ;
4) (i + j) + k = i + (j + k).
For example, factors (indicators) characterizing the effectiveness of the creation and implementation of new technology, according to their objective content, can be divided into technical, economic and social. On the other hand, these factors can be grouped according to their role in the process of creating and introducing new technology, highlighting, for example, indicators characterizing costs, quality, economic efficiency, etc.
Depending on the nature and purpose of the problem under study, the factors by which objects differ can be quantitatively comparable or incomparable with each other, partially comparable (that is, not any with any, but only some of them), ordered by the degree of their importance, etc. .d. The incommensurability of various factors is due not only to the need to use different units of measurement, but also to the fact that each factor, expressing a certain property, is at the same time an assessment of the attitude towards this property on the part of the decision maker.
In the practice of management at all its levels, situations often arise when it is necessary to make a decision taking into account many factors. The question of which factors should be considered the most important depends on the qualitative characteristics of the object of decision and the goals that this decision must meet.
For example, when considering several options for a plan or options for organizational and technical measures, factors of time, costs, technical and social results, economic efficiency, etc. should be taken into account. Usually, all the variety of factors is tried to lead to an unambiguous comprehensive assessment, and the most convenient and widespread such assessment is monetary.
However, since the consequences of any decision, especially decisions related to scientific and technological progress, go beyond the scope of cost indicators, measures are needed that characterize the significance and usefulness of a particular factor (or their complex). Such complex meters are widely used in assessing the quality of products, the technical and economic level of production, in assessing the results of the activities of scientific organizations and in a number of other tasks. Although the question of creating a sufficiently justified formalized system of such meters is still far from a final solution, it is possible to indicate some general features that provide an approach to the formalization of this process and to the use of one or another logical-mathematical apparatus.
In the case when all factors are specified on a nominal scale, i.e. a certain attribute a and an initial set of elements M are specified on this scale; the goal is to select a subset of elements M(a) possessing this attribute. In such cases, the elements, or rather their properties, are compared with a characteristic - a standard, and the result - a partition of the set - can be considered as an ordering on a two-element scale, on which each element is assigned a score equal to either zero or one.
In the case where factors are specified on an ordinal scale or on several ordinal scales, the goal is to order the elements of the original set, to identify, with the help of experts, the hidden ordering that is assumed to be inherent in this set. A necessary condition for solving this problem is the assumption of transitivity. The more completely the elements are ordered, the easier it is to apply logical-mathematical and combinatorial methods to solve such problems.
Depending on the essence or importance of a particular factor, various scales can be used at the stage of preparation and decision-making. Factors such as costs, profit, time can be assessed on an ordinal or interval scale (in rubles, days or conventional units). To assess factors such as payback period or comparative effectiveness of options, an interval scale can be used; qualitative or social factors can be assessed on ordinal or nominal scales.
3.1. Processing tasks
After conducting a survey of a group of experts, the results are processed. The initial information for processing is numerical data expressing the preferences of experts and a meaningful justification for these preferences. The purpose of processing is to obtain generalized data and new information contained in hidden form in expert assessments. Based on the processing results, a solution to the problem is formed.
The presence of both numerical data and meaningful statements from experts leads to the need to use qualitative and quantitative methods for processing the results of group expert assessment. The share of these methods significantly depends on the class of problems solved by expert assessment.
The whole set of problems can be divided into two classes. The first class includes problems for which there is a sufficient level of knowledge and experience, that is, there is the necessary information potential. When solving problems belonging to this class, experts are considered to be good measurers on average. The term “good on average” refers to the ability to obtain measurement results that are close to the true ones. For many experts, their judgments cluster around the true value. It follows that to process the results of group expert assessment of problems of the first class, methods of mathematical statistics based on data averaging can be successfully applied.
The second class includes problems for which sufficient information potential has not yet been accumulated. In this regard, expert judgments can vary greatly from each other. Moreover, the judgment of one expert, which is very different from other opinions, may turn out to be true. It is obvious that the use of methods for averaging the results of group expert assessment when solving problems of the second class can lead to large errors. Therefore, processing the results of a survey of experts in this case should be based on methods that do not use the principles of averaging, but on methods of qualitative analysis.
Considering that problems of the first class are the most common in the practice of expert assessment, the main attention in this chapter is on methods for processing examination results for this class of problems.
Depending on the goals of the expert assessment and the chosen measurement method, the following main tasks arise when processing survey results:
1) constructing a generalized assessment of objects based on individual expert assessments;
2) constructing a generalized assessment based on pairwise comparison of objects by each expert;
3) determination of the relative weights of objects;
4) determining the consistency of expert opinions;
5) determination of dependencies between rankings;
6) assessment of the reliability of processing results.
The task of constructing a generalized assessment of objects based on individual expert assessments arises during group expert assessment. The solution to this problem depends on the measurement method used by the experts.
When solving many problems, it is not enough to organize objects according to one indicator or some set of indicators. It is desirable to have numerical values for each object that determine its relative importance compared to other objects. In other words, for many tasks it is necessary to have evaluations of objects that not only organize them, but also allow one to determine the degree of preference of one object over another. To solve this problem, you can directly apply the direct estimation method. However, the same problem, under certain conditions, can be solved by processing expert assessments.
The consistency of expert opinions is determined by calculating a numerical measure characterizing the degree of similarity of individual opinions. Analysis of the value of the measure of consistency contributes to the development of a correct judgment about the general level of knowledge on the problem being solved and the identification of groupings of expert opinions. A qualitative analysis of the reasons for the grouping of opinions makes it possible to establish the existence of different views, concepts, identify scientific schools, determine the nature of professional activity, etc. All these factors make it possible to more deeply comprehend the results of a survey of experts.
By processing the results of expert assessment, it is possible to determine the dependencies between the rankings of various experts and thereby establish the unity and difference in the opinions of experts. An important role is also played by establishing the relationship between rankings based on various indicators for comparing objects. Identifying such dependencies makes it possible to reveal related comparison indicators and, perhaps, group them according to the degree of connection. The importance of the task of determining dependencies for practice is obvious. For example, if the indicators of comparison are various goals, and the objects are the means of achieving the goals, then establishing the relationship between the rankings that order the means from the point of view of achieving the goals allows us to reasonably answer the question to what extent the achievement of one goal with given means contributes to the achievement of other goals.
The estimates obtained from processing are random objects, so one of the important tasks of the processing procedure is to determine their reliability. Appropriate attention should be paid to solving this problem.
Processing the results of the examination is a labor-intensive process. Performing operations of calculating estimates and indicators of their reliability manually is associated with large labor costs, even in the case of solving simple ordering problems. In this regard, it is advisable to use computer technology and especially computers. The use of computers raises the problem of developing computer programs that implement algorithms for processing the results of expert assessment.
3.2. Group assessment of objects
In this section, we will consider algorithms for processing the results of expert assessment of multiple objects. Let m experts assessed n objects by l indicators. The assessment results are presented in the form of values, where j– expert number, i- object number, h– number of the indicator (sign) of comparison. If the objects are assessed using the ranking method, then the values represent ranks. If the assessment of objects is carried out by the method of direct assessment or the method of sequential comparison, then the values are numbers from a certain segment of the numerical axis, or points. Processing of assessment results significantly depends on the measurement methods considered.
Let us consider the case when values are obtained by methods of direct assessment or sequential comparison, i.e., they are numbers or points. To obtain a group assessment of objects in this case, you can (use the average assessment value for each object
(5.1)
where are the coefficients of weights of indicators for comparing objects, and are the coefficients of expert competence. The coefficients of weights of indicators and competence of objects are normalized values
(5.2)
The weight coefficients of the indicators can be determined expertly. If - weight coefficient h-th indicator given j-th expert, then the average weight coefficient h- indicator for all experts is equal
(5.3)
Obtaining a group expert assessment by summing individual assessments with weights of competence and the importance of indicators when measuring the properties of objects in cardinal scales is based on the assumption that the axioms of the von Neumann-Morgenstern utility theory are fulfilled for both individual and group assessment and the conditions of indistinguishability of objects in a group relation, if they are indistinguishable in all individual assessments (partial Pareto principle). In real problems, these conditions are usually met, so obtaining a group assessment of objects by summing individual expert assessments with weights is widely used in practice.
Expert competence coefficients can be calculated from a posteriori data, i.e., from the results of object assessment. The main idea of this calculation is the assumption that the competence of experts should be assessed by the degree of consistency of their assessments with the group assessment of objects.
The algorithm for calculating expert competence coefficients has the form of a recurrent procedure:
(5.4)
(5.5)
(5.6)
Calculations start from t=1. In formula (5.4), the initial values of the competence coefficients are assumed to be the same and equal. Then, according to formula (5.4), group assessments of objects of the first approximation are equal to the arithmetic average values of expert assessments
(5.7)
(5.8)
and the value of the competence coefficients of the first approximation according to formula (5.6):
(5.9)
Using the competence coefficients of the first approximation, you can repeat the entire calculation process using formulas (5.4), (5.5), (5.6) and obtain second approximations of the quantities
Repeating the recurrent procedure for calculating object assessments and competence coefficients naturally raises the question of its convergence. To consider this issue, we exclude variables from equations (5.4), (5.6) and present these equations in vector form
where are the matrices IN dimensions and WITH dimensions are equal
The quantity in equations (5.10) is determined by formula (5.5).
If matrices IN And WITH are non-negative and indecomposable, then, as follows from the Perron–Frobenius theorem, the prevectors and - converge to the eigenvectors of the matrices IN And WITH, corresponding to the maximum eigenvalues of these matrices
(5.12)
Limit values of vectors X And k can be calculated from the equations:
(5.13)
where are the maximum eigenvalues of the matrices IN And WITH .
Condition for non-negativity of matrices IN And WITH easily done by choosing non-negative elements of the matrix X assessments of objects by experts.
Condition for matrix indecomposability IN And WITH practically fulfilled, since if these matrices are decomposable, this means that experts and objects fall into independent groups. Moreover, each group of experts evaluates only the objects of its group. Naturally, there is no point in receiving a group assessment in this case. Thus, the conditions for non-negativity and indecomposability of matrices IN And WITH, and consequently, the convergence conditions of procedures (5.4), (5.5), (5.6) are satisfied in practical conditions.
It should be noted that the practical calculation of vectors of group assessment of objects and competence coefficients is easier to perform using recurrent formulas (5.4), (5.5), (5.6). Determining the limiting values of these vectors using equation (5.13) requires the use of computer technology.
Let us now consider the case when experts evaluate a set of objects using the ranking method so that the values are ranks. Processing the ranking results involves constructing a generalized ranking. To construct such a ranking, we introduce a finite-dimensional discrete ranking space and a metric in this space. Each ranking of a set of objects j- expert is a point in the ranking space.
The ranking can be represented as a matrix of paired comparisons, the elements of which are defined as follows:
Obviously, since every object is equivalent to itself. The elements of the matrix are antisymmetric.
If all ranked objects are equivalent, then all elements of the pairwise comparison matrix are equal to zero. We will denote such a matrix and assume that the point in the ranking space corresponding to the matrix is the origin.
Reversing the order of ranked objects results in a transposition of the pairwise comparison matrix.
Metric as distance between i th and j-rankings is determined uniquely by the formula
if the following 6 axioms are satisfied:
1. Moreover, equality is achieved if the rankings and are identical;
2. 
Moreover, equality is achieved if the ranking “lies between” the rankings and. The concept “lies between” means that a judgment about a certain pair of objects in the ranking coincides with a judgment about this pair either in, or in, or in
4.
where is obtained from not some permutation of objects, but from the same permutation. This axiom asserts the independence of distance from the renumbering of objects.
5. If two rankings are the same everywhere except n-element set of elements, which is simultaneously a segment of both rankings, then it can be calculated as if the ranking of only these n-objects. A ranking segment is a set whose complement is non-empty and all elements of this complement are either in front or behind each element of the segment. The meaning of this axiom is that if two rankings are completely consistent at the beginning and end of a segment, and the difference lies in the ordering of the averages n-objects, then it is natural to assume that the distance between rankings should be equal to the distance corresponding to the rankings of average n-objects.
6. The minimum distance is one.
The ranking space for two objects can be depicted as three points lying on the same straight line. The distances between the points are equal. With three objects, the space of all possible rankings consists of 13 points.
Using the introduced metric, we define a generalized ranking as a point that best agrees with the points representing the rankings of experts. The concept of best agreement in practice is most often defined as the median and average ranking.
The median is such a point in the ranking space, the sum of the distances from which to all points - rankings of experts - is minimal. According to the definition, the median is calculated from the condition
The average ranking is such a point, the sum of squared distances from which to all points – expert rankings – is minimal. The average ranking is determined from the condition
The ranking space is finite and discrete, so the median and average ranking can only be some points of this space. In general, the median and average ranking may not coincide with any of the expert rankings.
If the competence of experts is taken into account, then the median and average ranking are determined from the conditions:
where are the coefficients of expert competence.
If objects are ranked according to several indicators, then the median is first determined for each expert for all indicators, and then the median is calculated for many experts:
(j
=1,2,…,m);
where are the coefficients of indicator weights.
The main disadvantage of determining a generalized ranking in the form of a median or average ranking is the complexity of the calculations. The natural way of finding or searching through all points of the ranking space is unacceptable due to the very rapid increase in the uniformity of space with an increase in the number of objects and, consequently, the increase in the complexity of calculations. You can reduce the search problem to a specific integer programming problem. However, this does not reduce the computational difficulty very effectively.
The discrepancy in generalized rankings for various criteria occurs when the number of experts is small and their assessments are inconsistent. If the opinions of experts are close, then the generalized rankings based on the criteria of the median and mean value will coincide.
The complexity of calculating the median or average ranking has led to the need to use simpler methods for constructing a generalized ranking.
One of these methods is the method of sums of ranks.
This method consists of ranking objects according to the sums of ranks received by each object from all experts. The sums are compiled for the ranking matrix
To take into account the competence of experts, it is enough to multiply each i-th ranking on the competence coefficient j th expert In this case, calculating the sum of ranks for i th object is produced using the following formula:
(i =1,2,…,n).
The generalized ranking, taking into account the competence of experts, is based on the ordering of the sums of ranks for all objects.
It should be noted that constructing a generalized ranking based on sums of ranks is a correct procedure if ranks are assigned as places of objects in the form of natural numbers 1, 2, ..., n. If you assign ranks in an arbitrary manner, like numbers on an order scale, then the sum of ranks, generally speaking, does not preserve the condition of monotonicity of the transformation and, therefore, you can obtain different generalized rankings for different mappings of objects onto the number system. The numbering of object locations can be done in the only way using natural numbers. Therefore, with good agreement among experts, constructing a generalized ranking using the rank sum method gives results consistent with the results of calculating the median.
Another more theoretically sound approach to constructing a generalized ranking is to move from the ranking matrix to the pairwise comparison matrix and calculate the eigenvector corresponding to the maximum eigenvalue of this matrix. The objects are ordered according to the magnitude of the eigenvector components.
3.3. Assessing the consistency of expert opinions
When ranking objects, experts usually disagree on the problem being solved. In this regard, there is a need to quantify the degree of expert agreement. Obtaining a quantitative measure of the consistency of expert opinions allows for a more reasonable interpretation of the reasons for the divergence of opinions.
Currently, two measures of agreement between the opinions of a group of experts are known: dispersion and entropy concordance coefficients.
Dispersion coefficient of concordance. Consider the matrix of ranking results n objects by a group of m experts ( j =1,…,m ; i =1,…,n), where is the rank assigned j th expert i-th object. Let's compile the sums of ranks for each column. As a result, we obtain a vector with components
(i=1,2,…,n). (5.14)
We consider the quantities as realizations of a random variable and find an estimate of the variance. As is known, the optimal estimate of dispersion according to the criterion of minimum mean square error is determined by the formula:
, (5.15)
where is the estimate of the mathematical expectation equal to
The dispersion coefficient of concordance is defined as the ratio of the dispersion estimate (5.15) to the maximum value of this estimate
The concordance coefficient varies from zero to one because.
Let us calculate the maximum value of the variance estimate for the case of the absence of related ranks (all objects are different). Let us first show that the estimate of the mathematical expectation depends only on the number of objects and the number of experts. Substituting the value from (5.14) into (5.16), we obtain
Let us first consider summed over i at fixed j. This is the sum of ranks for j th expert. Since the expert uses natural numbers from 1 to n, then, as is known, the sum of natural numbers from 1 to n equal to
(5.19)
Substituting (5.19) into (5.18), we get
(5.20)
Thus, the average value depends only on the number of experts m and number of objects n .
To calculate the maximum value of the dispersion estimate, we substitute the value from (5.14) into (5.15) and square the binomial in parentheses. As a result we get
(5.21)
Considering that from (5.18) it follows
we get
(5.22)
The maximum value of dispersion is achieved at the largest value of the first term in square brackets. The value of this term depends significantly on the arrangement of ranks - natural numbers in each row i. Let, for example, everything m experts gave the same ranking for everyone n objects. Then each row of the matrix will contain identical numbers. Therefore, summing the ranks in each i-u line gives m- multiple repetition i-ro numbers:
By squaring and summing over i, we obtain the value of the first term in (5.22):
(5.23)
Now suppose that experts give divergent rankings, for example, for the case n =m all experts assign different ranks to one object. Then
Comparing this expression with m =n, we make sure that the first term in square brackets of formula (9) is equal to the second term and, therefore, the variance estimate is zero.
Thus, the case of complete coincidence of expert rankings corresponds to the maximum value of the dispersion estimate. Substituting (5.23) into (5.22) and performing transformations, we obtain
(5.24)
Let us introduce the notation
(5.25)
Using (5.25), we write the estimate of dispersion (5.15) in the form
Substituting (5.24), (5.25), (5.26) into (5.17) and reducing by a factor ( n-1), we write the final expression for the concordance coefficient
(5.27)
This formula determines the concordance coefficient for the case of the absence of related ranks.
If there are related ranks in the rankings, then the maximum value of the variance in the denominator of formula (5.17) becomes smaller than in the absence of related ranks. It can be shown that in the presence of related ranks, the concordance coefficient is calculated by the formula:
(5.28)
(5.29)
In formula (5.28) - the indicator of related ranks in j th ranking, is the number of groups of equal ranks in j th ranking, - the number of equal ranks in k-group of related ranks when ranking j th expert. If there are no matching ranks, then =0,=0 and, therefore, =0. In this case, formula (5.28) coincides with formula (5.27).
The concordance coefficient is equal to 1 if all expert rankings are the same. The concordance coefficient is zero if all rankings are different, i.e. there is absolutely no coincidence.
The concordance coefficient, calculated using formula (5.27) or (5.28), is an estimate of the true value of the coefficient and, therefore, is a random variable. To determine the significance of estimating the concordance coefficient, it is necessary to know the frequency distribution for various values of the number of experts m and number of objects n. Frequency allocation for W at and calculated in . For large values m And n You can use well-known statistics. With the number of objects n>7 the significance of the concordance coefficient can be assessed using the criterion. Magnitude Wm (n -1 ) has a distribution with v=n–1 degrees of freedom.
In the presence of related ranks, the distribution with v=n-1 degrees of freedom has the quantity:
(5.30)
Entropy concordance coefficient is determined by the formula (coefficient of agreement):
Where N– entropy, calculated by the formula
(5.32)
a is the maximum entropy value. In the formula for entropy - probability estimates j-th rank assigned i-th object. These probability estimates are calculated as the ratio of the number of experts who assigned a rank to the object j to the total number of experts.
The maximum entropy value is achieved with an equally probable distribution of ranks, i.e. when. Then
Substituting this relation into formula (5.32), we obtain
(5.35)
The agreement coefficient varies from zero to one. When the arrangement of objects by rank is equally probable, since in this case . This case may be due either to the impossibility of ranking objects according to a formulated set of indicators, or to a complete inconsistency of expert opinions. At , which is achieved at zero entropy ( H=0), all experts give the same ranking. Indeed, in this case, for each fixed object, all experts assign it the same rank j, therefore, , a Therefore and H =0.
A comparative assessment of the dispersion and entropy concordance coefficients shows that these coefficients give approximately the same assessment of the agreement of experts with close rankings. However, if, for example, the entire group of experts was divided in their opinions into two subgroups, and the rankings in these subgroups are opposite (direct and reverse), then the dispersion coefficient of concordance will be equal to zero, and the entropy coefficient of concordance will be equal to 0.7. Thus, the entropy coefficient of concordance allows us to record the fact of the division of opinions into two opposing groups. The amount of calculations for the entropy concordance coefficient is somewhat larger than for the dispersion concordance coefficient.
3.4. Handling paired object comparisons
When solving the problem of assessing a large number of objects (ranking, determining relative weights, scoring), difficulties of a psychological nature arise due to the experts’ perception of many properties of objects. Experts can solve the problem of pairwise comparison of objects with relative ease. The question arises: how to obtain an assessment of the entire set of objects based on the results of pairwise comparison, without imposing transitivity conditions? Let's consider an algorithm for solving this problem. Let m experts evaluate all pairs of objects, giving a numerical rating
(5.36)
If, when assessing a pair of experts, they spoke in favor of the experts’ preference, and the opposite experts spoke out, and the experts consider these objects to be equivalent, then the estimate of the mathematical expectation of the random variable is equal to
(5.37)
The total number of experts is equal to the sum
(5.38)
Determining from here and substituting it into (5.37), we obtain
(5.39)
It is obvious that the set of values forms a matrix on the basis of which it is possible to construct a ranking of all objects and determine the coefficients of the relative importance of objects.
Let us introduce a vector of coefficients of the relative importance of objects of order t with the following formula:
where is the matrix of mathematical expectations of evaluations of pairs of objects, 
- vector of coefficients of relative importance of order objects t
.
The value is
(5.41)
First-order relative importance coefficients are the relative sums of matrix row elements X. Indeed, believing t=1, from (5.40) we obtain
(5.42)
Second order relative importance coefficients ( t=2) there are relative sums of matrix row elements X 2 .
(5.43)
If matrix X is non-negative and indecomposable, then as the order increases, the quantity converges to the maximum eigenvalue of the matrix X
and the vector of coefficients of the relative importance of objects tends to the eigenvector of the matrix X, corresponding to the maximum eigenvalue
The determination of the eigenvalues and eigenvectors of the matrix is carried out by solving the algebraic equation
where E is the identity matrix, and systems of linear equations
Where k– eigenvector of the matrix X, corresponding to the maximum eigenvalue. Eigenvector components are coefficients of the relative importance of objects, measured on a ratio scale.
From a practical point of view, it is easier to calculate the coefficients of the relative importance of objects using a sequential procedure according to formula (5.40) with t=1, 2, ... As experience shows, 3-4 sequential calculations are enough to obtain the values and k, close to the limiting values determined by equations (5.46), (5.47).
The matrix is non-negative because all its elements (5.39) are non-negative. A matrix is called indecomposable if it cannot be reduced to a triangular form by rearranging rows (rows and columns of the same name)
(5.48)
where are the indecomposable submatrices of the matrix X. Matrix representation X in the form (5.48) means the division of objects into l dominant sets
At 1 =n matrix X is indecomposable, i.e. there is only one dominant set that coincides with the original set of objects. Matrix decomposability X means that there is great disagreement among experts in the assessment of objects.
If matrix X is indecomposable, then the calculation of relative importance coefficients makes it possible to determine how many times one object is superior to another object in terms of compared indicators. Calculating the coefficients of the relative importance of objects allows you to simultaneously construct a ranking of objects. Objects are ranked so that the first object is considered to be the object with the highest relative importance coefficient. The complete ranking is determined by a chain of inequalities
from which it follows
If the matrix X is decomposable, then the coefficients of relative importance can only be determined for each set. For each matrix, the maximum eigenvalue and the corresponding eigenvector are determined. The components of the eigenvector are the coefficients of the relative importance of the objects included in the set. Using these coefficients, the objects of a given set are ranked. The general ranking of objects is given by the relation
Thus, if the matrix X is indecomposable, then based on the results of pairwise comparison of objects, it is possible to both measure the preference of objects in the scale of relations and in the scale of order (ranking). If the matrix X decomposable, then only ranking of objects is possible.
It should be noted that the preference relation can be expressed by any positive number WITH. In this case, the condition must be met. In particular, you can choose WITH=2 so that if , then if then and if , then .
3.5. Determining the relationship between rankings
When processing ranking results, problems may arise in determining the relationship between the rankings of two experts, the connection between the achievement of two different goals when solving the same set of problems, or the relationship between two characteristics.
In these cases, a measure of the relationship can be rank correlation coefficient. A characteristic of the relationship between multiple rankings or goals will be a matrix of rank correlation coefficients. The Spearman and Kendall rank correlation coefficients are known.
Spearman's rank correlation coefficient is determined by the formula:
where is the mutual correlation moment of the first and second rankings, and is the dispersion of these rankings. Based on these two rankings, estimates of the mutual correlation moment and dispersion are calculated using the formulas:
(5.51)
(5.52)
Where n– the number of ranked objects, – ranks in the first and second rankings, respectively, – average ranks in the first and second rankings. Average rank estimates are determined by the formulas:
(5.53)
Let us calculate estimates of average ranks and variances under the assumption that there are no associated ranks in the rankings, i.e., both rankings give a strict ordering of objects. In this case, the average ranks (5.53) are the sums of natural numbers from one to n, divided by n. Therefore, the average ranks for both rankings are the same and equal
(5.54)
When calculating variance estimates, we note that if we open the parentheses in formulas (5.52), then under the sum sign there will be natural numbers and their squares. Two rankings can differ from each other only by permutation of ranks, but the sum of natural numbers and their squares does not depend on the order (permutation) of the terms. Consequently, the variances (5.52) for any two rankings (in the absence of related ranks) will be the same and equal
(i=1,2). (5.55)
Substituting the value from (5.51) and from (5.55) into formula (5.50), we obtain an estimate of the Spearman rank correlation coefficient
(5.56)
For practical calculations, it is more convenient to use another formula for the Spearman correlation coefficient. It can be obtained from (5.56) if we use the identity
In equality (5.57), the first two sums on the right side, as follows from expression (5.55), are identical and equal
Substituting the value of the sum from (5.57) into formula (5.56) and using equality (5.58), we obtain the following formula for the Spearman rank correlation coefficient, convenient for calculations:
(5.59)
The Spearman correlation coefficient ranges from –1 to +1. Equality to one is achieved, as follows from formula (5.59), with identical rankings, that is, when the Value occurs with opposite rankings (direct and reverse rankings). When the correlation coefficient is equal to zero, the rankings are considered linearly independent.
The estimate of the correlation coefficient, calculated using formula (5.59), is a random variable. To determine the significance of this assessment, it is necessary to set the probability value, decide on the significance of the correlation coefficient and determine the threshold value using the approximate formula
(5.60)
Where n– number of objects, - function inverse to function
for which there are tables. After calculating the threshold value, the correlation coefficient estimate is considered significant if.
To determine the significance of the Spearman coefficient estimate, you can use the Student’s test, since the value
approximately distributed according to Student's law with n – 2 degrees of freedom.
If there are related ranks in the rankings, then the Spearman coefficient is calculated using the following formula:
(5.62)
where is the estimate of the Spearman rank correlation coefficient, calculated using formula (5.59), and the values are
(5.63)
In these formulas, the number of different associated ranks in the first and second rankings, respectively.
The Kendall rank correlation coefficient in the absence of related ranks is determined by the formula:
Where n– number of objects, - ranks of objects, sign x– function equal to
A comparative assessment of the Spearman and Kendall rank correlation coefficients shows that the Spearman coefficients are calculated using a simpler formula. In addition, the Spearman coefficient gives a more accurate result, since it is an optimal estimate of the correlation coefficient according to the criterion of minimum mean square error.
It follows that in practical calculations of the correlation dependence of rankings, it is preferable to use the Spearman rank correlation coefficient.
CONCLUSION
The dynamism and novelty of modern national economic problems, the possibility of the emergence of various factors influencing the effectiveness of decisions, require that these decisions be made quickly and at the same time be well justified. Experience, intuition, a sense of perspective, combined with information, help specialists more accurately select the most important goals and directions of development, find the best options for solving complex scientific, technical and socio-economic problems in conditions where there is no information about solving similar problems in the past.
The use of the method of expert assessments helps to formalize the procedures for collecting, summarizing and analyzing the opinions of experts in order to transform them into a form that is most convenient for making an informed decision.
But it should be noted that the method of expert assessments cannot replace either administrative or planning decisions; it only allows one to replenish the information necessary for preparing and making such decisions. The widespread use of expert assessments is legitimate only where it is impossible to use more accurate methods to analyze the future.
Expert methods are constantly being developed and improved. The main directions of this development are determined by a number of factors, including the desire to expand the scope of applications, increase the degree of use of mathematical methods and electronic computer technology, and also find ways to eliminate emerging shortcomings.
Despite the successes achieved in recent years in the development and practical use of the expert assessment method, there are a number of problems and tasks that require further methodological research and practical testing. It is necessary to improve the system for selecting experts, increasing the reliability of group opinion characteristics, developing methods for checking the validity of assessments, and studying hidden reasons that reduce the reliability of expert assessments.
However, even today expert assessments in combination with other mathematical and statistical methods are an important tool for improving management at all levels.
BIBLIOGRAPHY:
2. Bekleshev V.K., Zavlin P.N. Labor standards in research institutes and design bureaus. M.: Economics, 1973. 203 p.
10. Dobrov G.M., Ershov Yu.V., Levin E.I., Smirnov L.P. Expert assessments in scientific and technical forecasting. Kyiv: Naukova Dumka, 1974. 263 p.
11. Evlanov L.G. Decision making under conditions of uncertainty. M.: IUNKh, 1976. 196 p.
12. Evlanov L.G., Kutuzov V.A. Expert assessments in management. M.: Economics, 1978. 133 p.
13. Kardanskaya N. Making management decisions. M.: UNITY, 1999. 407 p.
14. Kemeny D., Snell D. Cybernetic modeling. M.: Soviet radio, 1972. 234 p.
15. Kravchenko T.K. The process of making planning decisions. M.: Economics, 1974. 183 p.
16. Mirkin B.G. Group choice problem. M.: Nauka, 1974. 256 p. . 17. Mikheev V.I. Social and psychological aspects of management. Style and methods of work of the leader. M.: Young Guard, 1975. 181 p.
18. Pfanzagl I. Theory of measurements. M.: Mir, 1976. 278 p.
19. Tikhomirov Yu.A. Management decision. M.: Nauka, 1996. 278 p.
20. Fedorenko N.P. Economic optimization. M.: Nauka, 1977. 236 p.
21. Yampolsky S.M., Lisichkin V.A. Forecasting scientific and technological progress. M.: Economics, 1974. 302 p.
There are often many alternatives to choose from, each with different advantages. And how can you choose the best one, having the opinion of dozens, or even hundreds of experts?
Both the calculation of the rating of a computer game, based on critics’ assessments of graphics, gameplay and plot, and the collective selection of a priority task before the appearance of a customer, belong to the methods of expert assessments.
In cases of extreme complexity of the problem, its novelty, insufficiency of available information, and the impossibility of mathematical formalization of the solution process, one has to turn to the recommendations of competent specialists who know the problem perfectly well—experts. Their solution to the problem, argumentation, formation of quantitative estimates, processing of the latter by formal methods are called the method of expert assessments.
Ways to measure objects
Method for setting weight coefficients (a ij)
IN pairwise comparison There is no need, as in ranking, to order all objects; it is necessary to identify a more significant object in each pair or establish their equality. Paired comparison can be carried out with a large number of objects, as well as in cases where the difference between objects is so insignificant that ranking them is practically impossible.
When using the method, a matrix of size n x n is most often compiled, where n is the number of objects being compared. 
When comparing objects, the matrix is filled with elements a ij as follows (another filling scheme can be proposed):
And now, the best part...
Moreover, the result may consist of several algorithms, intertwined with others. For example, the algorithm for calculating the expert’s competence coefficient can influence the average statistical assessment of this expert, etc.
Average linear deviation 
Standard deviation 
Dispersion 
Spearman's rank correlation coefficient 
The coefficient (value) can vary in the range from –1 to +1. If the estimates are completely identical, the coefficient is equal to one. The coefficient is equal to minus one when there is the greatest discrepancy in expert opinions.
x ij – rank ( importance), assigned to the i-th object by the j-th expert, x ik is the rank assigned to the i-th object by the k-th expert, d i is the difference between the ranks assigned to the i-th object.
Kendell's coefficient of concordance
The coefficient can take values ranging from 0 to 1. With complete agreement of expert opinions, the concordance coefficient is equal to one, while complete disagreement is zero. The most realistic case is the case of partial agreement of expert opinions.
Calculation
The average rank of the set of characteristics is determined:
The deviation d j of the average rank of the j-th characteristic from the average rank of the population is calculated:
The number of identical ranks assigned by experts to the j-th attribute – t q – is determined.
The number of groups of identical ranks is determined - Q. The concordance coefficient is determined by the formula: 
Where 
Speaking about the consistency of expert opinions, it is worth mentioning that ranking does not (or does not always imply) distance. That is, for one expert, A>B>C means that A>>B>C, and for another, A>B>>C. And all sorts of correlations and calculations of average ratings will not help here. Alternatively, consider the consistency index. Something like the number of contradictory closed chains of expert opinions (the first believes that A is better than B, the second that B is better than C, and the third that C is better than A) to the number of all such chains.
So I take my leave. Happy holiday everyone, Ramin. And for those who came to look at the girls - here you go
Ministry of Education and Science of Russia
Mari State Technical University
Department of Control and Production of Radio Equipment
on the topic of: Expert assessment methods
Completed: Art. gr. BZD-41
Kopylova I.V.
Checked by: Prof. department Cyprus
Skulkin N.M.
Yoshkar-Ola 2012
Introduction
Expert assessment
Stages of expert assessment
Types of expert assessments
Processing the results of the expert survey
Conclusion
Bibliography
Introduction
Examples of expert assessment methods. How will the economic environment change over time? What will happen to the natural environment in ten years? How will the environmental situation change? Will the environmental safety of industrial production be ensured, or will a man-made desert begin to spread around? It is enough to think about these formulations of natural questions, to analyze how ten or even more so twenty years ago we imagined today to understand that there simply cannot be one hundred percent reliable forecasts. Instead of statements with specific numbers, you can only expect qualitative estimates. However, we engineers must make decisions, for example, about environmental and other projects and investments, the consequences of which will be felt ten, twenty, etc. in the future. years. What should I do? It remains to turn to the methods of expert assessments. What are these methods?
1. Expert assessment
Expert assessment- the procedure for obtaining an assessment of the problem based on the opinion of specialists (experts) for the purpose of subsequent decision-making (choice).
Experts(from the Latin "expertus" - experienced) - these are persons who have knowledge and are able to express a reasoned opinion on the phenomenon being studied.
Expert assessment methods - these are methods of organizing work with expert specialists and processing expert opinions.
The essence of expert assessment methods is that the forecast is based on the opinion of a specialist or a team of specialists, based on professional, scientific and practical experience. There are individual and collective expert assessments.
Expert judgments are often used in selection, for example:
one version of a technical device for launching into a series of several samples,
a group of astronauts from many applicants,
a set of research projects for funding from a mass of applications,
recipients of environmental loans from among many who wish,
when choosing investment projects for implementation among those presented, etc.
2. Stages of expert assessment
1. Setting the purpose of the study.
Expert assessment involves the creation of a certain mind that has greater abilities compared to the capabilities of an individual person. The source of multimind capabilities is the search for weak associations and assumptions based on the experience of an individual specialist. The expert approach allows you to solve problems that cannot be solved in a conventional analytical way, including:
· Selecting the best solution option among the available ones.
· Forecasting the development of the process.
· Searching for possible solutions to complex problems.
Before starting an expert study, it is necessary to clearly define its purpose (problem) and formulate an appropriate question for experts. It is recommended to adhere to the following rules:
· A clear definition of the conditions, time, external and internal limitations of the problem. * The ability to answer a question with accuracy accessible to human experience.
· It is better to phrase the question as a qualitative statement rather than as an estimate of a number. For numerical estimates, it is not recommended to specify more than five gradations.
· Experts evaluate possible options, and one should not expect them to construct a complete action plan or a detailed description of possible solutions.
2. Choosing a research form, determining the project budget.
Existing types of expert assessments can be classified according to the following criteria:
· According to the form of expert participation: full-time, correspondence. The face-to-face method allows experts to focus their attention on the problem being solved, which improves the quality of the result; however, the correspondence method can be cheaper.
· By the number of iterations (repetitions of the procedure to increase accuracy) - one-step and iterative.
· For the tasks to be solved: generating solutions and evaluating options.
· By type of answer: ideological, ranking, evaluating an object on a relative or absolute (numerical) scale.
· According to the method of processing expert opinions: direct and analytical.
· By the number of experts involved: no limit, limited. Typically 5-12 experts are used.
The most well-known methods of expert assessments are the Delphi method, brainstorming and the hierarchy analysis method. Each method has its own timing and need for experts. After choosing the expert assessment method, you can determine the costs of the procedure, which include paying experts, renting premises, purchasing office supplies, and paying a specialist to conduct and analyze the results of the examination.
3. Preparation of information materials, questionnaire forms, procedure moderator.
Before making a judgment, experts must consider the problem presented in a comprehensive manner. To carry out this procedure, it is necessary to prepare information materials describing the problem, available statistical data, reference materials, questionnaire forms, and equipment. The following mistakes should be avoided: mentioning materials developers, highlighting one or another solution option, expressing management’s attitude towards the expected results. The data should be versatile and neutral. It is necessary to develop questionnaire forms for experts in advance. Depending on the method, they can be with open and closed questions, the answer can be given in the form of a judgment, paired comparison, ranked series, in points or in the form of an absolute rating.
The procedure itself is carried out by an independent moderator of the procedure, who monitors compliance with the regulations, distributes materials and questionnaires, but does not express his opinion.
4. Selection of experts.
The problem of selecting experts is one of the most difficult in the theory and practice of expert research. Obviously, it is necessary to use as experts those people whose judgments will most help in making an adequate decision. But how to identify, find, select such people? It must be said frankly that there are no methods for selecting experts that will certainly ensure the success of the examination.
In the problem of selecting experts, two components can be distinguished - compiling a list of possible experts and selecting an expert commission from them in accordance with the competence of the candidates.
Experts must have experience in areas relevant to the tasks being solved. When selecting experts, one should take into account the moment of personal interest, which can become a significant obstacle to obtaining an objective judgment. For this purpose, for example, Schar's methods are used, when one expert, the most respected specialist, recommends a number of others and further along the chain until the necessary team is selected.
5. Carrying out an examination.
The procedure differs depending on the method used. General recommendations:
· Establish and comply with regulations. Increasing the time to make a decision beyond the optimal one does not increase the accuracy of the answer.
6. Statistical analysis of results . After receiving the experts' answers, it is necessary to evaluate them. This allows:
1) Assess the consistency of expert opinions. In the absence of significant expert agreement, it is necessary to identify the reasons for the inconsistency (presence of groups) and recognize the lack of consensus (negligible results).
)Evaluate the research error.
)Build a model of the object’s properties based on the experts’ answers (for analytical examination). The results of the expert assessment are presented in the form of a report. The report indicates the purpose of the study, the composition of the experts, the assessment obtained and the statistical analysis of the results.
7. Preparation of a report with the results of expert assessment.
. Types of expert assessments
Expert assessment methods can be divided into two groups:
§ methods of collective work of an expert group
§ methods for obtaining individual opinions of members of the expert group.
Methods of collective work of an expert group involve obtaining a common opinion during a joint discussion of the problem being solved. Sometimes these methods are called methods of directly obtaining collective opinion. The main advantage of these methods is the possibility of versatile analysis of problems. The disadvantages of the methods are the complexity of the procedure for obtaining information, the difficulty of forming a group opinion based on the individual judgments of experts, and the possibility of pressure from authorities in the group.
Teamwork methods include brainstorming, scenarios, business games, meetings and court.
Brain attack.It is organized as a meeting of experts, whose speeches are subject to one, but very significant restriction - you cannot criticize the proposals of others. You can develop them, you can express your ideas, but you can’t criticize them! During the meeting, experts, “infecting” each other, express more and more extravagant ideas. About two hours later, the meeting recorded on a tape recorder or video camera ends, and the second stage of brainstorming begins - analysis of the ideas expressed. Typically, out of 100 ideas, 30 deserve further development, out of 5-6 they make it possible to formulate applied projects, and 2-3 ultimately bring a useful effect - profit, increased environmental safety, improvement of the natural environment, etc. Moreover, the interpretation of ideas is a creative process. For example, when discussing the possibilities of protecting ships from a torpedo attack, the idea was put forward: “Line the sailors along the side and blow on the torpedo to change its course.” After development, this idea led to the creation of special devices that create waves that knock the torpedo off course.
Method "635"- one of the varieties of brain attack. The numbers 6, 3, 5 represent six participants, each of whom must write down three ideas within five minutes. The leaf goes around in a circle. Thus, in half an hour, everyone will write down 18 ideas, and all together - 108. The structure of ideas is clearly defined. Method modifications are possible. This method is widely used in foreign countries (especially in Japan) to select from a variety of ideas the most original and progressive solutions to certain problems.
Business gamesare based on modeling the functioning of the social management system when performing operations aimed at achieving the set goal. Unlike previous methods, where expert assessments are formed during a collective discussion, business games involve the active activity of an expert group, each member of which is assigned a certain responsibility in accordance with pre-drawn rules and a program.
The main advantage of business games is the ability to develop a solution in dynamics, taking into account all stages of the process under study with the interaction of all elements of the social management system. The disadvantage is the difficulty of organizing a business game in conditions close to a real problem situation.
Meeting method(“commissions”, “round table”) - the simplest and most traditional. It involves holding a meeting or discussion with the aim of developing a single collective opinion on the problem being solved. Unlike the brainstorming method, each expert can not only express his opinion, but also criticize the proposals of others. As a result of such thorough discussion, the possibility of errors in reaching a decision is reduced.
The advantage of the method is the simplicity of its implementation. However, at a meeting, the erroneous opinion of one of the participants may be accepted due to his authority, official position, perseverance or oratorical abilities.
Commission method- one of the methods of expert assessments, based on the work of special commissions. Groups of experts at a round table discuss a particular problem in order to harmonize points of view and develop a common opinion. The disadvantage of this method is that the group of experts in their judgments is guided mainly by the logic of compromise.
Script writing methodis based on determining the logic of a process or phenomenon over time under various conditions. It involves establishing a sequence of events that develop during the transition from the existing situation to the future state of the object. A unique scenario could be a description of the sequence and conditions for the international integration of countries’ economies, including the following questions:
from what simplest forms to more complex forms this process should go;
how it will affect the national economy and economic ties of countries;
what are the financial, organizational, social, legal problems that may arise during the internationalization of the economy.
The forecast scenario determines the development strategy of the forecasted object. It should reflect the general goal of the development of the object, the criteria for assessing the upper levels of the goal tree, the priorities of problems and resources for achieving the main goals. The scenario displays a sequential solution to the problem and possible obstacles. In this case, the necessary materials for the development of the forecast object are used.
The scenario should be written in such a way that, after reading it, the general goal of the work being carried out in the light of socio-economic objectives for the forecast period becomes clear.
It is usually multivariate in nature and covers three lines of behavior:
optimistic - development of the system in the most favorable situation;
pessimistic - development of the system in the least favorable situation;
working - development of the system taking into account counteraction to negative factors, the occurrence of which is most likely.
As part of the forecast scenario, it is advisable to develop a backup strategy in case of unforeseen situations.
The finished script must be analyzed. Based on the analysis of information found to be suitable for the upcoming forecast, goals are formulated, criteria are determined, and alternative solutions are considered.
Court methodis a variation of the “meetings” method and is implemented by analogy with the conduct of a trial.
The chosen solution options act as “defendants”;
in the role of “judges” - decision makers;
in the role of “prosecutors” and “defenders” - members of the expert group.
The role of “witnesses” is played by various selection conditions and expert arguments. When conducting such a “trial,” certain decisions are rejected or accepted.
The “court” method is advisable to use when there are several groups of experts adhering to different decision options.
Methods for obtaining individual opinions of members of the expert group are based on preliminary receipt of information from experts interviewed independently of each other, with subsequent processing of the received data. These methods include questionnaire methods, interviews and Delphi methods. The main advantages of the individual expert assessment method are their efficiency, the ability to fully use the individual abilities of the expert, the absence of pressure from authorities and the low cost of examination. Their main disadvantage is the high degree of subjectivity of the resulting assessments due to the limited knowledge of one expert.
Delphi method.The goal is to develop a program of sequential multi-round individual surveys. Individual surveys of experts are usually carried out in the form of questionnaires. Then they are statistically processed on a computer and a collective opinion of the group is formed, arguments in favor of various judgments are identified and summarized. The computer-processed information is communicated to experts, who can adjust the assessments, while explaining the reasons for their disagreement with the collective judgment. This procedure can be repeated up to 3-4 times. As a result, the range of assessments is narrowed and a consistent judgment is developed regarding the prospects for the development of the object. Features of the Delphi method:
a) anonymity of experts (members of the expert group are unknown to each other, interaction between group members when filling out questionnaires is completely excluded);
b) the possibility of using the results of the previous round of the survey;
c) statistical characteristics of group opinion.
This method helps to predetermine the development of long-term problem situations. Our specialists working in the field of scientific and technical forecasting are also developing methods for processing expert assessments. They are called heuristics.
Interview methodinvolves a conversation between a forecaster and an expert using a question-answer pattern, during which the forecaster, in accordance with a pre-developed program, poses questions to the expert regarding the prospects for the development of the forecasted object. The success of such an assessment largely depends on the ability of the expert to give impromptu opinions on various issues.
Analytical methodinvolves careful independent work by an expert to analyze trends, assess the state and development paths of the predicted object. The expert can use all the information he needs about the forecast object. He draws up his conclusions in the form of a memorandum. The main advantage of this method is the ability to make maximum use of the expert’s individual abilities. However, it is of little use for predicting complex systems and developing strategies due to the limited knowledge of one specialist expert in related fields of knowledge.
. Processing the results of the expert survey
expert collective individual survey
Let's move on to consider the procedures performed at the stage of processing survey results.
Based on expert assessments, generalized information about the object (phenomenon) under study is obtained and a decision is formed, specified by the purpose of the examination. When processing individual expert assessments, various quantitative and qualitative methods are used. The choice of one method or another depends on the complexity of the problem being solved, the form in which expert opinions are presented, and the goals of the examination.
Most often, when processing survey results, methods of mathematical statistics are used.
Depending on the goals of the examination, the following problems can be solved when processing assessments:
· formation of a generalized assessment;
· determining the relative weights of objects;
· establishing the degree of consistency of expert opinions, etc.
1)Formation of a generalized assessment
So, let a group of experts evaluate some object, then x j - assessment of the jth expert, , where m is the number of experts.
To form a generalized assessment of a group of experts, average values are most often used. For example, median (M E ), which is taken to be such an estimate in relation to which the number of large estimates is equal to the number of smaller ones.
A point estimate for a group of experts, calculated as the arithmetic mean, can also be used:
2)Determining the relative weights of objects
Sometimes it is necessary to determine how important (significant) a particular factor (object) is from the point of view of some criterion. In this case, they say that it is necessary to determine the weight of each factor.
One method for determining weights is as follows. Let x ij - assessment of factor i, given by the jth expert, , , n is the number of objects being compared, m is the number of experts. Then the weight of the i-th object, calculated according to the estimates of all experts (wi ), is equal to:
where w ij - the weight of the i-th object, calculated according to the estimates of the j-th expert, is equal to:
3)Establishing the degree of consistency of expert opinions
If several experts participate in a survey, discrepancies in their assessments are inevitable, but the magnitude of this discrepancy is important. A group assessment can only be considered sufficiently reliable if there is good agreement between the responses of individual experts.
To analyze the spread and consistency of estimates, statistical characteristics are used - measures of spread.
Variation range (R):
Xmax - x min ,
where x max - maximum assessment of the object; min - minimum assessment of the object.
The standard deviation, calculated using the well-known formula:
where xj is the assessment given by the j-th expert; is the number of experts.
The coefficient of variation (V), which is usually expressed as a percentage:
The approaches to consistency checking used when assessing objects using the ranking method are specific.
In this case, the result of the expert’s work is a ranking, which is a sequence of ranks (for expert j): x 1j , x 2j, …, x nj .
The agreement between the rankings of two experts can be determined using the Spearman rank correlation coefficient:
where xij is the rank assigned to the i-th object by the j-th expert; ik is the rank assigned to the i-th object by the k-th expert; i is the difference between the ranks assigned to the i-th object.
The value can vary from -1 to +1. If the estimates are completely identical, the coefficient is equal to one. The coefficient is equal to minus one when there is the greatest discrepancy in expert opinions.
In addition, the calculation of the rank correlation coefficient can be used as a way to assess the relationship between any factor and the resulting characteristic (reaction) in cases where the characteristics cannot be measured accurately, but can be ordered.
In this case, the value of the Spearman coefficient can be interpreted similarly to the value of the pairwise correlation coefficient. A positive value indicates a direct relationship between factors, a negative value indicates an inverse relationship, and the closer the absolute value of the coefficient is to one, the closer the relationship.
When it is necessary to determine the consistency in the rankings of a large (more than two) number of experts, the so-called concordance coefficient is calculated - the general rank correlation coefficient for a group consisting of m experts:
Note that the subtrahend in parentheses is nothing more than the average sum of ranks (when summed for each object) received by i objects from experts.
The coefficient W varies in the range from 0 to 1. Its equal to one means that all experts assigned the same ranks to objects. The closer the coefficient value is to zero, the less consistent the expert estimates are.
Conclusion
Experience, intuition, a sense of perspective, combined with information, help specialists more accurately select the most important goals and directions of development, find the best options for solving complex scientific, technical and socio-economic problems in conditions where there is no information about solving similar problems in the past.
The use of the method of expert assessments helps to formalize the procedures for collecting, summarizing and analyzing the opinions of experts in order to transform them into a form that is most convenient for making an informed decision. But it should be noted that the method of expert assessments cannot replace either administrative or planning decisions; it only allows one to replenish the information necessary for preparing and making such decisions. The widespread use of expert assessments is legitimate only where it is impossible to use more accurate methods to analyze the future.
Expert methods are constantly being developed and improved. The main directions of this development are determined by a number of factors, including the desire to expand the scope of applications, increase the degree of use of mathematical methods and electronic computer technology, and also find ways to eliminate emerging shortcomings. Despite the successes achieved in recent years in the development and practical use of the expert assessment method, there are a number of problems and tasks that require further methodological research and practical testing. It is necessary to improve the system for selecting experts, increasing the reliability of group opinion characteristics, developing methods for checking the validity of assessments, and studying hidden reasons that reduce the reliability of expert assessments. However, even today expert assessments in combination with other mathematical and statistical methods are an important tool for improving management at all levels.
Bibliography
1Orlov A.I. Expert assessments. // Factory laboratory. ? 1996. ? T. 62. ? No. 1. ? pp. 54-60.
2Orlov A.I. Expert assessments. Textbook allowance. - M.: 2002.
Beshelev S.D., Gurvich F.G. Expert assessments in making planning decisions. Textbook allowance. - M.: Economics, 1976. ? 287 p.
Evlanov L.G., Kutuzov V.A. Expert assessments in management. - M.: Economics, 1978. ? 133 p.
Management. Textbook allowance. / Ed. Zh.V. Prokofieva. - M.: Knowledge, 2000. - 288 p.
Beshelev S.D., Gurvich F.G. Expert assessments. - M.: Nauka, 1973. - 79 p.
Statistical methods for analyzing expert assessments. - M.: Nauka, 1977. - 384 p.
Moiseev N.N. Mathematical problems of system analysis. - M.: Nauka, 1981. - 487 p.
Litvak B.G. Expert assessments and decision making. - M.: Patent, 1996.
Characteristics of expert assessment methods [Electronic resource]: #"justify">Expert assessment. / Wikipedia. [Electronic resource]: #"justify">Expert assessments. // StatSoft: SPC Consulting. [Electronic resource]: http://www.spc-consulting.ru/app/expert.htm
Need help studying a topic?
Our specialists will advise or provide tutoring services on topics that interest you.
Submit your application indicating the topic right now to find out about the possibility of obtaining a consultation.
The essence of the expert assessment method is that experts carry out an intuitive-logical analysis of the problem with a quantitative assessment of judgments and formal processing of the results. The generalized expert opinion obtained as a result of processing is accepted as a solution to the problem.
The characteristic features of the expert assessment method as a scientific tool for solving complex non-formalizable problems are:
· scientifically based organization of all stages of examination, ensuring the greatest efficiency of work at each stage,
· the use of quantitative methods both in organizing the examination and in assessing expert judgments and formal group processing of results.
A special type of expert assessment method is an expert survey - a type of survey in which the respondents are experts - highly qualified specialists in a certain field of activity.
An expert is a competent person who has deep knowledge about the subject or object of research.
The method implies the competent participation of specialists in the analysis and solution of the problem under consideration.
In the practice of psychological and sociological research it is used for:
· forecasting the development of a particular phenomenon;
· assessment of the existing state of any phenomenon;
· collecting preliminary information about the research problem (probing);
· assessment of psychological and pedagogical characteristics of students;
· team assessments;
· personnel certification (the role of experts is played by the leaders of the team, public organizations or a special certification commission).
When making decisions about a diagnosis, we usually assume that the information used to support it is valid and reliable. But for many pedagogical and psychological problems this assumption cannot be proven. Practice shows that the main difficulties that arise when searching for and choosing solutions relating to various psychological situations are primarily due to the insufficiently high quality and incompleteness of available statistical information or the impossibility in principle of obtaining it. Then the expert method comes to the rescue, which allows you to look at the problem broadly and see a possible solution.
The reliability of assessments and decisions made on the basis of expert judgments is quite high and largely depends on the organization and focus of the procedure for collecting, analyzing and processing the opinions received. The results of a survey of expert groups differ significantly from decisions formed as a result of discussions at commission meetings, where the opinion of authoritative or merely “assertive” participants may prevail. This does not mean that the individual opinion of a particular specialist or the decision of such a commission is not significant. However, properly processed information obtained from a group of experts, as a rule, turns out to be more reliable and reliable.
This method will not work when:
· the initial statistical information is not reliable enough;
· some of the information is qualitative in nature and cannot be quantified;
· in principle, it is possible to obtain the necessary information, but at the time of making a decision it is not available, since this is associated with a large investment of time or money;
· there is a large group of factors that may affect the implementation of a decision in the future, but they cannot be accurately predicted.
The reliability of group expert assessment depends on the total number of experts in the group, the proportion of various specialists in the group, and the characteristics of the experts.
A difficult problem is the formation of a system of expert characteristics that can significantly affect the course and results of the examination. These characteristics should describe the specific properties of the specialist and possible relationships between people that influence the examination.
The selection of experts and the formation of expert groups is a rather complex task, the result of which to the greatest extent determines the effectiveness of the method and the correctness of the solutions obtained. The selection of specialists to participate in the expert survey begins with identifying scientific, technical and administrative problems directly related to solving the task at hand.
A list of persons competent in the required areas is compiled, which serves as the basis for selecting experts. An expert in the full sense of the word is an active participant in scientific research. An attempt to hide the purpose of the research from him, thus turning him into a passive source of information, is fraught with the loss of his trust in the organizers of the research.
Forming a group of potential experts begins with the “snowball” method. Having assessed the number of possible candidates for experts, the issue of the size of the expert group is decided. The optimal number is hardly possible to determine accurately, but it is obvious that in a small group of experts the final assessment is unduly influenced by the assessment of each expert. Too many experts make it difficult to form a consensus opinion. In addition, as the number of participants increases, the role of non-standard opinions, which differ from the majority opinion, but do not always turn out to be wrong, decreases. Involving a large number of experts makes it possible to at least partially compensate for the lack of information, make fuller use of individual and collective experience, and take into account specialists’ assumptions about the future states of objects.
Of fundamental importance is the possibility of ensuring “equality” of scientific specialists in different fields, with different levels of competence, taking into account the specifics of the problem under study.
However, it is possible to establish some general requirements, implying a list of qualities that the “ideal” expert with whom it is preferable to work should have:
· competence of a potential expert in the field under study,
· erudition in related fields,
· experience of scientific or practical work in a certain field,
· official position,
· integrity,
· objectivity,
· ability to think creatively,
· intuition.
To select specialists for the working group, some simple statistical methods and techniques, as well as their combination, are used:
1) experimental (using testing, checking the effectiveness of their previous expert activities);
2) documentary (based on socio-demographic data);
4) using self-assessment (assessment of the degree of competence of the problem being studied, which is given by the potential expert himself).
In addition to these methods, it is possible to calculate the reliability and accuracy of expert assessments presented by any of the potential experts in the past. To do this, calculate the degree of reliability of the expert, which is understood as the relative frequency of cases when the expert attributed the highest probability to hypotheses that were subsequently confirmed (that is, the number of forecasts made by the expert in general is divided by the number of predictions that came true). The concept of expert reliability and accuracy is based on the assumption that there is a class of problems for which an expert is either suitable or unsuitable.
The work of the person organizing the expert survey also includes preparing experts for work, in particular providing them with the most objective data on the problem. Care should be taken to ensure that participants are sufficiently informed about the sources of the problem under study and ways to solve similar problems in the past.
The work includes:
· compilation of special questionnaires for experts. The main tool for an expert survey is a questionnaire or interview form developed according to a special program. Unlike a mass survey, the expert survey program is not so detailed and is predominantly conceptual in nature. In it, first of all, the phenomenon to be considered is clearly formulated and possible variants of its outcome are provided in the form of hypotheses;
· selection of a group of experts;
· setting the problem and presenting questions to the experts;
· choice of method of expert assessments;
· the assessment itself;
· analysis of the obtained data.
It is necessary that the survey conditions contribute to obtaining the most reliable estimates. In order to ensure the independence of assessments, mutual influence of experts should be eliminated whenever possible and the influence of extraneous factors should be reduced. Of great importance is the correct formulation of the questionnaire questions, which allows you to express the expert’s attitude regarding each question in the form of a quantitative assessment and makes it possible to coordinate the assessments received from different experts. If the form of questioning experts involves their face-to-face interaction, care must be taken to ensure that the opinions of the most famous and authoritative experts do not set the tone for the discussions (for this, when speaking, the floor is first given to “ordinary” participants, and then to the most famous and authoritative).
It may also be necessary:
· checking the input data used for expert assessments;
· changing the composition of expert groups;
· repeated measurements on the same issues, followed by comparison of the results with objective information obtained by other methods.
After conducting a survey of a group of experts, the results are processed. The initial information for processing is numerical data expressing the preferences of experts and a meaningful justification for these preferences.
The purpose of processing is to obtain generalized data and new information contained in hidden form in expert assessments. Based on the processing results, a solution to the problem is formed. The presence of both numerical data and meaningful statements from experts leads to the need to use qualitative and quantitative methods for processing the results of group expert assessment.
One of the most critical stages in processing the collected information is the coordination of expert opinions, which can be done based on one of the following rules:
Majority rule - the assessment of a phenomenon or the solution to a problem that is adhered to by the majority of experts is chosen (however, it should be noted that there are often situations when experts who give more reliable assessments find themselves in the minority);
Average score rule—either a simple or weighted average score of expert opinions is determined.
Qualitative analysis of the reasons for the grouping of opinions allows us to establish the existence of different views, concepts, identify scientific schools, determine the nature of professional activity, etc. All these factors make it possible to more deeply understand the results of the expert survey.
Existing types of expert assessments can be classified according to the following criteria:
1 According to the form of expert participation:
· correspondence
The face-to-face method allows you to focus the expert’s attention on the problem being solved, which improves the quality of the result, but the correspondence method can be cheaper. The choice of options for working with experts is determined by the specifics of the problem and the situation. Face-to-face options for working with experts allow you to collect better information, although there are organizational difficulties and mutual influence of experts. Correspondence forms of working with experts make it possible to neglect geographical boundaries when interviewing experts, excludes their mutual influence, but makes the work of expert groups inefficient.
Types of face-to-face survey:
1. Free interview with experts. It has an intelligence purpose and is more often used when it is necessary to more accurately present the problem, clarify some nuances, more clearly interpret the concepts used and outline the main directions of research. The number of experts interviewed here is small (10-15), but the main thing is that the selected experts should be representatives of different professional and scientific points of view. This interview is conducted by an experienced sociologist.
2. Questionnaire survey of experts.
3. “Brainstorming”, “brain attack” - direct exchange of opinions, stimulating observation. The main goal is to find a solution or ways to solve any scientific or practical problem.
Types of correspondence survey:
1. Postal questionnaire survey of experts
2. Delphic technique - consists of developing consensus opinions by repeatedly repeating a survey of the same experts. After the first survey and generalization of the results, its results are communicated to the participants of the expert group. Then a repeat survey is conducted, during which the experts either confirm their point of view or change the assessment in accordance with the majority opinion. This cycle contains 3-4 passes. During such a procedure, an assessment is developed, but the researcher, of course, should not ignore the opinion of those who, after repeated surveys, remained at their point of view.
Obviously, this type of work with experts is very labor-intensive and complex, although the use of the Delphic technique also has its advantages: the anonymity of the survey is ensured by excluding interaction between experts; establishing feedback in the form of reporting processed information about the agreed point of view of experts at the previous stages of the survey; eliminating mutual influence of experts. The Delphi method is not intended to achieve complete unity of expert opinions on the substance of the issue, therefore, despite the convergence of points of view, differences in expert opinions will still exist. The disadvantage of this type of expert survey is the dependence of the assessments given by the experts on the wording of the questions and argumentation; influence of public opinion on experts.
2. According to the tasks to be solved:
· generating solutions;
· evaluating options.
3. By type of answer:
· ideological;
· ranking;
· evaluating an object on a relative or absolute (numerical) scale.
4. According to the method of processing expert opinions:
· direct;
· analytical.
5. By the number of experts involved:
· without restrictions;
· limited (usually 5 - 12 experts are used).
The most well-known methods of expert assessments are the Delphi method, brainstorming and the hierarchy analysis method. Each method has its own timing and need for experts.
Bibliography:
1. Beshelev S.D., Gurvich F.G. Expert assessments. M.: Nauka, 1973. 246 p.
2. Beshelev S.D., Gurvich F.G. Mathematical and statistical methods of expert assessments. M.: Statistics, 1980. 263 p.
3. Kardanskaya N. Making management decisions. M.: UNITY, 1999. 407 p.
The main stages of processing expert assessments:
· determination of the competence of experts;
· determination of a generalized assessment;
· construction of a generalized ranking of objects in the case of several evaluated objects or alternatives);
· determination of dependencies between rankings;
· assessment of the consistency of expert opinions. In the absence of significant agreement between experts, it is necessary to identify the reasons for the inconsistency (presence of groups) and recognize the lack of consensus (negligible results);
· assessment of research error;
· building a model of the properties of an object (objects) based on the answers of experts (for analytical examination);
· preparation of a report (indicating the purpose of the study, the composition of experts, the assessment obtained and analysis of the results).
26. Selection of experts and their survey.
It is often proposed to use methods of mutual assessment and self-assessment of the competence of experts. On the one hand, who can know the capabilities of an expert better than himself? On the other hand, when self-assessing competence, the degree of self-confidence of the expert is assessed rather than his actual competence. Moreover, the very concept of “competence” is not strictly defined. It can be clarified by highlighting its components, but this complicates the preliminary part of the expert commission’s activities.
When using the mutual assessment method, in addition to the possibility of showing personal and group likes and dislikes, the experts’ ignorance of each other’s capabilities plays a role. In modern conditions, only specialists who have worked together for many years can have a fairly good acquaintance with each other’s work and capabilities. However, attracting such pairs of specialists is not very advisable, since they are too similar to each other.
The use of formal indicators (position, academic degree and title, length of service, number of publications...) can obviously be of an auxiliary nature. Successful participation in previous examinations is a good criterion for the activities of a taster, doctor, judge in sports competitions, i.e. such experts who participate in long series of similar examinations. However, alas, the most interesting and important are the unique examinations of large projects that have no analogues.
If the expert survey procedure involves the joint work of experts, their personal qualities are of great importance. One “talker” can paralyze the activities of the entire commission. In such cases, it is important to comply with the work regulations developed by the WG.
There is a useful “snowball” method, in which from each specialist brought in as an expert, several names of those who may be experts on the topic under consideration are obtained. Obviously, some of these names were encountered earlier in the activities of the RG, and some are new. The process of expanding the list stops when new names stop appearing. The result is a fairly extensive list of possible experts. It is clear that if at the first stage all the experts were from the same “clan”, then the “snowball” method will most likely yield people from this “clan”; the opinions and arguments of other “clans” will be missed.
It must be emphasized that the selection of experts is ultimately the function of the Working Group, and no selection methods relieve it of responsibility. In other words, it is the Working Group that is responsible for the competence of the experts, for their fundamental ability to solve the task. An important requirement is for the decision maker to approve the list of experts.
There are a number of normative documents regulating the activities of expert commissions in certain areas. An example is the Law of the Russian Federation “On Environmental Expertise” of November 23, 1995, which regulates the procedure for the examination of “planned economic or other activities” in order to identify possible harm that the activity in question may cause to the natural environment.
27. Processing information received from experts, checking its consistency and reliability.
Expert assessments are the methods of a general group of scientific research methods used to evaluate complex systems at a qualitative level.
When using expert judgment, it is generally assumed that the opinion of a group of experts is more reliable than the opinion of an individual expert. Some theoretical studies note that this assumption is not obvious, but at the same time argue that, provided certain requirements are met, in most cases group assessments are more reliable than individual ones. Therefore, it is important when organizing expert surveys to introduce certain rules and use appropriate methods for obtaining and processing expert assessments.
The algorithm for organizing expert surveys includes stages at which the following issues are considered:
· problems of forming expert groups, including requirements for experts, group sizes, issues of training experts, assessing their competence;
· forms of expert surveys (various types of questionnaires, interviews, mixed forms of surveys) and methods of organizing surveys (including survey methods, brainstorming, business games, etc.);
· approaches to assessment (ranking, standardization, various types of ordering, including methods of preferences, paired comparisons, etc.);
· methods for processing expert assessments;
· methods for determining the consistency of expert opinions, the reliability of expert assessments (including statistical methods for assessing variance, probability for a given range of changes in assessments, Kendall, Spearman rank correlation, concordance coefficient, etc.) and methods for increasing the consistency of assessments through appropriate methods of processing results expert survey;
· options for interpreting the results obtained.
The appropriateness of using a particular method is determined by the nature of the problem being analyzed and the information used.
If only qualitative assessments of objects based on certain qualitative characteristics are justified, then methods of ranking, paired and multiple comparisons are used. If the nature of the information being analyzed is such that it is advisable to obtain numerical estimates of objects, then one or another method can be used, ranging from direct numerical estimates using scales to more subtle methods of sequential comparison, for example, Churchman-Ackoff.
Examination procedures
During the examination process, errors may arise - biases in assessments introduced by the very procedure of collecting and analyzing expert opinions. Therefore, experts in the field of expert methods pay great attention to the development of reliable rules for preparing and conducting examinations. The most significant rules and stages of conducting examinations are discussed below.
To prepare the examination, a group of organizing specialists must be formed. This group is designed to provide conditions for the effective work of experts and to develop an examination procedure that is most appropriate to the nature of the problem under consideration. The tasks of the group include:
· Statement of the problem, determination of the goals and objectives of the examination, its boundaries, main stages;
· Development of examination procedures;
· Selection of experts, verification of their competence and formation of expert groups:
· Conducting a survey and agreeing on assessments;
· Formalization of received information, its processing, analysis and interpretation.
The number of experts in the group should not be small, since in this case the meaning of forming expert assessments will be lost. In addition, group scores would be significantly influenced by each assessor's assessment. At the same time, with a very large number of experts, the assessment of each of them has almost no effect on the group assessment. Increasing the size of the expert group does not always increase the reliability of assessments, because often expanding the group of experts is only possible by attracting low-qualified specialists. With the increase in the number of experts, in addition, the difficulties associated with processing the survey results and coordinating the work of the group increase.
Before an expert survey, rules for its conduct and organization must be developed, containing a number of provisions that are mandatory for all experts. These rules should ensure compliance with conditions conducive to the formation of an objective opinion by experts. These conditions include:
· Independence of experts forming their own opinions about the events being assessed;
· Maintaining the anonymity of responses;
· Possibility of holding collective discussions of the events being assessed;
· Providing experts with the required information.
Depending on the importance and complexity of the problem and, accordingly, the tasks assigned to the group of organizers, it includes up to five to seven people - specialists in the given field (areas) of knowledge, as well as specialists in expert methods (sociologists, psychologists, mathematicians).
Preparation of the examination begins with problem statement. To do this, first of all, get acquainted with the background and state of the problem, establish its place and significance. After this, a preliminary analysis of the problem is carried out, all external and internal connections are clarified, and the boundaries of the material included in the consideration are determined. To do this, the organizers of the examination put forward a central question that constitutes the essence of the problem, and then “split” it into sub-questions, while limiting the “field” of consideration only to those sub-questions, without answers to which it is impossible to obtain an answer to the central question.
When solving complex socio-economic, scientific and technical problems, experts representing different fields of knowledge turn to concepts from different disciplines. Therefore, it is necessary to formulate the basic concepts. used during the examination.
Depending on the goals and objectives of the examination, on the selection of specialists participating in it, the organizers of the examination select survey method: individual or group (collective), personal (full-time) or correspondence, oral or written.
Regardless of whether we are talking about a questionnaire or an interview, the basis of the survey is a questionnaire, with the help of which the required information is collected. Translating the purpose and objectives of the examination into the language of questions requires complex and painstaking work from the organizers of the examination, knowledge of various types of questions, the ability to accurately formulate them, and arrange them in a certain sequence.
A questionnaire is a structurally organized set of questions, each of which is logically related to the central task of the examination. All questions in the questionnaire, depending on their content, can be divided into three groups: information about the expert himself (his age, position, work experience, education, scientific title, narrow specialization, etc.); questions on the essence of the problem under study; questions to assess the expert's motives in his analysis.
The form distinguishes between open, closed and semi-closed questions; direct and indirect. A question is considered open (free) if the answer to it can be given in any form and is not regulated in any way, closed - if its wording contains options for possible answers, and the expert must choose one or more; semi-closed, if the list provides for the possibility of any additional comments.
There are three types of questions for which expert assessment is given - questions whose answers contain a quantitative assessment (1), requiring a meaningful answer in a condensed (2) and expanded (3) form.
The work of selecting experts participating in the examination usually begins with compiling a list of persons competent in the field under study. This list serves as the basis for selecting experts using special methods for assessing their qualities. There are four main groups of such methods: self-assessment; assessment by a group of each specialist; assessment based on the expert’s past performance; methods for assessing the competence of candidate experts.
©2015-2019 site
All rights belong to their authors. This site does not claim authorship, but provides free use.
Page creation date: 2016-04-27
In the context of the implementation of the Federal State Educational Standard, it is of great importance. Over the course of decades, educational institutions have developed a system of work...
The manual discusses programs of domestic and foreign authors that are used in the work of modern...
How often do we see this monster with goat horns when we lay out Tarot cards. "Devil" is the personification of hell and death...
Navigation: Description and history of the map: To the side of the road there is a large oak tree, in its foliage birds find shelter and food...
Dear forum visitors, I am opening this topic for fans of Patrick Walesa's "Deviant Moon Tarot" deck. Tarot...
Tuscan order. Tuscan order, one of the five Roman architectural orders. The name is associated with Etruscan architecture...
LIBRETTO. If you love music (which you undoubtedly do, since you needed this book), then you'll probably...
Religion: paganism Birth: 942 (0942) Death: March on the Dnieper Family: Rurikovich Father:...
Masaccio (actually Tommaso di Giovanni di Simone Cassai (Guidi), Tommaso di ser Giovanni di Guidi; December 21...
(Published in Tel Aviv in 1958. Transmitted with minor abbreviations). Jews! Love each other,...
Take a ladle or saucepan and pour milk and cream into it. Add vanilla sugar (sugar with vanilla seeds). IN...
Hello everyone! Today we have a very tasty and simple recipe in front of us - sausages in dough in the oven. Do you feel...
Avocado is one of the healthiest fruits. This is a real storehouse of vitamins and nutrients. And him...
Roast pork is an ideal tasty and satisfying dish that can feed the whole family. Mistresses...
The manual discusses programs of domestic and foreign authors that are used in the work...
How often do we see this monster with goat horns when we lay out Tarot cards. "Devil" is the personification of hell...