To: MAB Distribution From: Gary C. Dixon John W. Gintell R. C. vonSeeburg Date: February 13, 1981 Subject: Revised MAB on TR Goals PLEASE REMOVE THIS COVER LETTER BEFORE FILING THIS MAB This MAB describes changes to MAB-044, which defines Multics TR Processing Goals. Most significant changes in definition of goals include the following: (1) On page 3 in discussion of response, the following sentence was removed. Response time includes the time needed to route the response back to the submitter. The TR System does not include such time. Perhaps it should, but it doesn't. (2) Removal of concept of installed-in-EXL being a resolution to a problem. This decision was made some time ago, but the MAB was never updated to reflect the policy change. (3) The TR goal is stated in terms of completing processing of 75% of TRs within the expected amount of time (times given in Table 1). The prior MAB did not state this 75% goal, implying our goal was to process the average TR within the expected time. The goal is measured as percent_completed_on_time for each type and processing stage, weighted by the number of reports completed in that stage. The exact algorithm is given in the MAB. All other changes are minor wording changes or clarifications. The intent of this revision is to reflect the current policies and procedures now in effect. MULTICS ADMINISTRATIVE BULLETIN MAB-044, Rev 1 To: MAB Distribution From: Gary C. Dixon John W. Gintell R. C. Von Seeburg Date: February 4, 1981 Subject: Goals for Processing Multics Trouble Reports
INTRODUCTIONThis bulletin defines performance goals for processing of Trouble Reports (TRs) entered by Multics users through their SiteSAs, and categorized in the TR System as external TRs.
DEFINITION OF TERMSThe criteria for measuring achievement of TR processing goals are defined using the following terminology.
Types of TRsThere are three kinds of trouble reports. 1. Problem Reports: reports of problems in installed system programs, info segments, or system documentation. 2. Questions: questions about operation of the system, or about installed system programs, info segments and documentation. 3. Suggestions: suggestions for changes/improvements to installed system programs, info segments, or system documentation. _________________________________________________________________ Multics Project internal working documentation. Not to be reproduced or distributed outside the Multics Project. _________________________________________________________________
Problem Report PrioritiesProblem reports are further divided into Priority groups, according to the significance of the problem. 1. Problem Reports A. Critical Priority, a problem which: - interferes with system operation to an extent that usefulness of the system is significantly limited for most users of the system. (1) _________________________________________________________________ (1) This includes, but is not limited to, situations in which a site is totally down, or is running in a highly crippled state. _________________________________________________________________ - irreparably compromises the security or integrity of data stored in the system. Data storage problems at system priority must affect many system files, or files which are difficult to retrieve or recreate. B. High Priority, a problem which: - causes installed system programs to fail to operate correctly in limited circumstances, with no known bypass for the problem. Such problems can be tolerated for short periods of time. - compromises data security or integrity for a small number of files in a way which can be repaired without significant loss of data. Such problems can be tolerated for short periods of time. C. Normal Priority, a problem which: - causes installed system programs to fail in a minor way which can be bypassed, or ignored. - results from incorrect documentation.
TR Processing OperationsProcessing of TRs involves the following steps. 1. Submission occurs when a SiteSA or user enters a problem report, suggestion, or question. A report is entered by using the enter_problem_report (epr), enter_suggestion (es), or enter_question (eq) command on System M. A SiteSA usually reports Critical problems to Multics Software Support (MSS) by telephone to avoid delay in processing. For performance tracking purposes, MSS personnel enter such telephoned TRs into the TR system after the initial telephone conversation is complete. 2. Verification of a report validates the user's statement of problem, or initially appraises suggestions, or answers questions when answer is known. Verification is performed by the TR Administrator or investigator in Phoenix. 3. Routing of a report forwards the report to developers, investigators, and interested parties. Routing is performed by the TR Administrator in Phoenix. 4. Response to a problem report: acknowledges existence of a bug, indicates a bug bypass if any, and states that problem will be fixed in a future release; or it indicates that the problem cannot or will not be fixed (a system limitation). No response is required for questions or suggestions. (2) _________________________________________________________________ (2) The answer to a question is provided when the question report is resolved. Current management policy states that response to, and resolution of, suggestions by the developer is optional. _________________________________________________________________ Response is made by the developer, investigator or TR Administrator, and is forwarded to the submitter by the TR Administrator. A response is entered by using the answer_trouble_report (atr) command. 5. Resolution of a problem report occurs when changes to installed system programs causing the problem are submitted for installation in the Multics System Libraries, or when documentation errors are corrected; or when the nature of a user error causing the problem is pointed out to the submitter; or when the report is unclear or unreproducible and additional information about the problem is required. Resolution of a question report occurs when the question is answered. Software problems are considered resolved when the programs are submitted for installation in System Libraries on System M, because at this point, an emergency fix can be shipped to the site if this is deemed necessary. Bug fixes are not normally distributed between releases however, and the time from submission of installation of software on System M until distribution of the bug fix is NOT included in the resolution time. Documentation is considered fixed when the corrected manual is sent to the Brighton Print Shop. A trouble report resolution is entered by using the answer_trouble_report (atr) command. Developers are responsible to entering notification that a problem fix has been submitted for installation into the System Libraries. The Manager of Multics Technical Documentation at CISL and the Project Leader for the Multics Documentation Project in Phoenix are responsible for entering final resolution notice when manuals containing corrections are sent to the Brighton Print Shop.
TR PROCESSING PERFORMANCE GOALThe goal for TR processing performance states that 75% of all TR processing operations will be completed within a specified amount of time, where times are measured from submission of the TR into the TR system to completion of the TR processing operation. The procedures for TR processing state the order in which TRs are to be processed, and the expected amount of time needed to perform each TR processing operation.
Expected TR Processing TimesThe expected amount of time required to perform each TR processing operation is shown in the table below. Times are expressed as the number of calendar weeks which should elapse from time of submission to the time of operation completion, averaged over all TRs whose operations have completed during the reporting period. Because the times are averages, they do not state the processing performance expected for a particular TR. They instead address our overall TR processing performance. Table 1: Average Calendar Weeks from Submission to Completion of TR Processing Operation OPERATION Problem Problem COMPLETED Critical High Query Normal Suggestion ________________________________________________________________ | | Verification | 0.2 1 1 2 4 | | | Response | 1 2 4 13 -- | | | Resolution | 4 13 4 52 -- | |________________________________________________| Verification and routing operations are performed as a single processing operation, and are therefore shown as a single processing goal in the table above. Also, as the table indicates, no response is required for questions. No response or resolution is required for suggestion reports since current management policies state that response and resolution of suggestions is optional.
Formula For Computing Performance To Goal_____ _____ \ \ # ops completed total_ops = > > since 07/01/80 /____ /____ for problems (critical, for verification, high & normal) and response and questions resolution ops _____ _____ \ \ # ops %_on_time %_on_time = > > completed * for op /____ /____ since 07/01/80 for problems (critical, for verification, high & normal) and response and questions resolution ops %_performance = %_on_time / total_ops Our goal is: %_performance >= 75%
Order of Processing TRsTRs should be processed in the following order of importance: 1. Problem Reports, Critical Priority 2. Problem Reports, High Priority 3. Questions 4. Problem Reports, Normal Priority 5. Suggestions Type of report and priority of problem reports are initially specified by the submitter of the report, but can be adjusted by investigators, TR Administrator (who routes TRs) and developers as the report becomes better understood.
REPORTING OF PERFORMANCETR Processing performance will be reported to the Director of Software Development on a quarterly basis. It will be reported weekly to the Manager of Multics Software Development in status reports. Reporting will begin with statistics for reports submitted during the third quarter, 1979.