This System Is Made of Twelve Parts ── Component Design of AI Material Review

01The whole map ── three planes and twelve parts

First, the whole. There are twelve parts. They are placed on three planes. A plane is a shelf for parts with the same kind of job.

The first is the "gathering plane." It takes the material and the documents it will be compared with, and puts them in a fixed form. The second is the "judging plane." It makes flags from clear-cut checks and from the difference in the picture left in the reader's head. The third is the "learning plane." It receives the human's final verdict and counts the misses. Figure 2 shows the layout.

The planes are separated to contain the effect of a change. If a law or guideline is revised, only the gathering plane moves. If the AI is replaced with a newer product, only the AI-using parts on the judging plane sit the test. If the human's criteria change, only the notebooks on the learning plane grow. A change on one plane leaves the other planes untouched.

02The promises between parts ── the words for verdicts, and the shape of a flag

The parts connect through fixed promises. There are two.

The first is the words used for verdicts. On the legal side there are five labels: "clear violation," "touches a higher rule," "slips through a gap," "against the spirit of the rules," and "no issue." On the public-eye side there are four: "to management," "to a human," "keep on record," and "no issue." The two sets are never mixed. A flag can be light on the legal side and heavy on the public-eye side. The kinds of flag are also fixed. There are eight: going beyond the approval, making safety look lighter, overstating the effect, hinting, running down a competitor, faulty evidence, a required statement missing, and against common social sense.

The second is the shape of a flag. Every flag carries the following. Where in the material (a quotation). Which check item it hit. Where in the original text of the rule. Which "picture in the head," and which difference in it. Which way of reading made that picture. Past verdicts on similar material. Which part, and which version of it, made the judgment. A flag missing even one of these is not accepted by the program. Because the promises are fixed, the parts can be swapped.

03The gathering plane ── intake clerk, approved-facts store, rule-change tracker

The intake clerk turns the material's PDF or web page into a map a machine can read. The map lists every sentence, figure and note, with its position, its size, how much it stands out, and which note belongs to which figure. Column layout is worked out here, and sentences cut mid-way are joined back together. Nothing unreadable is thrown away. It is kept with a mark, "check with a human." A program does this work.

The approved-facts store holds the documents the material is compared with, in a fixed form. There are three: the official leaflet (the package insert), the government's review report, and the safety-measures plan. The difference taker always compares the picture with these three. Which version of each document was used is recorded for every review. A program does this work.

The rule-change tracker spots revisions to laws and guidelines, updates the check items, confirms them with an exam that has an answer key, and starts a re-examination of material already passed. A program spots the revision, the AI drafts how the check items change, and a human approves.

04The judging plane ── the rule-checker

The rule-checker settles the clear-cut checks first. There are 694 check items in all, and about half have an answer without asking the AI. Is a forbidden word present? Does the chart's vertical axis start at zero? Does the number of cases meet the standard? Is a required statement missing? Checks like these give the same answer every time the same material goes in. A program does this work.

This part has a second job: passing its results to the next part as cues. The fact "this chart's vertical axis does not start at zero" points the image maker toward a candidate picture, "the effect may look larger than it is." Facts found by the program guide the AI's reading, and the AI's reading is judged by the program. That order is the backbone of the system.

05The judging plane ── the image maker and the difference taker

The image maker is the center of this design. The image is the picture left in the head of someone who has finished reading the material. This part takes the map of the material and makes the picture in four ways of reading. Read one sentence alone. Read two neighboring items in a row. Read the booklet from start to finish. Look at one chart cut out on its own. For each way of reading, the AI makes the picture three separate times, and everything that came out is combined. Any implication that appeared even once is kept. There is no majority vote. This is to prevent misses. The AI does this work.

Each sentence in the picture carries marks. Is it written plainly in the material, or an implication read from the arrangement? Which combination of sentences and figures produced it? Some arrangements tend to produce a wrong picture: the word "treatment" in a title next to a safety statement, a competitor's side effects next to one's own mildness, or a relative figure with no actual figure. These are the places this part directs its attention when reading neighbors.

The difference taker compares the four pictures with the approved-facts store. Things not in the approved facts. Things stated more lightly than the approved facts (safety watered down). Things in the approved facts but missing from the picture (safety left out). These three kinds are the differences. Which label a difference gets is decided by the program under fixed rules. Stating plainly that the drug works for a disease that is not approved gets the heaviest label. Hinting at it gets the second label. Three things get the heaviest label whatever the context: guaranteeing safety outright, writing that something unapproved is approved, and tampering with a chart's axis. If the picture gets worse when neighbors are read, or when the whole is read, the label goes up one step. If many sentences and figures feed the same picture, there is one flag with several paths. A program does this work.

06The judging plane ── the public eye and the exception desk

The public eye is the part that stops material which hit no rule from being marked "no issue" anyway. It has the picture of the whole material read by a fixed cast of characters. The cast includes the patient, a family member, a journalist, a government official, a competitor, a lawyer, a physician, a shareholder, and oneself five years from now. If someone feels strongly uneasy, and the material resembles a past incident, a public-eye label is attached. The cast is managed by version, and the version used is recorded on the flag. This label has no power to stop the material. It only decides whether the flag goes to a human or to management. The AI does this work.

The exception desk is the only place a flag can be dropped. Only three fixed kinds may be dropped. A description of a cited clinical trial where only the dose or the interval differs from the official leaflet. Results carrying a fixed note such as "includes doses above the amount approved in Japan." Descriptions of early-stage trials where the stage of development is clearly stated. When a flag is dropped, the reason and the version of the rule are recorded. Even with a note, three things are never dropped: widening the effect or the target disease, watering down safety, and fudging the statistics. A program does this work.

07The judging plane ── merge and score, the virtual review panel, the record and precedent notebook

Merge and score combines overlapping flags on the same picture into the heaviest label, and gives a score. The score is weight × certainty × visibility. Weight comes from the label's level, and going beyond the approval or making safety look lighter adds one step. Visibility is high for patient-facing or web-published material and low for material restricted to healthcare professionals. The material's overall rating starts from the highest score and is adjusted by the number of flags. A single heaviest label automatically gives the lowest rating. Public-eye labels are not scored; they only raise a flag. A program does this work.

The virtual review panel puts only the heavy flags before 60 virtual reviewers. Which flags go there is set by rule. All public-eye and ethics flags. Legal flags at the top two levels, when certainty is half or more. The 30 debaters (experienced reviewers, package-insert specialists, and advertising-regulation lawyers, 10 of each) must give reasons to fix and reasons not to fix, plus minority opinions. A majority of the 30 voters (safety measures, approval review, and advertising surveillance, 10 of each) makes a "should fix" candidate. When the vote looks unanimous, when it is close, when it disagrees with the machine's weight, or when the AI gave no answer, a "check with a human" mark is always added. The panel neither deletes nor softens a flag. The AI does this work.

The record and precedent notebook keeps two notebooks. The first is the record. For every review it adds the material received, the check items, rules, cast and AI version used, the path of each flag, and every drop with its reason. The second is the precedents. It stores the human's final verdict (fix / no fix) and the reason, tied to the type of picture and difference. When a similar picture appears in a later review, this precedent is attached to the flag. If the earlier verdict and this candidate disagree, and it cannot be shown whether the rule version or the precedent changed, the flag goes to a human. A program does this work.

08The learning plane ── the miss ledger

The miss ledger is the only part on the learning plane. When the human's final verdict comes back, it is matched against the machine's candidates. Where the human said "fix" and the machine stayed silent is a miss. Where the machine flagged and the human said "no fix" is an overreach. Misses go into the ledger, with the check item that should have fired, the way of reading that should have been made, and the precedent that should have been consulted.

The ledger is totaled weekly. If misses at the heaviest label exceed 1%, or misses at the second label exceed 5%, live operation stops. Even if they do not, for each type of miss it drafts how to fix the check items, the thresholds, or the instructions to the AI. A human approves the draft, it is confirmed with the exam that has an answer key, and only then does it return to the judging plane. A program does the totals; the AI writes the drafts.

Without this part, "a miss is the gravest failure" stays a declaration. With it, the declaration becomes a weekly number.

09Parts table

Of the twelve parts, the AI works in three, the image maker, the public eye and the virtual review panel, and in two more only for drafts. The rest are programs running fixed steps. The words for verdicts and the shape of a flag are promises that sit outside the parts, so a part can be swapped as long as it keeps the promises. The eight gates (checks you cannot pass until their condition is met) are the same eight listed at the end of the design philosophy article. The order in which this layout runs is followed in the execution process article next.