Measure Agent Quality, Cost, and Time Saved

Measure an AI agent by the quality and cost of a reviewed result, including the work a person must do before and after generation. A fast first draft is not a time saving if checking and repairing it takes longer than completing the task directly.

Choose one repeatable task, define what “ready to use” means and compare equivalent inputs. Keep setup time, model and tool charges, human review, corrections and failures separate. This guide provides an evaluation worksheet; its numbers are fictional examples, not measured Finta performance or prices.

Define the result before timing the run

For an investor meeting brief, the unit is one brief that the founder has checked and can use. It is not a model response, a tool call or a paragraph. Define its required fields: correct person and firm, source-dated facts, meeting objective, unresolved questions and an appropriate next step.

Use the same evidence packet and task boundary for the manual and assisted versions. Comparing a manual deep-diligence report with an AI summary would tell you little. Also record whether the manual reviewer already knows the investor, because that changes the work required.

Score quality with observable checks

Criterion	0: not ready	1: needs correction	2: ready for the stated use
Identity	Wrong person or organization	Ambiguous entity needs resolution	Person, firm and role match the supplied evidence
Source traceability	Important claims lack a source	Some claims need better locators or dates	Material claims point to the supporting passage and date
Completeness	A required field is missing	Required fields exist but need clarification	The agreed brief fields are complete or explicitly unknown
Authority boundary	Output implies an unauthorized action	Draft state needs clearer labeling	Preparation, permission and actual action are clearly distinct
Decision usefulness	No usable next question or decision	The next step is plausible but vague	The reviewer can name the next decision and its evidence

The maximum is ten points, but a total should not conceal a critical error. A wrong recipient or an unauthorized-action claim can make the brief unusable even when the other columns score well. Decide these stop rules before reviewing outputs.

In the fictional example, a brief scores 2 for traceability, 1 for identity, 1 for completeness, 2 for authority boundaries and 1 for decision usefulness: seven out of ten. It still needs correction. “Seven” is a diagnostic, not permission to send it to an investor.

Capture the whole cost of a usable result

Cost or time item	Record	Common mistake
Initial setup	Instructions, connections, packet preparation and evaluation design	Leaving setup out because it happened before the first run
Preparation per task	Finding the correct records and providing missing context	Calling all source collection automated
Model and tool use	Observed charge or credit consumption for the specific run	Treating an assumed price as the account’s actual bill
Human review	Time spent checking identity, facts, scope and output	Counting review as optional when it is necessary for use
Corrections and reruns	Editing, new inputs, failed calls and repeated work	Reporting only the fastest successful generation
Disposition	Used, corrected, rejected, blocked or abandoned	Treating a completed run as an accepted output

For money, use observed account charges where available and identify the currency and billing basis. If the product shows credits rather than a cash price, retain that unit. Do not turn credits into currency using an invented conversion rate.

For time, distinguish elapsed waiting from hands-on work. A run may take longer on the clock while reducing attention, or finish quickly while requiring extensive review. Record both when they affect the decision.

Work through an illustrative comparison

A fictional founder usually spends 35 hands-on minutes producing a reviewed meeting brief. Setting up the assisted process takes 20 minutes. For the next four comparable briefs, the founder allocates that setup as five minutes per brief. One assisted brief then needs four minutes of input preparation, twelve minutes of review and six minutes of correction.

The allocated hands-on total is 5 + 4 + 12 + 6 = 27 minutes. Compared with the 35-minute manual baseline, the illustrative difference is eight minutes per brief. This is an assumption-based teaching calculation, not an observed saving or a general forecast.

If only one brief is produced, setup is not spread over four results: 20 + 4 + 12 + 6 = 42 minutes. If a second run fails, its preparation and review still count. The answer changes with actual reuse and output acceptance.

Use a small run register

Run	Input version and task	Quality and disposition	Hands-on time	Observed model/tool units	Correction to test next
A	Same packet, manual baseline	Reviewed and accepted	[minutes]	Not applicable or actual tools used	Note prior knowledge
B	Same packet, assisted process	[criterion scores]; used/corrected/rejected	[setup + preparation + review + correction]	[actual units, or unknown]	One specific instruction or source fix
C	Missing-source case	Should identify the missing evidence	[minutes]	[actual units, or unknown]	Check whether the failure was exposed
D	Conflicting-source case	Should retain both assertions and uncertainty	[minutes]	[actual units, or unknown]	Check the source-selection rule

Keep rejected and blocked runs in the register. Otherwise the results favor the tool simply because unsuccessful work disappeared from the denominator. If you report an acceptance rate, define it as accepted reviewed outputs divided by all comparable assigned tasks in the chosen period, and state exclusions.

Choose the next experiment from the evidence

If identity errors dominate, improve the entity inputs before adjusting writing style. If review takes most of the time, require precise claim locators and shorter output. If the task changes each time, repeated setup may make a simple manual checklist preferable.

Change one important condition at a time and keep the old input and output. A better second run does not establish reliability across clients, sources or models. Recheck after material instruction, model, tool or source changes.

Common questions

Can I use run count as a productivity metric?

Run count measures activity. Use accepted reviewed outputs and the decisions they support to judge usefulness.

Where do I get Finta costs?

Use the actual account’s available usage and run evidence. Current implementation includes cost and credit paths, but this guide has not executed a metered product run.

What if the agent is blocked?

Keep the blocked attempt in the register and use the recovery worksheet. Review the task and sources before retrying.

Sources checked October 22, 2026. The scorecard and arithmetic are original fictional educational examples. Explore Aurora by Finta for current product context.