Finding: Access to AI changed work that individuals could change independently, including less time spent on email and moderately faster document completion. It did not significantly change time spent in meetings, a coordination-dependent activity.
Impact: Strengthen separate measurement of individual throughput and coordination outcomes. Reject productivity change as a proxy for improved collaboration.
Boundary: The report describes access and work-pattern effects, not the quality, safety, or persistence of cross-repository coordination. Next: which repeated-use measures observe handoff quality, correction, and ownership changes rather than only time use?
Flags: working paper; coordination outcome coverage limited
Finding: AI assistance increased issues resolved per hour by about 15% on average, with larger gains for less experienced and lower-skilled workers. The highest-skilled group saw smaller gains and some quality decline.
Impact: Strengthen expertise-sensitive baselines and distributional reporting. Reject one average productivity effect as a routing rule.
Boundary: One support setting with a purpose-built assistant does not establish transfer to open-ended work. Next: when does AI diffuse expert practice without flattening expert judgment?
Flags: selection and rollout confounding; support-work domain
Finding: Pooled results associated coding-assistant access with a 26.08% increase in completed tasks, with higher adoption and gains among less experienced developers.
Impact: Watch as counterevidence to 022. Strengthen context, experience, adoption, and task-mix reporting.
Boundary: The experiments are noisy individually, use completed-task measures, and do not resolve code quality or long-term maintenance. Next: which design differences explain the opposite productivity directions in 022 and 023?
Flags: preprint; pooled organization and adoption effects
Boundary: The sample is small, unusually experienced, and limited to early-2025 tools and familiar repositories. Next: which task and familiarity features predict when assistance becomes overhead?
Flags: small participant count; early-2025 tools; open-source setting
Finding: A structured pair-use protocol was associated with lower document quality and substantially lower document production than unstructured use. A cognitive framing intervention was associated with higher individual quality at the top of the distribution, subject to reported design limitations.
Impact: Watch protocol and framing interventions. Reject structure alone as evidence of better collaboration. No runtime change.
Boundary: This is a preprint with an AM/PM session confound, differential attrition, and LLM-grading sensitivity to document length. Next: can a preregistered, replicated field study show durable benefit without reducing throughput?
Flags: preprint; reported confounds; one organization
Finding: Individuals with AI matched teams without AI on average solution quality, while the study's decomposition found that AI improved idea generation and human teams retained an advantage in evaluative selection.
Impact: Watch role-specific collaboration hypotheses. Strengthen stage-specific outcome measurement instead of a single collaboration score.
Boundary: This is a one-day organizational experiment, not evidence of durable teamwork or authority transfer. Next: can repeated-use studies preserve human evaluative selection while testing whether AI-assisted generation remains beneficial over time?
Finding: Human-AI teams increased communication and some text productivity, but human-human teams produced stronger images and overall campaign results were similar. Trait pairing changed outcomes in both directions.
Impact: Watch participant and modality fit. Strengthen disaggregated outcome and coordination-cost measurement.
Boundary: This is a preprint in a marketing setting with platform-specific interventions. Next: can a matched evaluation identify when another agent changes the result rather than only increasing messages and cost?
Flags: preprint; domain and trait-manipulation transfer
Finding: ChatGPT access reduced completion time by about 40% and raised rated output quality by about 18%, while compressing performance differences between participants.
Boundary: The tasks were short, self-contained, and text based. Next: do gains survive multi-stage work with persistent context, verification, and accountability?
Flags: short writing tasks; no maintenance outcome
Finding: AI improved speed and quality on tasks within its capability frontier, but reduced correctness on a task outside that frontier even when the tasks appeared similarly difficult.
Impact: Strengthen capability-boundary detection and task-specific evaluation. Reject provider-level capability claims as sufficient routing evidence.
Boundary: The frontier was constructed for one model, occupation, and task set. Next: how can a workflow detect boundary crossings before an apparently competent output becomes authoritative?
Flags: one occupation; frontier defined by tested model and tasks
Finding: On average, human-AI combinations performed worse than the better human-only or AI-only baseline. Effects varied: decision-making tasks showed losses, content-creation tasks showed gains, and augmentation depended on which side initially performed better.
Impact: Strengthen matched baseline evaluation. Reject any general claim that adding AI creates synergy.
Boundary: The reviewed evidence is experimental and heterogeneous, with possible publication bias and differing study designs. Next: which coordination mechanisms improve the best-baseline comparison in sustained, privacy-constrained work?
Flags: heterogeneous tasks and interventions; publication bias possible
Finding: The abstract reports that repeated exposure explained several changes over time, while memory-based and survey-based personalization produced different disclosure and regret patterns.
Impact: Watch exposure and personalization mode as separate variables for CI-004 and CI-009.
Boundary: Five days of advice-seeking does not establish durable benefit, calibrated reliance, or a safe memory policy. Next: can an owning-system test separate exposure, current-session context, interaction-derived memory, and explicitly supplied profile data while measuring disclosure and regret? Candidate; source identity verified, abstract claims screened; not adopted
Flags: abstract-only; brief advice-seeking setting; exposure and personalization effects must remain separate; disclosure is not benefit
Finding: GenAI assistance improved the assisted task, but did not establish a lasting independent-performance benefit. In several collaboration-to-solo comparisons, intrinsic motivation fell and boredom rose more than in solo-only sequences.
Impact: Watch continuity and agency effects. Strengthen separate outcome, burden, and post-handoff measures.
Boundary: Professional-role simulations and short task sequences do not establish durable workplace effects. Next: how should a protocol distinguish a useful handoff from a reliance pattern that leaves later independent work worse?
Finding: Repeated interaction with biased AI altered human judgments and amplified bias more than comparable human interaction. Participants often underestimated the AI influence.
Impact: Strengthen correction, counterevidence, and independence checks. Reject passive repeated exposure as a safe reliance model.
Boundary: Judgment tasks and selected algorithms do not map directly to repository work. Next: which correction and provenance signals reduce feedback-loop effects in sustained, open-ended collaboration?
Finding: Human-authored rules could identify useful delegation regions in the studied setting, but creating reliable rules became harder under distribution shift and explanations did not guarantee better performance.
Impact: Watch bounded conditional delegation for stable, observable regions. Strengthen distribution-shift checks before reusing a rule.
Boundary: The work uses relatively low-stakes keyword moderation and an experimental interface. Next: can delegation rules remain legible and safe in open-ended workflows with changing models?
Flags: preprint; content-moderation labels; induced distribution shift
Finding: A more accurate model update can reduce team performance when its new errors violate the human participant's learned expectations.
Impact: Strengthen version-change compatibility and revalidation. Reject higher model accuracy as sufficient evidence that an established workflow improved.
Boundary: The experiments use structured classification decisions. Next: which model, prompt, tool, and policy changes should invalidate prior delegation and reliance evidence?
Finding: Automation, augmentation, and human-only work are task-allocation choices. Their value depends on between-task and within-task complementarity, task difficulty, and the ability to reallocate human effort.
Impact: Test as a candidate capability: evidence-based work allocation. Do not convert it into a universal routing rule.
Boundary: The experiment uses idealized judgment tasks with known ground truth. Next: which non-performance factors, including consequence, accountability, and privacy, must constrain allocation?
Finding: AI assistance produced some complementary performance, but explanations did not increase it. Explanations increased acceptance of recommendations whether they were right or wrong.
Impact: Reject explanation presence as a proxy for collaboration quality. Strengthen separate measurement of human-only, AI-only, and team outcomes.
Boundary: The tasks were bounded classification and reasoning tasks. Next: which outcome set captures quality, effort, reversibility, and accountability in open-ended work?
Finding: Cognitive-forcing interventions reduced overreliance more than simple explanation interfaces, but the strongest interventions received the least favorable subjective ratings and effects varied with need for cognition.
Impact: Strengthen deliberate review at consequential boundaries. Test proportionate friction rather than adding a universal interruption.
Boundary: One experimental task does not establish the right intervention for sustained professional work. Next: what is the least disruptive intervention that improves error detection for a given consequence level?
Finding: Team performance depends partly on whether people can form a usable mental model of the AI's error boundary, which is affected by task complexity and error predictability.
Impact: Strengthen capability-boundary legibility. Reject aggregate accuracy as evidence that a participant can anticipate when to rely.
Boundary: Controlled classification tasks do not capture open-ended tool use or authority. Next: what minimal capability statement remains accurate after configuration and version changes?
Finding: Predictable performance can be worth a small loss in stand-alone accuracy when final decision quality, verification cost, human skill, and mistake cost are considered.
Impact: Watch team utility as an analysis aid. Reject collapsing Breadcrumbs' separate quality, burden, authority, and recovery measures into one score.
Boundary: The utility model assumes a structured accept-or-solve interaction. Next: which variables can inform a decision without becoming a universal optimization target?
Finding: Participants lost confidence in an algorithm faster than in a human after observing comparable mistakes, even when the algorithm performed better.
Boundary: Forecasting choices do not cover open-ended work. Next: how should systems show comparative failure rates without encouraging blanket reliance?
Finding: Limited ability to modify algorithmic forecasts increased use and improved performance, apparently by restoring some control.
Impact: Watch bounded editability and contestability. Reject user control as evidence that the combined result is accurate or fair.
Boundary: Numeric forecasting with restricted edits is narrower than repository work. Next: which edits improve correction without creating untraceable overrides?
Flags: limited numeric edits; use is not correctness
Finding: Lay participants often weighted algorithmic advice more than human advice, but appreciation weakened when comparing advice with their own judgment and among experienced forecasters.
Impact: Strengthen expertise and comparison-frame controls. Reject algorithm aversion as a universal behavior.
Boundary: The tasks center on estimates and forecasts. Next: when does appreciation become overreliance in open-ended work?
Flags: short judgment tasks; novice-expert framing
Finding: Reliance on algorithms was lower for tasks perceived as subjective, and task framing changed perceived objectivity and use.
Impact: Strengthen task-characteristic reporting. Reject adoption differences as direct evidence of capability.
Boundary: Consumer decisions and marketing interventions may not transfer to professional accountability. Next: which task properties predict appropriate, not merely increased, reliance?
Flags: consumer and marketing settings; framing sensitivity
Finding: Risk advice changed how participants weighted risk, increasing detention disparity in one setting and reducing aid in another, despite potential predictive gains.
Impact: Strengthen value-shift and policy-outcome checks. Reject prediction accuracy as sufficient decision quality.
Boundary: Simulated government decisions do not prove field effects. Next: how can a workflow detect when advice changes the objective being optimized?
Finding: Simpler transparent models improved simulation of predictions but did not increase beneficial following and reduced detection of large model mistakes in one condition.
Impact: Strengthen behavioral interpretability tests. Reject transparency or simulability as automatic error-correction evidence.
Boundary: Synthetic prediction tasks do not cover all explanation goals. Next: which explanation improves a named decision behavior without information overload?
Finding: AI assistance improved individual ratings, especially for less creative writers, while making outputs more similar to each other.
Impact: Strengthen portfolio-diversity measurement. Reject average item quality as sufficient collective value.
Boundary: Short creative-writing tasks do not establish organizational innovation effects. Next: when does repeated model assistance narrow the research or design portfolio?
Finding: The abstract describes revisable analysis stages and evidence-linked reporting; incomplete tasks remained in the study.
Impact: Watch stage visibility as a bounded review hypothesis for CI-006 and CI-012. Existing equivalents are the cooperative evaluation protocol and work checkpoint template.
Boundary: Short, domain-specific tasks and participant perceptions of completed work do not establish general workflow effectiveness. Next: does stage visibility improve correction and completion evidence while retaining failures and review burden? Candidate; source identity verified, claims not assessed; not adopted
Flags: abstract-only; small domain-specific task study; incomplete tasks retained; completed-task perceptions do not establish general effectiveness
Finding: The abstract reports initial evidence that manuscript-specific questions can measure whether a person can critically assess an artifact attributed to them.
Impact: Watch task-specific verification capacity for CI-012 and CI-017.
Boundary: A small proctored authorship study does not establish general expertise, workplace accountability, or a fair gate. Next: can a low-stakes review test verify understanding of a specific artifact without inferring broad competence or creating inaccessible barriers? Candidate; source identity verified, abstract claims screened; not adopted
Flags: abstract-only; proctored manuscript-specific task; small sample; automated score is not general competence or a workplace gate
Finding: A single-case study combines lab-in-field experiments and interviews to develop testable propositions about complementarity.
Impact: Watch CI-001, CI-002, CI-004.
Boundary: Primary abstract only; full-text download returned an error. A conceptual model from one case is not a causal demonstration or general effect estimate. Next: Retrieve full methods and comparator details before promoting propositions into guidance. Added candidate with abstract-only appraisal; not full-text validated, replicated or adopted
Flags: abstract-only: Primary abstract only; full-text download returned an error. A conceptual model from one case is not a causal demonstration or general effect estimate.
Finding: Configurable agents, tools, human input, and conversation patterns can compose multi-agent applications. The paper demonstrates feasibility and flexibility, not a general advantage over one agent or proof of governance.
Impact: Watch as implementation evidence. Do not promote conversational multiplicity into a coordination principle or performance claim.
Boundary: Pilot applications do not isolate when extra agents help, duplicate work, or amplify error. Next: what matched evaluation distinguishes coordination benefit from added calls and shared context?
Flags: builder-evaluated; no matched best baseline
Finding: Multi-agent systems helped parallelizable work but degraded sequential planning by 39% to 70%. Independent agents amplified errors more than centralized systems, and tool density increased coordination cost.
Impact: Strengthen task-decomposability and error-containment checks. Reject agent count as a quality proxy.
Boundary: Benchmark architectures and tasks do not prove the same effects in governed repository work. Next: can a bounded owning-system test predict when centralized delegation beats one agent?
Finding: Coordination topology and planning strategy changed milestone progress and task results. A graph topology performed best in one research scenario, while cognitive planning improved milestone achievement by 3%.
Impact: Watch milestone and topology measures. Reject transfer of one topology result into a default architecture.
Boundary: Simulated scenarios and model-specific rankings do not establish operational superiority. Next: which milestone signals remain meaningful in real handoffs with authority and privacy boundaries?
Flags: scenario and topology dependence; no mature-repository field test
Finding: Accuracy-only evaluation can reward complex, expensive agents, obscure the source of gains, and encourage benchmark overfitting. Downstream users and model developers also need different evaluation evidence.
Impact: Strengthen joint cost-and-outcome evaluation, holdouts, and downstream-fit reporting.
Boundary: This is a methodological critique, not a field test of one collaboration workflow. Next: what minimum holdout and replay design prevents Breadcrumbs patterns from overfitting their own examples?
Flags: preprint; benchmark and cost-definition dependence
Finding: Models that performed well on isolated tasks struggled when those tasks were interleaved. Long-term-memory augmentation sometimes matched or exceeded larger context windows.
Impact: Strengthen interleaving and recovery tests. Reject context-window size as continuity evidence.
Boundary: Synthetic conversations do not reproduce repository authority or real user stakes. Next: can fresh-session evaluation separate retrieval success from correct task ownership and stale-context rejection?
Finding: Full execution traces improved agent-level and step-level failure attribution over output-only evidence. The benchmark attributes failure to the missed recovery responsibility when an upstream mistake remained correctable.
Impact: Strengthen complete trace capture and recovery-aware responsibility. Reject final-output inspection as sufficient incident evidence.
Boundary: Attribution accuracy remains limited and varies by architecture and trace completeness. Next: which minimum trace fields support reliable correction without retaining unnecessary private content?
Flags: trace sampling and annotator dependence; benchmark systems
Finding: Tool-using agents failed many ordinary tasks and remained vulnerable to prompt injection from untrusted tool data. Security and task utility must be evaluated together.
Impact: Strengthen instruction-data separation and adversarial evaluation. Reject successful tool execution as evidence of safe delegation.
Boundary: The benchmark is simulated and cannot enumerate future attacks. Next: how should a governed workflow preserve provenance and authority when retrieved content contains instructions?
Flags: simulated content and tools; defense coverage changes quickly
Finding: Final success rate hides partial progress and failure location. AgentBoard adds a progress-rate measure and process views across multiple agent environments.
Impact: Strengthen stage and progress evidence. Reject completion status as the only workflow measure.
Boundary: Benchmark progress does not itself capture human authority, burden, or recovery. Next: which process milestones should be standardized without creating a misleading composite score?
Flags: benchmark and process-metric construct validity
Finding: Humans completed 72.4% of tasks while the best evaluated agents completed less than 12.2%, exposing major grounding and operational-knowledge gaps in realistic computer use.
Impact: Strengthen executable-environment validation and bounded authority. Reject fluent planning as evidence of reliable computer action.
Boundary: Models and systems have advanced since the study, and benchmark success is not deployment safety. Next: what current, owning-system test is required before a computer-use capability receives write authority?
Finding: Optimizing a model for the combined human-machine result can outperform optimizing the model alone when human error patterns are known.
Impact: Watch complementary-error allocation as a candidate evaluation. Reject stand-alone model accuracy as the only routing signal.
Boundary: The tasks are structured and depend on labeled or estimable human behavior. Next: how can complementary error regions be estimated without profiling people beyond the owning workflow's need?
Finding: Long-term memory requires information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention. Evaluated systems showed substantial degradation across sustained interactions.
Impact: Strengthen update, temporal, and abstention tests in the memory exam. Reject recall accuracy or context length as sufficient memory evidence.
Boundary: The histories and questions are benchmark constructs and do not test authority, provenance, or repository custody. Next: how should correction survival and source precedence be evaluated alongside these abilities?
Flags: generated histories; benchmark and model-judge dependence
Finding: Long-context models and retrieval improved some results but still lagged humans on long-range temporal and causal understanding, question answering, event summaries, and multimodal dialogue.
Impact: Strengthen causal, temporal, and multi-session evaluation. Reject a large context window as proof of durable memory.
Boundary: The conversations are generated and edited rather than operational records. Next: which failures persist when the source of truth is versioned repository evidence?
Flags: generated and human-edited dialogues; no repository custody
Finding: Memory evaluation changes materially by memory level, interaction role, and resource dimension. One recall measure cannot represent the capability.
Impact: Strengthen multi-dimensional memory evaluation and participant-versus-observer distinctions. Reject one memory score.
Boundary: The benchmark does not evaluate governance, cross-context isolation, correction authority, or repository ownership. Next: which governance dimensions can be added without obscuring benchmark performance?
Flags: benchmark construct choices; no governance evaluation
Finding: Explicit memory tiers and control flow can move information between limited active context and external storage.
Impact: Watch tiered context and explicit control-flow patterns. Reject storage movement as proof of truth, provenance, retention policy, or authority.
Boundary: The paper demonstrates an architecture and benchmark applications, not a governed memory system. Next: what evidence must accompany every move between context tiers?
Flags: builder-evaluated; architecture demo; no authority semantics
Finding: A long-term conversation system can retrieve, summarize, update, and selectively retain interaction-derived information.
Impact: Watch explicit update and retention evaluation. Reject an anthropomorphic forgetting curve as a sufficient retention or deletion policy.
Boundary: Companion-style interactions and simulated evaluation do not establish correction, consent, or governance. Next: which retention changes require an owning-system rule or human decision?
Flags: simulated users; anthropomorphic retention model
Finding: An observation stream, retrieval, reflection, and planning produced more believable simulated behavior than ablated variants.
Impact: Watch reflection-linked retrieval and planning as an implementation pattern. Reject believability as evidence of factual correctness or workflow benefit.
Boundary: The target was simulated social believability, not truth, authority, or professional task performance. Next: which reflection outputs can be checked against source evidence before reuse?
Flags: simulation; believability outcome; model-judge and rater dependence
Finding: Reusing verbal feedback from prior attempts improved performance on several tested decision, reasoning, and programming tasks.
Impact: Watch feedback-linked episodes. Strengthen the rule that self-reflection cannot independently verify a claim or widen authority.
Boundary: Benchmark feedback can be incomplete or wrong, and improvement does not establish durable correction. Next: when must feedback come from an independent test or human-owned source?
Finding: The same model's feedback-and-revision loop improved average task performance over one-step generation in the reported experiments.
Impact: Watch inexpensive iterative refinement. Reject same-model critique as independent validation or durable learning.
Boundary: The tasks emphasize bounded output quality, not external truth, correction survival, or workflow ownership. Next: where does same-model refinement stop helping or amplify a shared blind spot?
Flags: same-model critique; task and judge dependence
Finding: Selective memory extraction and consolidation can reduce context use and improve benchmark retrieval relative to several reported baselines.
Impact: Watch selective consolidation as an implementation candidate only through a reproducible owning-system comparison. Reject vendor-reported benchmark gains as governance or deployment validation.
Boundary: The paper is authored by the system's builders, uses benchmark and model-judge evaluation, and does not test repository custody, correction authority, or private-context isolation. Next: can an independent replay reproduce gains while preserving source evidence and deletion semantics?
Flags: builder-authored; benchmark and model-judge dependence
Finding: Optimizing a model as part of a model-plus-decision-maker system can improve system accuracy or fairness relative to isolated rejection rules.
Impact: Watch system-aware deferral as an evaluation concept. Reject synthetic gains as authority to automate routing.
Boundary: The experiments largely simulate decision makers. Next: can prospective human evidence reproduce benefits under changing expertise and costs?
Finding: A classifier and rejector can be learned jointly from expert decisions so the system predicts or defers where each is expected to be stronger.
Impact: Watch learned deferral regions. Reject past expert labels as permanent routing authority.
Boundary: Structured labeled tasks do not capture accountability, drift, or private side information. Next: how quickly must deferral evidence expire after expert or distribution change?
Finding: A prior consistent deferral formulation was not calibrated for expert correctness; the proposed one-vs-all method produced calibrated probabilities in tested tasks.
Impact: Strengthen calibration testing for deferral estimates. Reject consistency or accuracy alone as calibrated routing evidence.
Boundary: Classification benchmarks do not validate open-ended delegation. Next: what calibration target is meaningful when correctness is delayed or multidimensional?
Finding: Prediction sets can trade off misleading experts against task difficulty without requiring users to decide when to trust a single label.
Impact: Watch bounded option sets and measured coverage. Reject forced-choice sets where the valid action may be abstention or escalation.
Boundary: Multiclass prediction is narrower than open-ended work and experiments partly use simulated experts. Next: how should prediction-set logic expose missing-option risk?
Finding: Multiagent debate improved reported reasoning and factuality on selected benchmarks without model training.
Impact: Watch debate as a task-specific candidate. Reject preprint benchmark gains as a general multi-agent advantage or independent verification.
Boundary: Agents share model-family weaknesses and the work is not an owning-system field test. Next: when does debate outperform equal-cost sampling or one strong verifier?
Finding: Debate increased divergence relative to self-reflection on tested tasks, but gains required controlled interaction and model judges were not always fair across different agents.
Impact: Watch bounded disagreement and stopping rules. Reject the judge agent as neutral by default.
Boundary: Two benchmark families do not establish durable workflow value. Next: how should judge bias and debate cost be measured prospectively?
Finding: Inception prompting and role assignment can sustain autonomous agent conversations and enable study of cooperative behavior.
Impact: Watch role-scoped communication as implementation evidence. Reject sustained conversation as task success or governance.
Boundary: Generated tasks and conversational studies do not compare against the best single-agent baseline. Next: which role boundaries add unique information rather than duplicated language?
Flags: generated tasks; sustained conversation is not success
Finding: Encoding standard operating procedures into role-specific intermediate artifacts can structure multi-agent software generation.
Impact: Watch typed intermediate artifacts. Reject simulated software-company roles as proof of real team capability.
Boundary: Benchmarks and generated projects do not establish maintainability, security, or cost benefit in an owned repository. Next: which artifacts improve an equal-cost code task under independent tests?
Finding: Structured role conversations and communicative clarification can produce software artifacts across staged development tasks.
Impact: Watch explicit clarification and phase artifacts. Reject role play or generated code as evidence of production quality.
Boundary: Evaluation does not establish superiority on mature private repositories. Next: does communicative clarification reduce defects more than direct requirements repair?
Flags: generated software tasks; production transfer absent
Finding: Stage-based recruitment, decision, execution, and evaluation make agent-team composition configurable and expose emergent behaviors.
Impact: Watch stage-specific orchestration. Reject emergence as benefit, safety, or authority.
Boundary: Simulation and benchmark outcomes do not validate cross-repository professional work. Next: which stage can be removed without reducing outcome quality?
Flags: emergence and simulation outcomes; limited governance
Finding: Selecting agents and stopping communication dynamically improved reported reasoning and code benchmarks relative to fixed collaboration baselines.
Impact: Watch adaptive team size and early stopping. Reject agent-importance scores as human authority or universal expertise.
Boundary: Benchmark contribution scores may not transfer across tasks, versions, or repositories. Next: can dynamic selection beat one agent under equal cost and unseen tasks?
Flags: benchmark contribution scores; version sensitivity
Finding: Combining parametric generation with explicit non-parametric retrieval improved several knowledge tasks and made source updates and provenance more tractable than parametric memory alone.
Impact: Strengthen source-linked retrieval. Reject retrieval as proof that the generated claim follows from or faithfully represents the source.
Boundary: Wikipedia benchmarks do not cover repository authority, correction, or deletion. Next: what evidence binds each durable claim to retrieved source spans and version?
Flags: Wikipedia tasks; retrieval is not entailment
Finding: Models often used relevant information less reliably when it appeared in the middle of long contexts, even when the context fit the advertised window.
Impact: Strengthen position-robust context tests and retrieval. Reject context-window size as effective-memory evidence.
Boundary: Tested models and context layouts will age. Next: which current models remain position-sensitive on repository-derived evidence?
Flags: model and layout sensitivity; synthetic key-value task
Finding: Learned reflection tokens enabled on-demand retrieval and source critique, improving reported quality and factuality over several baselines.
Impact: Watch retrieval gating and evidence critique. Reject self-critique as independent source validation.
Boundary: Training and benchmarks do not test repository custody or correction authority. Next: what external check is required before critique changes durable memory?
Flags: learned critique is not independent validation
Finding: Recursive summaries can support retrieval across different abstraction levels and improve several long-document benchmarks.
Impact: Watch hierarchical retrieval for global and local questions. Reject generated summaries as source truth.
Boundary: Summaries can omit or distort details and the tests do not cover correction survival. Next: how can every abstraction retain auditable links to leaf evidence and invalidation events?
Flags: generated summaries may distort leaf evidence
Finding: A knowledge graph plus Personalized PageRank improved reported multi-hop retrieval quality and efficiency over selected RAG baselines.
Impact: Watch graph retrieval for cross-document association. Reject biological analogy or QA gains as governed-memory validation.
Boundary: Benchmark graphs do not establish provenance, source precedence, deletion, or access control. Next: how should graph edges inherit source authority and tombstones?
Flags: multi-hop QA; biological analogy; no tombstones
Finding: Graph-based community summaries improved reported comprehensiveness and diversity over conventional RAG for global questions over large corpora.
Impact: Watch global-versus-local retrieval modes. Reject model-generated graph structure or summaries as canonical repository truth.
Boundary: This is a builder-authored preprint with model-based evaluation. Next: can independent evaluators reproduce gains while tracing each synthesis claim to versioned sources?
Flags: builder-authored; model-generated graphs and judges
Finding: The abstract reports that task-specific hierarchies combining concurrent and prerequisite-dependent work outperformed four compared multi-agent approaches across the simulated missions.
Impact: Watch organizational structure as a task-specific coordination candidate for CI-016.
Boundary: Preprint, abstract-only screen, simulated environment, builder evaluation, and no real Jarvis or mature-repository test. Next: does the full study support an equal-cost comparison against one strong agent on unseen tasks? Candidate; not promoted or adopted
Flags: abstract-only; builder-evaluated; simulated wildfire missions; comparator and cost equivalence not fully appraised; no real Jarvis validation
Finding: The publisher record reports that decomposing answers into claims produced a useful uncertainty signal, while raw estimates still required post-hoc calibration.
Impact: Watch claim-level evidence and calibration as a candidate measurement approach for CI-003 and CI-017.
Boundary: One small benchmark, one answer and verifier model family, automated judging, and no human participants do not establish human trust, independent verification, or kit gains. Next: does the signal remain calibrated across models, datasets, evidence sources, and human review decisions? Candidate; not promoted or adopted
Flags: small single-benchmark evaluation; fixed calibration and held-out split; one model family; no human participants; closest calibrated baseline differences require caution; self-verification is not independent evidence
Finding: The public article reports that generated tools are tied to source code, checked against reference outputs, and locked after validation before conversational use.
Impact: Watch executable evidence packages as a bounded provenance and reproducibility pattern for CI-012 and CI-017.
Boundary: Case-study reproduction of computational papers does not establish human understanding, safe use on new data, or general agent reliability. Next: does a source-bound, test-locked tool improve reproducibility and review behavior against static documentation at comparable cost? Candidate; public article screened, framework claims not independently replicated; not adopted
Flags: builder evaluation; computational research domains; reference-output reproduction does not establish human understanding or safe transfer to new data
Finding: The screened preprint abstract describes prioritizing human screening with learned relevance.
Impact: Watch screening burden and human-owned relevance decisions under CI-004 and CI-012.
Boundary: Published identity checked; abstract screened from related preprint, not journal full text. Ranking a collected corpus does not discover new records or establish a safe stopping rule. Next: replay against a fully labeled, frozen corpus and measure misses as well as burden. Ingested screened candidate; no adoption
Flags: Simulation and stopping-rule transfer; journal full text inaccessible in this pass; R-313 is the same family
Finding: The primary abstract describes conditional error-handling stages and evaluation on TEACH. Embodied-task results do not establish software handoff reliability or net review savings.
Impact: Watch CI-006 failure attribution and recovery, with CI-008 revision limits.
Boundary: Abstract screened only; next: full-text comparator, failure taxonomy, recovery cost and holdout appraisal. Ingested screened candidate; backlog, no adoption
Flags: The primary abstract describes conditional error-handling stages and evaluation on TEACH. Embodied-task results do not establish software handoff reliability or net review savings.
Finding: The primary abstract describes typed failure-conditioned routing, shared intermediate state, and recovery transitions learned from unsuccessful traces. Benchmark claims remain author-reported; no independent replay or production transfer is established.
Impact: Watch CI-006 failure attribution and recovery, with CI-008 revision limits.
Boundary: Abstract screened only; next: full-text comparator, failure taxonomy, recovery cost and holdout appraisal. Ingested screened candidate; backlog, no adoption
Flags: The primary abstract describes typed failure-conditioned routing, shared intermediate state, and recovery transitions learned from unsuccessful traces. Benchmark claims remain author-reported; no independent replay or production transfer is established.
Finding: Trace annotation distinguishes specification, inter-agent alignment, and verification failures; the taxonomy can structure recovery diagnosis.
Impact: Watch CI-001, CI-002, CI-006.
Boundary: Purposive framework/task sample and model-assisted annotation; framework failure rates use different benchmarks and are not directly comparable. Not a universal production failure estimate. Next: Reconcile human labels and judge errors, then test taxonomy coverage on an unseen public-safe trace set. Added candidate with selected-full-text appraisal; not full-text validated, replicated or adopted
Flags: selected-full-text: Purposive framework/task sample and model-assisted annotation; framework failure rates use different benchmarks and are not directly comparable. Not a universal production failure estimate.
Finding: A clarification module treats message handoffs as intervention points for missing or corrupted information.
Impact: Watch CI-004, CI-006, CI-010.
Boundary: Builder benchmarks and learned gating do not establish real human clarification burden or repair of internal agent errors. Published cost claims differ from the earlier preprint; do not combine their effect sizes. Next: Appraise comparator budgets and clarification ablations, measuring unnecessary questions and missed handoff errors. Added candidate with selected-full-text appraisal; not full-text validated, replicated or adopted
Flags: selected-full-text: Builder benchmarks and learned gating do not establish real human clarification burden or repair of internal agent errors. Published cost claims differ from the earlier preprint; do not combine their effect sizes.
Finding: Interference-heavy contexts test retrieval of revised facts, historical states and aggregate answers across changing records.
Impact: Watch CI-008, CI-009.
Boundary: Generated questions and selected revision histories need independent label checks; benchmark correctness does not validate stored permissions or retention policy. Next: Separate current facts from historical truth and authority, and audit generated labels before a synthetic memory replay. Added candidate with selected-full-text appraisal; not full-text validated, replicated or adopted
Flags: selected-full-text: Generated questions and selected revision histories need independent label checks; benchmark correctness does not validate stored permissions or retention policy.
Boundary: Some tasks repackage existing datasets, including LongMemEval; these are not independent corroboration. Budget-limited system coverage and assumed context caching bound cost comparisons. Next: Preserve dataset overlap and distinguish measured memory updates from task learning and model knowledge. Added candidate with selected-full-text appraisal; not full-text validated, replicated or adopted
Flags: selected-full-text: Some tasks repackage existing datasets, including LongMemEval; these are not independent corroboration. Budget-limited system coverage and assumed context caching bound cost comparisons.
Finding: Interdependent sessions connect stored experience to later actions instead of testing recall alone.
Impact: Watch CI-002, CI-009.
Boundary: Human-crafted benchmark tasks are not longitudinal workplace evidence. Authors overlap MemoryAgentBench; shared research lineage and model judges limit independence. Next: Audit task dependency and success scoring, then compare a no-memory baseline with equal-budget memory variants. Added candidate with selected-full-text appraisal; not full-text validated, replicated or adopted
Flags: selected-full-text: Human-crafted benchmark tasks are not longitudinal workplace evidence. Authors overlap MemoryAgentBench; shared research lineage and model judges limit independence.
Finding: Synthetic tools and task-specific grading distinguish refusal from completion of harmful multi-step requests.
Impact: Watch CI-005, CI-007, CI-017.
Boundary: English prompts, no multi-turn attacker follow-up, custom-tool dependence and grading-path misses bound transfer; refusal is not evidence of safe owner-authorized behavior. Next: Keep harmfulness, capability and permission compliance separate and use only harmless synthetic fixtures for local evaluation. Added candidate with selected-full-text appraisal; not full-text validated, replicated or adopted
Flags: selected-full-text: English prompts, no multi-turn attacker follow-up, custom-tool dependence and grading-path misses bound transfer; refusal is not evidence of safe owner-authorized behavior.
Finding: Simulated users, domain policies and final database state test outcome correctness and repeated-run consistency.
Impact: Watch CI-002, CI-007, CI-017.
Boundary: Model-simulated users are not human participants; goal-state matching does not capture every valid interaction. Current tau2 repository contains newer grading and domains, not this paper's exact evaluated release. Next: Pin task and grader revisions and compare repeated-run outcomes without mixing incompatible benchmark versions. Added candidate with selected-full-text appraisal; not full-text validated, replicated or adopted
Flags: selected-full-text: Model-simulated users are not human participants; goal-state matching does not capture every valid interaction. Current tau2 repository contains newer grading and domains, not this paper's exact evaluated release.
Finding: Long generated dialogues probe contradiction, updates, abstention and temporal reasoning rather than simple recall alone.
Impact: Watch CI-008, CI-009.
Boundary: Synthetic narrative generation and its judge setup are not real long-term user behavior. The authors' memory method and benchmark form one evidence family. Next: Audit construction and grading for contamination, then test revised-state and abstention cases with independent labels. Added candidate with selected-full-text appraisal; not full-text validated, replicated or adopted
Flags: selected-full-text: Synthetic narrative generation and its judge setup are not real long-term user behavior. The authors' memory method and benchmark form one evidence family.
Finding: Human-expert comparisons across time budgets show why task duration and resource allocation must be part of an agent baseline.
Impact: Watch CI-001, CI-004, CI-005.
Boundary: Small, selected engineering environments, noisy rare successes, test feedback and costly GPU access limit generalization. Reported token/labor comparisons exclude GPU cost in the cited figure. Next: Compare matched time and total cost, retain failed runs and avoid treating the average expert as a universal comparator. Added candidate with selected-full-text appraisal; not full-text validated, replicated or adopted
Flags: selected-full-text: Small, selected engineering environments, noisy rare successes, test feedback and costly GPU access limit generalization. Reported token/labor comparisons exclude GPU cost in the cited figure.
Finding: The authors describe revoking obsolete execution authority while handing reusable completed state to a successor.
Impact: Watch CI-006, CI-007, CI-008.
Boundary: Abstract-only: correctness assumptions, comparators, resource costs and implementation coverage remain unappraised. This serving-layer result does not establish human review authority or general agent safety. Next: inspect revision boundaries and failed or adversarial transitions in the full methods. Candidate ingested at abstract depth; not a promoted finding, replicated result or adopted practice
Flags: abstract-only: full methods, correctness envelope, comparator fairness and total resource burden are unreviewed. Serving publication authority is narrower than human authorization.
Finding: Higher confidence in GenAI was associated with less reported critical-thinking enactment and effort, while confidence in one's own ability was associated with more. Critical work shifted toward verification, integration, and stewardship.
Impact: Strengthen visible verification and human-owned synthesis. Watch cognitive-effort displacement.
Boundary: Self-reported associations do not establish causal skill loss. Next: which observable review behaviors preserve critical engagement without adding performative friction?
Finding: Medical AI can augment clinicians, but full complementarity was uncommon. Concurrent review of cases and AI output outperformed sequential review, and junior clinicians benefited more than senior clinicians.
Impact: Watch interaction order and expertise as contextual variables. Reject a universal handoff sequence.
Boundary: Safety-critical clinical evidence cannot prescribe a general operating protocol. Next: when should a low-consequence workflow compare independent and concurrent review, while retaining an auditable human decision?
Finding: The abstract reports a preference for short explanations made from related arguments, with participant explanations often shorter than the formal definitions produced.
Impact: Watch concise explanation plus separately retained justification as a hypothesis for CI-012.
Boundary: An explanation-choice experiment does not show that shorter explanations improve decisions, trust calibration, or kit effectiveness. Next: does a concise visible explanation with inspectable supporting evidence improve a named review behavior without hiding material limits? Candidate; not promoted or adopted
Flags: publisher-page screen only; constrained argument-selection task; preference is not decision quality, calibrated reliance, or kit effectiveness
Finding: The abstract reports overall narrowing and sustained novelty associated with some users' distinctive, revised input.
Impact: Watch human-input diversity under existing CI-015 portfolio-diversity research debt.
Boundary: Association does not establish causation. The social-agent setting and selected respondents do not establish a benefit in other workflows. Next: can a bounded comparison separate input diversity, output diversity, quality, and human effort? Candidate; source identity verified, claims not assessed; not adopted
Flags: abstract-only; association, not causation; social-agent setting and selected respondents limit transfer; no repository benefit shown
Finding: Failures reduced trust more than successes increased it, and adjustments also reflected outcome bias and contrast with the participant's own result.
Impact: Strengthen event-sequence reliance measurement. Reject one post-task trust snapshot as a sufficient account of collaboration.
Boundary: The task was bounded image recognition and trust was self-reported after each trial. Next: which behavioral measures capture dynamic reliance without requiring intrusive continuous surveys?
Finding: Specialization, credibility, and coordination are separable dimensions of a transactive memory system and relate to team performance.
Impact: Watch the three dimensions as evaluation prompts. Reject importing the human-team scale as an AI score.
Boundary: The scale was validated for human teams, not agents or repositories. Next: which trace evidence can operationalize each dimension without anthropomorphism?
Flags: human-team self-report scale; direct transfer rejected
Finding: Psychological safety was associated with learning behavior, which mediated team performance in the studied setting.
Impact: Watch safe correction and escalation as human-workflow conditions. Reject interface affordances as proof that people feel safe to challenge outputs.
Boundary: One organizational setting does not establish causality across AI-assisted work. Next: how can an evaluation detect suppressed disagreement without collecting sensitive free text?
Flags: observational team evidence; organizational confounding
Finding: Higher automation reduced situation awareness and failure-recovery performance, while active participation moderated the loss.
Impact: Strengthen active-review and reentry tests. Reject routine success as evidence of takeover readiness.
Boundary: The controlled navigation task predates generative systems. Next: which active participation preserves readiness without erasing useful automation gains?
Flags: laboratory control task; historical interface
Finding: Explanations can include who used the system, what they decided, and why, not only model internals, making organizational context visible.
Impact: Watch social transparency as a trace-design candidate. Reject social precedent as authority or truth.
Boundary: Interview and speculative evidence does not validate the proposed design. Next: which social context aids accountability without exposing private content or normalizing past errors?
Flags: small qualitative sample; proposed design not deployed
Finding: Human and machine strengths can complement each other, but value depends on redesigning work around collaboration rather than adding AI to an unchanged process.
Impact: Strengthen complementarity and workflow redesign. No new principle.
Boundary: This is organizational evidence, not proof of durable cross-session protocols. Next: what makes a redesigned workflow persist after participants or models change?
Flags: non-systematic selection; reported company evidence
Finding: The literature converges on redefined human-machine relationships, interaction design, and organizational-societal implications. It still lacks longitudinal evidence, unified evaluation frameworks, and sufficient theoretical grounding.
Impact: Strengthen the collaborative frame. Watch evaluation and longitudinal continuity as research debt, not claimed contributions.
Boundary: A literature gap does not prove this repository fills it. Next: what evaluation scheme can compare collaboration quality without collapsing unlike contexts into one score?
Flags: review quality not fully appraised; broad construct variation
Finding: Trust affects reliance when full understanding is impractical, but the design goal is appropriate reliance calibrated to context and automation performance, not maximum trust.
Impact: Strengthen calibrated reliance and evidence display. Reject confidence or user acceptance as sufficient evidence of safe collaboration.
Boundary: The review predates current generative systems. Next: which observable behaviors show appropriate reliance when outputs and system capability change between runs?
Finding: Overreliance, underuse, and inappropriate automation imposed by designers or managers are distinct failure modes rather than one trust problem.
Impact: Strengthen separate reliance, disuse, and designer-authority checks. Reject adoption rate as evidence of appropriate collaboration.
Boundary: The framework predates modern generative systems and supplies categories rather than a deployment test. Next: which observable workflow events distinguish justified nonuse from harmful disuse?
Finding: Trust is shaped by dispositional, situational, and learned factors, so a single end-state trust measure hides materially different causes.
Impact: Strengthen layered trust evidence. Reject trust as a unitary score or direct proxy for appropriate reliance.
Boundary: The review is broad but largely predates current generative assistants. Next: which layers predict actual reliance under changing model and task conditions?
Flags: evidence ends in 2013; trust constructs vary
Finding: Trustworthy joint activity depends on responding to changing situations and recovering from mismatches, not only on initial calibration or reliability.
Impact: Strengthen responsivity and recovery as separate protocol outcomes. Reject static trust calibration as sufficient continuity evidence.
Boundary: This is a conceptual synthesis, not a controlled evaluation of a generative workflow. Next: which traces show that a participant noticed, communicated, and recovered from a changed situation?
Finding: Automation bias appeared across settings, while its frequency and mitigation depended on task, user, workload, accountability, training, and interface design.
Impact: Strengthen verification and accountability design. Reject alerts, confidence displays, or explanations as universal mitigations.
Boundary: Much of the evidence concerns clinical or legacy decision support and reporting was heterogeneous. Next: which mitigations transfer to open-ended generative work without excessive burden?
Finding: Participants coordinate by establishing enough mutual understanding for the current purpose, with grounding costs shaped by the communication medium.
Impact: Strengthen explicit grounding checks at handoff boundaries. Reject transcript volume as proof of common ground.
Boundary: Human communication theory does not validate an AI protocol. Next: which observable acknowledgement and repair signals show sufficient grounding without adding excessive process?
Flags: human communication; medium and task transfer
Finding: Team knowledge includes both shared and distributed structures, so individual knowledge aggregation can miss coordination-relevant organization.
Impact: Strengthen separate tests for common ground and distributed expertise. Reject one pooled knowledge measure.
Boundary: Measurement methods are task-sensitive and human-team focused. Next: which representation preserves specialization while still testing shared constraints?
Finding: Automation can leave people responsible for rare, difficult failures while reducing the practice and information needed to intervene effectively.
Impact: Strengthen recovery-readiness and skill-retention tests. Reject nominal takeover authority without practiced capability.
Boundary: Industrial process control differs from generative knowledge work. Next: what low-cost rehearsal shows that a human can actually recover a current workflow?
Finding: Flexible automation modes create new monitoring demands and failures when operators cannot determine the active mode or its implications.
Impact: Strengthen visible mode, authority, and routing state. Reject hidden transitions among agent or approval modes.
Boundary: Aviation evidence does not prescribe a software interface. Next: which state changes require persistent display and which can remain in trace evidence?
Flags: supervisory-control domain; no current agent trial
Finding: Reliable autonomy can still reduce situation awareness and make exception handling harder, so reliability and oversight readiness are distinct.
Impact: Strengthen readiness and exception-recovery measures. Reject reliability alone as an autonomy-widening criterion.
Boundary: The synthesis emphasizes safety-critical domains. Next: what proportional evidence is enough for low-consequence knowledge work?
Flags: broad review; current-agent transfer untested
Finding: Higher automation improved routine performance and workload but reduced situation awareness and failure performance, especially across the analysis-to-action-selection boundary.
Impact: Strengthen function-specific benefit and recovery measurement. Reject one global automation level.
Boundary: The studies vary and mostly predate LLM systems. Next: do the same tradeoffs appear when generative agents move from analysis to consequential action?
Finding: Robot performance and attributes were stronger trust predictors than environmental factors in the reviewed evidence.
Impact: Strengthen performance-history evidence. Reject trust as appropriate reliance and anthropomorphic presentation as competence evidence.
Boundary: Human-robot interaction differs from language-agent work and trust is not calibration. Next: which performance-history display improves decisions rather than only confidence?
Finding: Eighteen guidelines span initial use, regular interaction, failure, and change over time, emphasizing expectation setting, correction, feedback, and adaptation.
Impact: Watch the guidelines as an interaction coverage checklist. Reject guideline presence as outcome validation.
Boundary: Relevance ratings do not prove implementation benefit. Next: which guidelines materially improve the specific workflow under matched evaluation?
Finding: Human explanations are selective, contrastive, and social, so technically complete accounts may be poor explanations for a user's actual question.
Finding: Complementarity is not always achievable, and lower average loss can conflict with fair distribution of benefit or increase group error disparities.
Impact: Strengthen impossibility checks and subgroup outcomes. Reject average complementarity as sufficient benefit.
Boundary: The model abstracts human behavior and task structure. Next: which empirical conditions reveal that a workflow cannot beat its better component?
Flags: modeled losses and groups; no open-ended workflow
Finding: AI contributes analytical processing under complexity while humans retain holistic and intuitive strengths under uncertainty and equivocality.
Impact: Strengthen human judgment under uncertainty. Add decision context to future reviews. No new principle.
Boundary: Conceptual framing does not measure a deployed collaboration protocol. Next: how should a system recognize when uncertainty calls for human judgment rather than more model effort?
Finding: Useful teammates must support coordination, common ground, mutual predictability, and direction rather than behave as isolated tools.
Impact: Strengthen the protocol layer. Test whether handoffs expose enough status, intent, limits, and correction state for another participant to coordinate safely.
Boundary: A challenge framework is not a validation of this repository's protocol. Next: which challenges can be turned into observable conformance checks across session and model changes?
Flags: classical automation transfer; no protocol trial
Finding: Automation can operate at information acquisition, analysis, decision selection, or action implementation, and each function can receive a different level of automation.
Impact: Strengthen function-specific authority allocation. Reject one global autonomy level for a workflow or agent.
Boundary: This is a pre-generative-AI model, not an empirical validation of a current system. Next: which function-level authority choices improve performance without weakening accountability or recovery?
Flags: pre-LLM taxonomy; no direct authority trial
Finding: A group can coordinate knowledge through awareness of who knows what rather than by duplicating all information in every member.
Impact: Strengthen source ownership and knowledge-location routing. Reject central copying as the default continuity strategy.
Boundary: Human group-memory theory does not establish repository behavior. Next: how should source-location knowledge remain current when owners, tools, or artifacts change?
Finding: Teamwork depends on several interacting behaviors and coordinating mechanisms, including monitoring, backup, adaptability, shared mental models, communication, and trust.
Impact: Watch the taxonomy as a coverage check for protocols. Reject direct transfer as a validated agent-team design.
Boundary: This is a human-team synthesis with propositions rather than an AI intervention test. Next: which elements add predictive value beyond existing handoff, recovery, and correction measures?
Finding: Cognitive work is distributed across people, artifacts, and environments, so the whole functional system is the proper unit of analysis.
Impact: Strengthen workflow-level evaluation and artifact traceability. Reject model-only performance as system evidence.
Boundary: The framework is analytical, not a measured implementation result. Next: what minimum system boundary captures the people, repositories, tools, and external constraints that affect an outcome?
Flags: framework-level transfer; no intervention test
Finding: Human-centered XAI should begin with user needs, test pitfalls, and evaluate experiences rather than selecting an explanation algorithm first.
Impact: Strengthen goal-specific explanation evaluation. Reject explainability as a context-free property.
Boundary: The paper is a survey chapter, not a deployment result. Next: which user goal and consequence should each explanation support?
Flags: preprint; selective coverage; no direct outcome test
Finding: Trust involves user vulnerability and expectations about an implicit or explicit contract; trust can be warranted or unwarranted depending on its cause.
Impact: Strengthen explicit capability contracts and warranted-reliance evidence. Reject trust maximization as a goal.
Boundary: The formalization does not supply a complete measurement instrument. Next: which observable contract violations should immediately narrow reliance?
Flags: formalization without complete measurement instrument
Finding: High human control and high automation are not mutually exclusive, and function-specific designs can avoid a single autonomy continuum.
Impact: Strengthen function-specific control and automation. Reject one human-versus-machine slider.
Boundary: The framework is conceptual rather than comparative evaluation. Next: which functions can raise automation while preserving meaningful control?
Flags: framework-level claims; no direct validation
Evidence Note: A local personal-assistant Gateway connecting models, tools, and messaging channels (WhatsApp, Telegram, Slack, Discord) that also supports shared team deployments under a named 'trusted gateway, untrusted execution' authority model.
Evidence Note: Seven sequential skills (brainstorm, worktree, plan, execute, TDD, review, complete) structure subagent-driven development with mandatory human checkpoints at design validation, plan approval, severity-blocking code review, and branch-completion decisions, plus a two-stage subagent review (spec compliance then code quality) and a referenced drill eval harness from superpowers-evals.
Evidence Note: Agent-harness framework defining 68 named subagent roles, a staged plan/test/implement/verify/remember workflow with resumable sessions, an explicit approve/request-changes gate on plans (Plan Canvas), and a persistent Memory Vault; README states an explicit without-ECC vs with-ECC comparison table but names no independent third-party verifier.
Evidence Note: A skill collection with human-gated requirement-alignment sessions before implementation, a two-axis (coding-standards vs spec-compliance) code review run as parallel sub-agents, a shared CONTEXT.md domain vocabulary, and explicit user veto authority over architecture changes.
Evidence Note: README exposes visual agent construction, scheduled and triggered runs, run status, integrations, and cost visibility while separating hosted and self-hosted surfaces.
Evidence Note: README exposes a visual AI workflow, RAG pipeline, agents, model management, testing, and observability integrations while separating hosted service claims.
Evidence Note: Library of 230+ role-specific agent prompt files organized into 19 divisions, with dedicated 'Reality Checker' and 'Evidence Collector' roles gating release on documented evidence, and conversion scripts that target Claude Code, GitHub Copilot, Cursor, and other tools from one source format.
Evidence Note: README exposes agent construction and deep agents with planning, subagents, and file systems, while routing durable orchestration and hosted evaluation to companion products.
Evidence Note: README identifies Claude Code and exposes plugins with custom commands and agents, while deferring most runtime behavior to external documentation.
Evidence Note: Six-phase spec-driven workflow (constitution through implement/converge) integrating 30+ coding agents via slash commands, with a documented brownfield 'evolving specs' path and self-upgrade mechanics for material change; no durable task-state persistence, formal evaluation harness, or independent verifier is documented.
Evidence Note: 23-skill Claude Code toolkit spanning plan/build/review/ship stages with required human approval gates at each stage, checkpoint-based session recovery, a stale-state evidence tracker, and an independent cross-vendor review step via OpenAI's Codex CLI.
Evidence Note: README identifies a local coding agent and separates it from the cloud agent; repository-level mechanisms are only partially described at README depth.
Evidence Note: A plugin enforcing a 7-step minimalism decision ladder on AI coding agents across 20+ platforms, reporting an average 54% code-generation reduction and benchmarked 20% cost reduction / 27% faster execution versus baseline agent performance.
Evidence Note: Local tree-sitter AST parsing turns a codebase into a queryable knowledge graph invoked via `/graphify` inside Claude Code, Cursor, or Gemini CLI, registers as an MCP server, and tags edges EXTRACTED versus INFERRED for transparency.
Evidence Note: README encodes specialized analyst, researcher, trader, and risk roles in a graph-based deliberation workflow for a bounded decision domain.
Evidence Note: A five-hook (SessionStart/UserPromptSubmit/PostToolUse/Stop/SessionEnd) memory-compression system for Claude Code using a local worker, SQLite, and Chroma vector search, with a three-layer progressive-disclosure retrieval path claimed to cut token use roughly 10x versus immediate full retrieval.
Evidence Note: 25 structured skills spanning Define/Plan/Build/Verify/Review/Ship, with a human approval gate before autonomous `/build auto` execution, four specialist personas that explicitly do not invoke each other, and an optional cross-model escalation path for high-stakes review.
Evidence Note: README exposes reference MCP servers and SDK links for controlled access to tools and data, while separating reference from community implementations.
Evidence Note: README exposes a self-hosted control center for multiple coding-agent backends, scheduled or event-driven automations, service integrations, and Agent Client Protocol compatibility.
Evidence Note: An orchestration platform giving AI agents an org chart (titles, reporting lines, permissions, budgets) alongside humans, with persistent per-agent task state across heartbeats, an immutable tool-call audit log, revisioned/rollback-capable config, and scoped budget policies with hard stops across agent providers.
Evidence Note: Spec-driven development framework with explicit explore/propose/apply/archive stages, durable per-change folders under openspec/changes/, and a continue/fast-forward resume command; the README names Spec Kit and Kiro as explicit comparison baselines and 'no spec layer' as the implicit default.
Evidence Note: Design-quality tool for AI coding agents with 61 deterministic (no-LLM, no-API-key) detector rules plus LLM critique checks, a fixed command pipeline (shape/craft/audit/polish), and hook-based enforcement across 14+ agent hosts; the Codex integration requires an explicit human approval step to enable the hook.
Evidence Note: README marks AutoGen as maintenance mode, points new users to Agent Framework, and retains agents that act autonomously or alongside humans.
Evidence Note: README separates role-based Crews from event-driven Flows; some observability and governance claims belong to the commercial control plane.
Evidence Note: Orchestrator that fans one prompt across multiple coding-agent runs in isolated git worktrees for the user to compare and merge the winner, with cross-account usage and rate-limit tracking across 30+ CLI agent platforms.
Evidence Note: Seven-stage resumable video-production pipeline (research through compose) with explicit human approval gates at each stage and budget caps that pause for confirmation above a configurable threshold (default $0.50), plus a per-decision audit log with a replay feature; no rigorous baseline comparison against a simpler process is documented.
Evidence Note: README exposes data connectors, indices, graphs, retrieval and query interfaces, agent workflows, and a boundary between the open framework and hosted document-agent products.
Evidence Note: MCP server giving coding agents Chrome DevTools/browser control; documents that 'multiple agents or subagents sharing a server instance' route tool calls to a specific pageId, and gates remote-debugging access behind a click-to-allow permission prompt.
Evidence Note: Distributed tasks, stateful actors, immutable objects and debugging links expose worker topology. Cluster execution does not prove multi-agent reasoning benefit; compute and coordination costs need measurement. Pinned README screen only; no runtime test or adoption.
Evidence Note: Agent platform documenting subagent delegation ('isolated child agent runs with their own context windows'), human-in-the-loop tool approval that pauses a run for input and resumes it, per-agent memory isolation, and native MCP tool support.
Evidence Note: README treats language-model systems as modular programs and exposes algorithms for optimizing prompts and weights across RAG pipelines and agent loops.
Evidence Note: A frontend SDK giving agents and UI a synchronized shared-state layer, with agents able to pause execution to request user input, confirmation, or edits before continuing, plus interaction history captured across sessions and an AG-UI protocol used to integrate LangChain, CrewAI, Mastra, and PydanticAI backends.
Evidence Note: README exposes an MCP interface to browser automation and explicitly distinguishes general MCP clients from coding-agent CLI and skills use.
Evidence Note: README frames a research data pipeline for graph-structured context, states maintenance mode, separates demonstration from supported offering, and warns that indexing can be expensive.
Evidence Note: README identifies one coding agent across CLI, VS Code, and JetBrains surfaces but provides limited authority, recovery, and evaluation detail at README depth.
Evidence Note: Filesystem-style context database (viking:// protocol) unifying agent memory, RAG, and skills behind deterministic ls/tree/find-style browsing instead of opaque vector search, with tiered on-demand loading reported to cut token spend and to raise memory accuracy from a 24-57% baseline to 80-83% across three agent integrations.
Evidence Note: README exposes a zero-code multi-agent platform, software-role collaboration, incremental development, and a linked evolving-orchestration research implementation.
Evidence Note: README exposes a self-hosted code-assistance platform, repository context, IDE integrations, and an agent preview rather than a fully documented collaborative runtime.
Evidence Note: Self-hosted AI chat platform where users build custom agents with actions/MCP connections; enterprise tier adds role-based access control over agent/action access and query auditing, plus custom code execution to filter PII from queries.
Evidence Note: A local code-intelligence graph exposing 30 MCP query tools, benchmarked across 6 real repos against naive-corpus, LSP, RAG, and grep baselines (median ~65x token reduction, 0.69 F1 vs ground truth), with a --verify mode that cross-checks its own token estimates against OpenAI's cl100k_base tokenizer.
Evidence Note: State machines, statecharts, actors and model-based testing utilities expose mode transitions. This is application state modeling, not a durable execution service or agent permission system. Pinned README screen only; no runtime test or adoption.
Evidence Note: README exposes configurable human labeling across data types, projects, model-assisted labeling, review workflows, and exportable datasets.
Evidence Note: Packs a codebase into a single AI-consumable file and exposes it via an MCP server (pack_codebase, grep_repomix_output); a --sandbox flag confines MCP file tools to one workspace directory for untrusted clients, and Secretlint scanning plus a --token-budget CI gate run before output is shared.
Evidence Note: Persistent-memory MCP server for coding agents implementing 4-tier memory consolidation (working/episodic/semantic/procedural) and hybrid BM25+vector+graph retrieval, reporting 95.2% R@5 on the LongMemEval-S benchmark and roughly 92% token reduction versus full-context loading; humans retain sole deletion authority over stored memories.
Evidence Note: Converts technical-book PDFs into on-demand, per-chapter Claude Code / Copilot CLI / Amp skill files conforming to the open Agent Skills standard, claiming 24x-51x fewer tokens than dumping the book into context; runs locally with no cloud upload and includes a skill-validation tool.
Evidence Note: A Dolt-backed dependency-aware issue graph for coding agents, replacing markdown TODO lists with atomic claim-based task discovery, hash-based IDs to prevent multi-agent/multi-branch merge collisions, and a schema-version guard that blocks stale binaries from querying migrated databases.
Evidence Note: A widely-cited set of 12 design principles for building reliable LLM-agent software, covering context-window ownership, pause/resume execution state, and contacting humans via tool calls as a first-class step; presented as pattern documentation with example code rather than a runnable framework.
Evidence Note: README exposes local evaluations, model and prompt comparison matrices, red teaming, vulnerability reports, and CI review for AI applications.
Evidence Note: Python flows/tasks, scheduling, event-triggered deployments and manual UI/CLI runs expose workflow state. Managed cloud and self-hosted server have distinct data and operation boundaries. Pinned README screen only; no runtime test or adoption.
Evidence Note: README exposes an end-to-end multimodal ingestion, parsing, hybrid retrieval, and query workflow across text, tables, equations, and images.
Evidence Note: Deterministic workflow engine for AI coding with defined phases, validation gates, and an explicit approve node (interactive:true) that pauses execution for human input; workflow runs and node sessions persist to a database, but no formal evaluation harness or independent verifier is documented.
Evidence Note: A vision-LLM browser-automation agent offering multi-step UI workflow orchestration, a livestreaming mode for real-time human intervention, and a published 64.4% accuracy on the WebBench benchmark with per-task-type breakdowns.
Evidence Note: README exposes a reproducible social simulation, agent histories, environment state, and replay for the Generative Agents research artifact.
Evidence Note: An agent memory system claiming SOTA on the LongMemEval benchmark with results independently reproduced by Virginia Tech researchers and The Washington Post, built on four parallel retrieval strategies (semantic, BM25, graph, temporal) plus a PII/secret-scanning write policy.
Evidence Note: Training architecture, trace aggregation and released reinforcement-learning examples are linked. Builder benchmark gains are unreplicated; compute, reward hacking and stored training traces need appraisal. Pinned README screen only; no runtime test or adoption.
Evidence Note: Task checkpoints, retries, replay/cancel actions and human approval waitpoints expose paused execution. A waitpoint is not independent permission verification or exactly-once effect evidence. Pinned README screen only; no runtime test or adoption.
Evidence Note: Declarative data-asset dependencies, lineage, observability and testability are visible. Asset freshness does not establish source appraisal quality or human-agent complementarity. Pinned README screen only; no runtime test or adoption.
Evidence Note: README exposes a secure MCP server and runtime framework for structured queries, semantic search, and database tools, plus an explicit repository rename history.
Evidence Note: README exposes long-task planning, selective context, multi-model execution, context caching, and versioned plan branches for comparison and recovery.
Evidence Note: A sidecar memory service that distills agent session histories into readable Markdown skills shared across Codex, Claude Code, Cursor, OpenClaw and other hosts via one configured backend; the memory service makes no LLM calls itself, leaving judgment inside the connected agent.
Evidence Note: README exposes semantic search, RAG, agents, multi-model workflows, workflow composition, and web plus MCP APIs across several languages.
Evidence Note: AI PR-review bot (describe/review/improve/ask commands) supporting multiple LLM providers via LiteLLM and multiple git platforms (GitHub, GitLab, Bitbucket, Azure DevOps, Gitea); each tool call is framed as a single ~30-second low-cost LLM call, but the README does not document approval gates, cross-PR memory, or independent verification.
Evidence Note: README exposes executable flows, dataset evaluation, quality and performance checks, CI integration, and an explicit cloud collaboration boundary.
Evidence Note: A shared context-graph layer (AgentContext API, native Agno integration) gives every agent on a team one queryable graph of entities and decisions rather than separate per-agent memory silos, with every fact linked to its source via W3C PROV-O provenance and audit trails exportable to JSON/CSV/RDF for regulator submission.
Evidence Note: README exposes detection, anonymization, redaction, and de-identification components for sensitive data across text, images, and structured data.
Evidence Note: MCP server giving Claude terminal/filesystem control with session management for long-running commands, comprehensive audit logging of every tool call, and third-party verification badges (AgentAudit Verified, Archestra Trust Score, Smithery, MseeP) in place of self-reported trust claims.
Evidence Note: Local-first, on-device multi-agent framework with eight built-in agents, an orchestrator for automatic tool selection, a cross-agent skills catalog following the agentskills.io standard, and a learning loop over local trace data that treats energy/FLOPs/latency/dollar cost as first-class evaluation constraints.
Evidence Note: Repository owns the versioned MCP specification and schemas for context, tools, and resources rather than a multi-agent outcome runtime.
Evidence Note: README exposes a modular vulnerability scanner with probes, detectors, generators, and reproducible attack suites across model interfaces.
Evidence Note: Canonical GitHub Action wrapping Claude Code for PR/issue automation across multiple model backends (Anthropic direct, Bedrock, Vertex, Foundry); README documents a v1.0 migration guide and references an 'agent-approval-check' directory but does not detail approval semantics, memory, or evaluation facilities.
Evidence Note: Pinned README describes durable tasks, Postgres-backed execution history, retries, pause/resume and worker routing. These expose recovery mechanisms, not safe supersession or agent collaboration benefit. Open issue 4900 reports old retry work advancing after replacement on an earlier release; current-head behavior was not tested. Requires a server and data-retention review before any operational use.
Evidence Note: README exposes input and output guards, risk detection and mitigation, structured output, and a published benchmark comparing guardrail performance and latency.
Evidence Note: README exposes programmable controls between application code and an LLM, including output constraints and predefined dialog paths, with a linked technical evaluation.
Evidence Note: README exposes a minimal software-engineering agent, migration history, benchmark harnesses, and a deliberately small implementation surface.
Evidence Note: Deterministic application-layer policy gates intercept every agent tool call (require_approval routes to named approver groups, four-tier privilege rings, a kill switch), backed by tamper-evident merkle audit logs, 992 conformance tests, 7 fuzz targets, a red-team CLI, and an OWASP Agentic Top 10 verify command, with framework adapters for Semantic Kernel, LangGraph, CrewAI, and AutoGen.
Evidence Note: Container-environment evaluation spans multiple agents and Terminal-Bench tasks. Isolation is not a security certification; the Terminal-Bench predecessor is excluded as superseded lineage. Pinned README screen only; no runtime test or adoption.
Evidence Note: Containerized issue-to-patch evaluation and engineer-checked task subsets expose outcome tests. Public-task contamination and environment versions limit inference; a passing patch is not deployment evidence. Pinned README screen only; no runtime test or adoption.
Evidence Note: Durable functions, step retries, event waits and historical replay are explicit. Server license differs from SDK licenses; stored events require retention controls. Pinned README screen only; no runtime test or adoption.
Evidence Note: Rust hook that intercepts shell commands from AI coding agents (Claude Code, Copilot, Codex CLI, etc.) and blocks destructive operations by default, with configurable trust-level allowlists, escape hatches, and fail-closed handling of unparseable input.
Evidence Note: Replay graphs, tracing decorators, integrations and spend tracking are named. Dashboards do not include all human review costs or verify independent completion; no service was activated. Pinned README screen only; no runtime test or adoption.
Evidence Note: Provider-neutral, stateful control plane layered on top of Codex, Claude Code, Cursor, or dsh that tracks objectives, gates, todos, evidence, quota, and handoffs across restarts, explicitly framed around 'governance, recovery, and human-agent collaboration' while preserving human decision authority.
Evidence Note: Modular office-automation skills (PPT generation, data analysis) following the Agent Skills SKILL.md convention, built for agent runtimes like OpenClaw and hermes-agent; output quality is checked by the same VLM performing generation via multi-round review, not a distinct verifier, and no memory, cost, or baseline mechanism is documented.
Evidence Note: Cross-CLI memory tool with SessionStart/PreCompact/PostCompaction/SessionEnd lifecycle hooks, per-file checkpoint/restore, TTL-expiring memory pages, and a require_approval gate for auto-improvement proposals; ships an MCP server plus HTTP API for cross-agent handoff between Claude Code, Codex, and other harnesses.
Evidence Note: Compiles Markdown-plus-YAML agentic workflow definitions into standard GitHub Actions with a safe-outputs validation buffer, read-only sandboxed default execution, and an explicit requirement that authors review permissions, tools, network access, and generated files before deployment.
Evidence Note: Multi-language code RAG combining a knowledge graph, tree-sitter AST parsing, and an MCP server for querying/editing monorepos; README states a diff-preview-before-commit pattern and a dynamic call-tracing overlay, references an 'agentic QA benchmark harness' without detailing its methodology, and documents three separate versioning tracks with expedited security releases.
Evidence Note: An MCP server exposing 95+ Home Assistant control tools with named per-tool approval policies and a read-only mode toggle, plus an explicit breaking-change note tied to a version number that tells clients to refresh their tool list.
Evidence Note: OpenTelemetry wrapping and instrumentation are distinguished from the closed UI/backend. The MIT SDK does not license the platform; HTTP/database/LLM export needs a privacy boundary. Pinned README screen only; no runtime test or adoption.
Evidence Note: MCP-native knowledge graph stored as local Markdown files with semantic retrieval, letting a session 'pick up right where you left off' across Claude, Codex, Cursor, ChatGPT, and Obsidian; the cloud tier adds snapshots and point-in-time restore.
Evidence Note: Multi-agent CLI orchestrator with a coordinator that launches parallel specialist agents (frontend/backend/tester/lead) atop GitHub Copilot, persisting per-agent history and an orchestration log across sessions; human retains approval authority over merges via a 4-tier escalation model.
Evidence Note: Trace/metric signal status and a common instrumentation interface are visible. Telemetry is not independent verification; exporters may persist or transmit payloads and need minimum-data controls. Pinned README screen only; no runtime test or adoption.
Evidence Note: Claude Code skills scaffold a data portal end to end (architecture advisory through generated Next.js UI) against a pluggable backend abstraction (CKAN, GitHub, Frictionless); README does not document approval gates, memory, evaluation, or a baseline comparison.
Evidence Note: Seven-stage agentic workflow (classify through report) with hard failure bounds and an approval gate before irreversible actions, plus a formalized fable-judge role that re-runs claimed checks by diffing and executing rather than reading reports; README cites 15 eval rounds across 260+ agent runs comparing bare models to method-guided models (e.g. Haiku catching 5/5 planted frauds with the method vs 3-4/5 without).
Evidence Note: Simulated agent-user-tool domains and a grading-version incompatibility are documented. The pinned README includes newer tau3 extensions; this is not the original paper release or a real-human study. Pinned README screen only; no runtime test or adoption.
Evidence Note: Self-hosted personal-assistant OS built on a temporal knowledge graph shared across voice, chat, and messaging interfaces; tasks are drafted as plans requiring explicit user approval before execution, and the README cites an 88.24% average accuracy on the LoCoMo benchmark plus CASA Tier 2 certification.
Evidence Note: Local voice assistant with persistent cross-session memory (knowledge-graph organization plus embedding-based semantic search), automatic redaction of emails/tokens/passwords before storage, and a linked EVALS.md tracking accuracy over time; connects to external tools via MCP.
Evidence Note: README exposes a game engine for generative agents and game masters with memory, reasoning, social scenarios, and service-evaluation use cases.
Evidence Note: Postgres checkpoints, durable queues and step resumption provide recovery primitives. Persistent payloads and database operations add retention and maintenance obligations. Pinned README screen only; no runtime test or adoption.
Evidence Note: Browser automation infra that captures a reusable 'sitemap' of learned site actions across agent runs; publishes BU Bench V1 numbers claiming highest accuracy and lowest per-task cost against comparison methods, plus up to 90% token-spend reduction.
Evidence Note: Recursive multi-agent framework generating repository-level documentation via hierarchical decomposition and dynamic task delegation; ships CodeWikiBench, a first-party benchmark for doc-quality assessment across 21 repositories, and supports incremental re-docs via --compare-to <commit-hash>.
Evidence Note: Persistent Markdown project wiki grounded in a deterministic code graph with task-aware context loading; README cites a 24-session internal pilot against a files-only-search baseline (-56.6% estimated cost, 95.7% required source spans returned) but states the benchmark is internal and descriptive, not independently verified.
Evidence Note: Reusable web-agent environments span named benchmarks, with a research-only caution. Benchmark-specific setup/data terms remain separate; environment access is not real-account authority. Pinned README screen only; no runtime test or adoption.
Evidence Note: README exposes official graph and hybrid retrieval, graph traversal, knowledge-graph construction pipelines, and explicit experimental feature boundaries.
Evidence Note: Deterministic workflows, activities, replay and sandboxing expose resumable state. Requires a Temporal service and replay discipline; external effects still need idempotency. Pinned README screen only; no runtime test or adoption.
Evidence Note: README exposes trace trees, side-by-side evaluations, production and experiment organization, and a narrowed focus on tracing and evals.
Evidence Note: MCP server combining symbol-level code navigation, a persistent SQLite/vector memory engine, and bash-output compaction, benchmarked against a stated 'plain Claude' baseline (78.3% accuracy) over 96 coding tasks; the README discloses a since-withdrawn re-measurement after discovering the MCP tools had never actually been invoked.
Evidence Note: README describes human labels, active-learning record prioritization, saved decisions, duplicate hiding, and simulation on labeled datasets. No stopping-rule validity or scouting benefit is established.
Evidence Note: Builds a semantic graph of a codebase and pre-loads relevant context into AI-assistant prompts instead of letting the agent explore via tool calls; the README reports a before/after benchmark of cost ($0.49 to $0.27 per prompt), average turns (11.7 to 3.5), and a 0-100 quality score (76.6 to 86.6) across 12+ supported coding-agent platforms.
Evidence Note: A fully-local RESTful memory abstraction layer for AI agents, distinguishing long-term (persistent collections) from short-term memory, with configurable chunk overlap for context continuity and, on the Postgres backend, hybrid BM25 plus semantic vector search with independently weighted components.
Evidence Note: Claude skill enforcing staged execution: a written stage plan, delegation of independent sub-parts to model-pinned subagents (Opus/Sonnet/Haiku variants), a failable per-stage verification check, and a final self-review; the self-review is same-model critique rather than a distinct verifier, and no human approval or memory mechanism is documented.
Evidence Note: Claude Skill guiding a human 'PM' and an AI 'engineer' through problem discovery, evidence-based validation, and phased build plans with explicit checkpoints where the agent stops to explain what was built before continuing; ships a low-cost 'riskiest assumption' validation test as Build Phase 0.
Evidence Note: Incremental memory competencies, shared-history questions and model-judge setup are explicit. Reused LongMemEval data and shared authorship with MemoryArena limit independence; API cost is unmeasured here. Pinned README screen only; no runtime test or adoption.
Evidence Note: A zero-dependency, git-backed micro-lesson library letting AI agents BM25-search 310+ verified debugging lessons over an MCP server, with an intake path for agents to submit new incident reports that a maintainer reviews before conversion into a lesson.
Evidence Note: A CLI for YAML-defined multi-agent workflows with condition-based routing (explicitly 'no LLM in the orchestration loop'), human-in-the-loop pause points surfaced in a dashboard, per-agent token/cost tracking, and registry-tag versioning of workflows across multiple agent providers (Copilot, Claude, OpenAI, Hermes).
Evidence Note: README exposes gridworld environments for large populations of interacting reinforcement-learning agents and cites the artificial collective-intelligence platform paper.
Evidence Note: CLI plus a machine-readable rule pack (41 patterns) and a Claude plugin that scores prose 0-100 for AI-generated language and gates via a git pre-commit hook or CI; scoring is rule-based rather than the generating model grading itself.
Evidence Note: README exposes a preview drop-in client for configured guardrails while placing most policy details and setup in a hosted configuration surface.
Evidence Note: A Canvas LMS MCP server with 80+ role-scoped tools used by 40+ agent clients, routing bulk actions through a size-tiered decision tree (single-item tool call vs a bulk-grade tool vs sandboxed code execution), with preview-before-send confirmation on multi-recipient operations and optional FERPA-conscious anonymization.
Evidence Note: Claude Code toolkit of 65 skills, 7 autonomous subagents, and 18 slash commands with scoped read/write tool permissions, a persistent-memory MCP server (memory_map) for cross-session history, and pre/post-tool hooks that block unsafe commands.
Evidence Note: A skills collection whose orchestrator skill probes a runtime's concurrency/isolation/messaging capabilities before assigning work by role, and explicitly stops instead of doing the work itself when native subagent delegation is unavailable; paired with a profiler that emits a per-agent Perfetto trace with token/cost counters.
Evidence Note: A company-wide workspace/OS layer for running AI agents, with sealed workspace-scoped credentials agents fetch at use-time and real revocation, plus per-member identity separation so commits and token spend are attributed to the human, not the workspace.
Evidence Note: Engineering tasks link to expert comparison and a common task contract. Anti-contamination requests and costly execution prerequisites require care; no protected solutions were extracted or evaluated. Pinned README screen only; no runtime test or adoption.
Evidence Note: Generated long dialogues probe abstention, contradiction, updates and temporal reasoning. Benchmark and LIGHT method share a family; synthetic dialogues and model judges need independent labels. Pinned README screen only; no runtime test or adoption.
Evidence Note: A Claude Code skill enforcing a spec-to-plan-to-TDD-to-verify workflow; rule changes to the skill itself ship only through normal PR-plus-human-approval, and an evals/ directory turns real changelog misses into checkable regression scenarios.
Evidence Note: 21-skill agent framework treating LLM context as bounded process memory backed by an Obsidian wiki, with finite-budget loop/graph engineering, a verify-before-claim proof gate checked by the same agent, and a supervised multi-agent team-ownership mode.
Evidence Note: Claude Code plugin (repo since renamed to Chisle) that compresses AI coding-agent output and tool-output context, reporting a benchmarked 52% of baseline token bill on coding tasks and generating per-agent rule files for several other coding tools.
Evidence Note: Deterministic step replay, configurable retries and restart from a selected step are explicit. Convex dependence and external-action idempotency constrain transfer. Pinned README screen only; no runtime test or adoption.
Evidence Note: README presents distributed durable async/await and version compatibility constraints for Python services. It does not establish independently witnessed handoff completion, agent reasoning quality, or exactly-once external effects.
Evidence Note: A two-tier architecture splits a Haiku 'Interpreter' (translates a user's map edits into instructions) from a Sonnet/Opus 'Builder' (locates files and writes code), with code changes occurring only when the user explicitly presses Build or sends instructions; maps are described as snapshots that go stale and require a re-scan after significant codebase changes.
Evidence Note: Offline, regex/NER-based PII redaction library with a Claude Code hook that gates agent tool calls (shell, web requests, file writes, MCP tools) and a LiteLLM guardrail for gateway-level redaction; no test/eval framework or cost discussion documented beyond latency benchmarks.
Evidence Note: Local SQLite-backed FIFO queue that serializes expensive operations across agents to prevent concurrent resource thrashing; the README's own worked example reports 3:45 queued versus 10+ minutes of thrashing for two agents sharing one build queue.
Evidence Note: Plan-first Claude Code companion running a seven-stage headless pipeline (Architect, Approval Gate, Builder, Medic, Reviewer, Security, Pilot); live execution is double-gated (plan must be approved AND ADEPTLY_LIVE=1 set), and every run writes an append-only audit.jsonl plus a token-usage ledger with estimate-vs-actual cost.
Evidence Note: A machine-only social platform for AI agents (posting/voting/commenting via JSON API and MCP), with identity issued by secret key, explicit exclusion of humans from governance decisions, and merkle-proofed checkpoints witnessed by GitHub Actions for tamper-evident public records.
Evidence Note: CPU-only cross-repository memory system for coding agents using tree-sitter AST parsing and local embeddings; re-verifies retrieval results against the live filesystem at query time (STRONG/WEAK/REBUILT/STALE trust verdicts) and ships a 166-test suite covering concurrency and idempotency invariants.
Evidence Note: An end-to-end encrypted peer channel letting two coding agents on different machines split work and hand off tasks, gated by a human-approved 'door-knock' join step, with local jsonl/markdown transcripts replayable after a restart or /clear.
Evidence Note: A code-grounded comparative field guide reviewing memory-system implementations against pinned Git commits, with a repeatable research/integration workflow, portable acceptance tests, and a security-screening script (screen_repo.py) run before reading or executing a reviewed system's code.
Evidence Note: Deterministic context-payload linter that scores LLM inputs and can block calls entirely in LangChain 'block' mode; ships a decoupled 1,500-sample benchmark (ContextBench) and exports/analyzes traces from LangSmith, Helicone, and Datadog.
Evidence Note: A memory system that runs five specialized consolidation agents (decay analysis, clustering, semantic merge, promotion, relationship discovery) over short-term memory that decays on an Ebbinghaus-curve formula, with dry-run preview before changes apply and 13 MCP tools for cross-agent access. Top commit dated Feb 17, 2026, over six months stale as of this review.
Evidence Note: Typed memory store for AI agents with a SHA-256 hash-chain audit trail (an audit/verify endpoint recomputes the chain to detect tampering), an admin console that gates release of quarantined or contradictory memories, and a reported 91.4% score on the LongMemEval benchmark.
Evidence Note: MCP server extracting reusable Claude Code skills from YouTube/articles/PDFs, with a public verify_artifacts endpoint returning source URL, quality scores, and a sha256 bundle hash, an estimate_cost pre-execution check, and tiered pricing (free/Pro/pay-per-call); provenance lookup is self-reported by the generating service, not a distinct verifying party.
Evidence Note: Skills exposing 300+ third-party generative models via an OpenAI-compatible endpoint and an MCP server; README states per-model pricing and a live model catalog, but documents no collaboration topology, memory, or evaluation mechanism.
Evidence Note: Nine markdown skill files instructing coding agents how to use Basic Memory's MCP tools for reflection, defragmentation, and task/schema lifecycle management; documents no human approval gates or evaluation framework, framed around agent autonomy rather than human-in-the-loop control.
Evidence Note: An agent is a folder of state (agent.md, modules/, archive/) served over MCP by a local HTTP server so multiple runtimes (Claude Code, Claude Desktop, Codex, Cursor) share one persistent state; secrets are injected at runtime and scrubbed from output so the agent uses keys but never sees their values.
Evidence Note: README describes citation metadata input and duplicate/unique outputs, and links its comparative evaluation. This is deterministic bibliographic preprocessing, not an agent runtime or proven repository-fork deduplicator.
Evidence Note: A shared-memory server lets multiple coding-agent instances send session events into a three-tier memory model (ephemeral, working, canonical), with a human 'trusted dreamer' reviewing a curation queue before anything is promoted to canon; the whitepaper reports a stress test of 5,000+ runs across a dozen models where unscoped agents broke builds consistently and provenance-scoped retrieval brought failures to zero.
Evidence Note: Audits CLAUDE.md/AGENTS.md files via static checks plus a behavioral backtest that replays rules against local session transcripts to measure per-rule compliance, and drafts a candidate memory file from session history when none exists. All hook/enforcement proposals require explicit user adoption; nothing auto-installs.
Evidence Note: 10 installable agent skills (e.g. 'route-cheap-escalate-hard', 'free-model-triage') that route work by task complexity to cheaper models or tiers, each shipping a runnable proof executed by CI on every push and weekly rather than a self-graded claim.
Evidence Note: A single MCP memory server shared across Claude Code, Codex, and any MCP client, with a knowledge-graph store agents write notes into for handoffs, plus an advisory file-claim layer that auto-expires after 2 hours to recover from crashed agents.
Evidence Note: Claude Code plugin that intercepts prompts via the UserPromptSubmit hook and blocks submission when local regex scanning detects secrets or PII; debug logs record only match metadata and explicitly never the matched secret/PII values.
Evidence Note: A portable plain-Markdown context layer with graded Load Levels (0-4) so any AI tool loads only the relevant context bundle, sessions end by asking whether anything durable was learned, and agent-proposed patches to context files require explicit user approval before saving; sensitive detail is isolated in git-ignored *_PRIVATE.md overlays.
Evidence Note: Cross-session memory tool combining a git-synced markdown vault with TF-IDF/hybrid retrieval, lifecycle hooks (SessionStart/PreCompact/SessionEnd), and a promoter with explicit 'explain' (audit-only) vs 'apply' modes; reports LongMemEval R@5 of 97.2-98.8% against the paper's official flat-BM25 baseline.
Evidence Note: Cryptographic protocol (HMAC-SHA256, Ed25519) separating a credential-free AI agent from a signing collector process (O-Node); commands require a signed proposal-approval-outcome chain with reuse and expiry checks, stored in a tamper-evident SQLite log (2,024+ artifacts across 81 sessions per README), alongside formal ProVerif protocol verification and 101 test cases.
Evidence Note: A durable, project-local, append-only hash-linked event log (.agent-mesh/events.jsonl) records requests, decisions, and cross-instance handoffs, with humans stated as the sole approval authority for durable decisions and a public contract-pack test suite covering the request/response flow and tamper detection.
Evidence Note: 43 Claude Code engineering-guardrail skills distilled from production defects across four repos, including a push-gate confirmation hook, a per-commit WORKLOG checkpoint log, hook-behavior test suites, and a 'commission-the-roster' step requiring role/budget/scope approval before spawning a multi-agent fleet.
Evidence Note: A cross-machine agent messaging skill/protocol where a human-only 'host' identity is reserved and agents get named roles; message history and reply-threading are retrievable via a stateless, replay-safe API, with an open Agent Skills / MCP interoperability layer across Claude Code, Cursor, and Codex.
Evidence Note: Claude Code hook system running five background agents (heartbeat, nightly memory consolidation, AFK task worker, audit, curation) across active/AFK/maintenance modes with a three-layer memory architecture; README states autonomous code writing is queue-based only and documents no human approval gate.
Evidence Note: A starter pack of Claude Agent skills extracted from a production app, built around a two-agent develop/review handoff (Claude Code plus a secondary reviewer like Codex before merge), a PR-review polling script, and a spend-analysis skill, motivated by a documented 26-round review-loop case study.
Evidence Note: A self-hosted MCP-native memory server exposing six multi-agent coordination primitives (claim/release/who_has/claims/handoff/pulse_others) to prevent concurrent-edit conflicts, plus append-only checkpoint/reflection memory with soft-archive deletion; explicitly alpha, with stated breaking changes expected before 1.0.
Evidence Note: Local-first hybrid (dense + BM25) semantic search engine storing documents in a single SQLite file with no cloud dependency, plus a cross-session 'journal' agent-memory feature and an MCP server exposing 16 tools; README cites a benchmark against ColBERTv2 on 5 BEIR datasets, methodology self-conducted rather than independently verified.
Evidence Note: Per-user agent memory plugin that structurally quarantines third-party claims (received email, external documents) as a distinct edge type from user-asserted facts to resist prompt-injection/confabulation, validated by 44 offline tests plus an opt-in live acceptance eval benchmarked against a published research paper (arXiv:2607.21962).
Evidence Note: Orchestrator that decomposes goals into tickets, spawns multiple coding agents (different LLMs per ticket) in isolated git worktrees, and requires a PM-side review checklist plus explicit user confirmation before merge; tracks per-vendor/per-model spend and keeps an append-only audit-timeline ledger per ticket.
Evidence Note: TUI planning tool that turns a plan into a versioned, reviewable artifact (word-level diffs across revisions) with an explicit Plan -> Review -> Revise -> Execute -> Resume flow and a named hand-off between a planning agent (Claude Code) and a building agent (Codex); no memory, evaluation, cost, or independent-verification mechanism is documented.
Evidence Note: Audits and reduces persistent Claude Code/Codex context (measured roughly 35K to 13K tokens) against a stated fresh-session baseline, applying only human-approved changes item by item; no durable workflow state or independent verifier is documented.
Evidence Note: Git-inspired, content-addressed provenance store for agent beliefs: atomic micro-commits, a reflog of all pointer movements, blame/bisect over reasoning history, and a golden-vector interop test suite validating byte-for-byte parity between the Python and Rust implementations.
Evidence Note: Launcher that dispatches an isolated, context-stripped Claude Code session (restricted tool set, a 'strict mode' that disables tool use) to answer one self-contained prompt, logging session events to a per-session JSONL hook log a human can inspect or attach to.
Evidence Note: A local semantic-memory MCP server that scores facts on a stated importance/confidence/recency/frequency formula, fades (never overwrites) contradicted facts via 'soft supersession,' and self-reports its own limitations in the README, including that its automatic correction path 'has no lock.'
Evidence Note: A transparent LLM proxy that pauses risk-graded tool calls (email, file deletion) for human approval via Slack or webhooks, reserves budget with a hold/settle pattern before requests reach providers, and halts stuck-loop/retry-exhaustion failures.