@timeless-hayoka3
Julien MaloneFounder▲5
Founder, PHI//DRIFT
59m ago · Testing and Evals · Virginia Beach

I'm trying to prove my ai architecture wrong before claiming it works. Here's the experiment

I'm building INFJ-Bot, an experimental cognitive AI system powered by DRIFT.

The central hypothesis:

Can structured memory and cognitive control produce measurable improvements in AI decision-making compared with simpler context-management methods?

We're investigating this through controlled A/B evaluations.

One recent measurement exposed a major problem:

Only 4 of 54 model responses followed the required structured output format.

That meant formatting failures were obscuring the actual decisions.

Rather than treating those results as evidence of cognitive improvement, we're correcting the evaluation protocol.

The next experiment compares three conditions:

A — Full replay

The model receives the available interaction history in chronological order.

B — Filtered replay

The model receives only relevant feedback for the correct simulated partner, ordered newest first.

C — Structured organizer

The model receives context selected and organized by the cognitive memory mechanism.

All conditions use matched histories, model settings, token limits, and structured A/B outputs.

The task bank includes:

  • Changing preferences over time.
  • Contradictory feedback.
  • Context-specific exceptions.
  • Irrelevant historical information.
  • Restart-persistence challenges.
  • Decisions where the newest memory alone is insufficient.

Our acceptance criteria:

The organizer must demonstrate either:

  • At least a 10-percentage-point reliable accuracy improvement over filtered replay.
  • Comparable accuracy with at least 25% fewer prompt tokens.

If it fails both criteria, we favor the simpler baseline.

We're preserving raw outputs and distinguishing completed experiments from unverified results.

I'd like the evaluation community to challenge this design.

What confounders are we missing?

How would you measure reliable accuracy gains?

What would convince you that structured memory provides a genuine advantage rather than simply repackaging context?

GitHub: github.com/timeless-hayoka/infj-bot

I'm looking for rigorous criticism, reproducible methods, and people interested in testing the system.

#LLMEvals #AIResearch #AgentEvaluation #Reproducibility #MachineLearning

Member ReposAI Builders ToolkitNewsResearch

Comments

No comments yet
Start the conversation.
Log in to join the thread.