update readme and add some tools (#36)

Signed-off-by: SamYuan1990 <yy19902439@126.com>

Author: SamYuan1990

README.md

@@ -1,73 +1,74 @@
-# Structured Context Language
+# Structured Context Language (SCL)
 
-## Overview
-People are familiar with SQL (Structured Query Language), which is used to interact with databases. Today, as we face Large Language Models (LLMs), the focus is shifting from prompt engineering to context engineering.
+## Vision
 
-In this repository, we aim to build a Structured Context Language (SCL) to occupy a niche analogous to SQL, drawing inspiration from context engineering practices.
+Everyone is familiar with SQL for interacting with databases. In the era of large language models, our focus is shifting from prompt engineering to context engineering.
 
-We hope that through this effort, we can distill a middleware solution. This middleware would provide a standard interface for AI agents, much like Hibernate serves as a standard ORM interface for Java applications.
+In this project, we aim to build a Structured Context Language (SCL), drawing on the practices of context engineering to occupy a niche similar to that of SQL.
+
+Through this practice, we hope to distill a middleware layer. This middleware will provide a standardized interface for agents, playing a role analogous to Hibernate for Java applications.
 
 ## Deconstructing SCL
 
-If we consider prompts as a query language for Large Language Models (LLM), then context engineering is undoubtedly an implementation of this query language. We can deconstruct context engineering along three independent dimensions:
-
-- Business Content: Specific instructions for particular prompts and scenarios.
-- Tool Invocation: Various tools the LLM can use to obtain additional external data.
-- Memory Management: In multi-turn conversation scenarios, determining which historical content is relevant to the current query.
-
-> We can view tool invocation as a spatial expansion of information and memory management as an expansion of information along the temporal dimension.
-
-Considering that in engineering practice, we can implement interactions for memory management through tool invocation, the extended querying of information within context engineering can therefore be accomplished using a standardized interface and further summarized into a standardized workflow.
-
-Inspired by the progressive loading mechanism of Claude Skill, we have also observed that autonomous selection of tools by the LLM can be achieved through progressive loading across different tools. Unlike stored procedures in SQL, which are defined and explicitly called for execution, progressive loading provides an additional layer of autonomy.
-
-## Use case
-> The Autonomy Slider —— Reference Karpathy's speech on Software 3.0. Show me the diff in vivid.
-
-```
-Configurable + Autonomy by LLM via feedback control
-Autonomy by LLM via feedback control(metric or history)
-Autonomy by LLM
-Configurable
-HardCode
-```
-
-- [ ] Should we make a middleware just input as prompt and output as result?(Autonomy)
-- [ ] We provides workflow and let people able to config it.(Configurable)
-- [ ] We provides sdk let people implements their own.(Hardcode)
-
-- [x] Obversbility —— otel.
-> 
-```
-docker run -p 8000:8000 -p 4317:4317 -p 4318:4318 ghcr.io/ctrlspice/otel-desktop-viewer:latest-amd64
-export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"
-export OTEL_TRACES_EXPORTER="otlp"
-export OTEL_EXPORTER_OTLP_PROTOCOL="grpc"
-```
-
-- [ ] Function selction.
-   - [x] "Progressive loading" base on RAG. (Autonomy)
-   - [ ] Hard code memory tool invoke. (Autonomy or defualt? tbd)
-   - [x] Hardcode control by human, as index hint for SQL.
-
-- [ ] File format Autonomy, took PDF format as example.
-    - [ ] Context auto into markdown.(Autonomy)
-    - [ ] Context auto embedding for RAG.(Autonomy)
-    - [ ] Or Hardcode control by human outside our process.
-
-- [ ] Content Autonomy.
-    - [ ] RAG support by default.(Autonomy)
-    - [ ] Hard code as input prompt content.(Hardcode control by human)
-```
-for EMBEDDING service, using siliconflow fow now as poc
-export EMBEDDING_API_KEY=<your_siliconflow_api_key>
-```
-
-```
-docker run -d --name pgvector -e POSTGRES_PASSWORD=postgres -e POSTGRES_USER=postgres -e POSTGRES_DB=postgres -p 5432:5432 ankane/pgvector:v0.5.1
-``
-
-## todo
-Article/Blog
-Investigating how to reuse powermom?
-Find some agent bench mark for testing.——Mind2Web or WebArena？
+If we treat a prompt as a query language for large language models (LLMs), then context engineering is certainly one implementation of that query language. We can deconstruct context engineering along three independent dimensions:
+
+- **Business content**: Concrete instructions tailored to specific prompts and scenarios.
+- **Tool calling**: Various tools available to the LLM to fetch additional external data.
+- **Memory management**: In multi-turn conversations, deciding which historical content is relevant to the current query.
+
+> We can view tool calling as a spatial expansion of information, and memory management as a temporal expansion of information.
+
+In engineering practice, we can manage memory through tool calls. Therefore, within context engineering, the expansion of information can be accomplished via a standardized interface, which can be further distilled into a standardized workflow.
+
+Inspired by the progressive loading mechanism of Claude Skills, we also see that between different tools, progressive loading can allow the model to autonomously select tools. Unlike stored procedures that are explicitly defined and called in SQL, this provides extra autonomy through progressive loading.
+
+## SCL Agent Loop: A Unified Agent Runtime
+
+Building on the above ideas, SCL provides a standardized agent loop as the runtime middleware for context engineering. It unifies the processing of these three dimensions into an event-driven execution model.
+
+### Design Principles
+
+- **Minimalist YOLO Mode**  
+  No built-in TODO lists, planning, sub-agents, or background bash processes. Developers externalize state through files, compose tools through bash, and implement skill execution by spawning new tasks. We want the framework to do one thing——run an agent——and give the user full control and observability.
+
+- **Unified Provider Interface**  
+  A single API supporting Anthropic, OpenAI, Google, xAI, Groq, Cerebras, OpenRouter, and any OpenAI-compatible endpoint. Provides streaming, tool calls with TypeBox schema validation, reasoning/thinking support, seamless cross-provider context handoff, and token and cost tracking.
+
+- **Tool Registration and Selection**  
+  Built-in tool registry that maintains tool metadata and descriptions. The agent uses a RAG-based mechanism to progressively load available tools, injecting only the relevant tool definitions into the context when needed. This avoids context bloat from full tool descriptions and preserves model autonomy while respecting progressive loading.
+
+- **Pluggable Content Compression**  
+  A hot-swappable interface for content compression strategies. During long conversations, a customizable compressor can distill historical messages to reduce token consumption while retaining critical information, improving stability for long-running agents.
+
+- **Prompt Templates**  
+  Structured template support for business content, facilitating reuse, version management, and team collaboration. Templates can incorporate proven prompt patterns from context engineering practice.
+
+- **Multiple Runtime Forms**  
+  - **RESTful (containerized)** — Deployed as a service, providing API access.  
+  - **Local TUI** — Interactive terminal usage with slash commands and session management.  
+  - **Library** — Directly imported for secondary development and deep integration.
+
+- **Observability**  
+  Transparent event streaming across the entire workflow: tool call parameters and results, every incremental model output, and internal state changes are all traceable. This is essential for debugging, evaluation, and building trust.
+
+- **Built-in Toolset**  
+  Out-of-the-box tools covering common coding and operations tasks:
+  - File read/write
+  - Search (grep, find)
+  - Bash
+  - Git
+  - Create cron jobs
+  - Other tools available for function call branching
+
+  Tools can be extended through the registration mechanism, supporting both native implementations and CLI-wrapped commands.
+
+### How It Reflects SCL’s Philosophy
+
+- **Unified Information Expansion**  
+  Both tool calling (spatial expansion) and memory management (temporal expansion) are handled within the agent loop through the same standardized tool interface. Memory management is no longer a special internal mechanism; it is invoked, recorded, and observed like any other tool.
+
+- **Progressive Loading Engineered**  
+  The RAG-based tool selection extends the progressive loading concept from “skill files” to all tools. The model first understands the need, then fetches tool descriptions on demand, and autonomously decides which resources to use. This balances context efficiency with autonomy.
+
+- **Middleware Positioning**  
+  Just as Hibernate provides a standard abstraction for persistence in Java applications, the SCL Agent Loop aims to provide a standardized interface for context management in agent applications. It encapsulates provider differences, tool execution, compression strategies, and runtime modes, allowing developers to focus on business prompts and task-flow design.

README_CN.md

@@ -20,3 +20,54 @@
 考虑到在工程实践中，我们可以通过工具调用来实现记忆管理的交互，因此在上下文工程中，对于信息的扩展查询可以使用一种标准化接口来完成，并进一步总结为一个标准化流程。
 
 受益于 Claude Skill 的渐进式加载机制，我们也看到在不同工具之间，可以通过渐进式加载的方式实现大模型对工具的自主选择。这与在 SQL 中定义并显式调用执行的存储过程不同，它通过渐进式加载提供了额外的自主性。
+
+## SCL Agent Loop：统一智能体运行时
+
+基于上述思想，SCL 提供一个标准化的智能体循环（Agent Loop），作为上下文工程的运行时中间件。它将上述三个维度的处理统一到一个事件驱动的执行模型中。
+
+### 设计原则
+
+- **极简 YOLO 模式**  
+  不内置 TODO、计划、子智能体、后台 bash。开发者通过文件外化状态、通过 bash 组合工具、通过新建 task 实现技能执行。我们希望框架只做“运行智能体”这一件事，并给使用者完全的控制权和可观测性。
+
+- **统一供应商接口**  
+  一个 API 支持 Anthropic、OpenAI、Google、xAI、Groq、Cerebras、OpenRouter 及任何兼容 OpenAI 的端点。提供流式传输、基于 TypeBox 模式校验的工具调用、思维/推理支持、无缝跨供应商上下文交接，以及令牌和成本追踪。
+
+- **工具注册与选择**  
+  内置工具注册中心，维护工具的元数据与描述。智能体通过 RAG 机制渐进式加载可用工具，只在需要时将相关工具定义注入上下文，避免全量工具描述带来的上下文膨胀。这种按需选择的方式继承了渐进式加载的思想，同时保留模型的自主性。
+
+- **可插拔的内容压缩**  
+  提供内容压缩策略的热插拔接口。在长对话中，通过可定制的压缩器精简历史消息，在保留关键信息的前提下降低令牌消耗，提升智能体长期运行时的稳定性。
+
+- **Prompt 模板**  
+  为业务内容提供结构化模板支持，便于复用、版本管理和团队协作。模板可结合上下文工程实践，固化有效的提示模式。
+
+- **多种运行形态**  
+  - **RESTful（容器）** —— 作为服务部署，提供 API 接入。  
+  - **本地 TUI** —— 在终端中交互式使用，支持斜杠命令、会话管理。  
+  - **库文件** —— 直接引用，支持二次开发与深度集成。
+
+- **可观测性**  
+  全流程事件流透明输出：工具调用的参数与结果、模型输出的每一步变化、内部状态的变更均可追踪。这对调试、评估和信任构建至关重要。
+
+- **内置工具集**  
+  开箱即用的基础工具，覆盖常见编码与运维任务：
+  - 文件读写
+  - 查找（grep、find）
+  - bash
+  - git
+  - 创建 cron job
+  - 以及其他可被 function call 分支选用的工具
+
+  工具可以通过注册机制扩展，既支持内置实现，也支持通过 CLI 包装已有命令。
+
+### 如何呼应 SCL 的思想
+
+- **信息扩展的统一**  
+  工具调用（空间扩展）和记忆管理（时间扩展）都在 agent loop 内通过标准化工具接口完成。记忆管理不再是一种特殊的内部机制，而是像其他工具一样被调用、记录和观察。
+
+- **渐进式加载的工程化**  
+  工具选择的 RAG 机制将渐进式加载从“技能文件”延伸到所有工具，模型先理解需求，再按需获取工具说明，自主决定使用哪些资源。这平衡了上下文效率与自主性。
+
+- **中间件定位**  
+  正如 Hibernate 为 Java 应用提供持久化的标准抽象，SCL Agent Loop 旨在为智能体应用提供上下文管理的标准化接口。它封装了供应商差异、工具执行、压缩策略和运行时模式，使开发者可以更专注于业务提示和任务流设计。

scl/__init__.py

@@ -0,0 +1,15 @@
+# Package initialization for scl
+
+# This package contains the Structured Context Language core modules
+
+__version__ = "0.1.0"
+
+# Import main modules for convenience
+from . import meta
+from . import processor
+from . import queue
+from . import storage
+from . import listener
+from . import otel
+from . import embeddings
+from . import capabilities