Expose db change batches over control
This commit is contained in:
parent
8ffe1d714c
commit
a44eef3126
8 changed files with 130 additions and 15 deletions
|
|
@ -120,9 +120,9 @@ Roadmap items should be actionable and checkable:
|
||||||
`cas cleanup` evicts unpinned blobs while retaining pinned blobs.
|
`cas cleanup` evicts unpinned blobs while retaining pinned blobs.
|
||||||
- DB resources can be registered locally and report local-only status plus a
|
- DB resources can be registered locally and report local-only status plus a
|
||||||
read-only SQLite schema summary/hash and `crsql_changes` metadata when
|
read-only SQLite schema summary/hash and `crsql_changes` metadata when
|
||||||
present. The DB crate can extract typed read-only `crsql_changes` batches for
|
present. The DB crate and daemon can extract typed read-only `crsql_changes`
|
||||||
future sync messages. cr-sqlite loading, applying remote changes, and sync are
|
batches through `geth db changes` for future sync messages. cr-sqlite loading,
|
||||||
still roadmap work.
|
applying remote changes, and sync are still roadmap work.
|
||||||
- KV stores support local SQLite-backed create/set/get. Iroh Documents
|
- KV stores support local SQLite-backed create/set/get. Iroh Documents
|
||||||
replication and command-level prefix-capability enforcement are still roadmap
|
replication and command-level prefix-capability enforcement are still roadmap
|
||||||
work. The auth evaluator already understands `kv.write_prefix:<prefix>`
|
work. The auth evaluator already understands `kv.write_prefix:<prefix>`
|
||||||
|
|
|
||||||
|
|
@ -89,7 +89,8 @@ The bootstrap implementation provides:
|
||||||
`cleanup`, `list`
|
`cleanup`, `list`
|
||||||
- local DB resource registration: `geth db add <name> <path>` and
|
- local DB resource registration: `geth db add <name> <path>` and
|
||||||
`geth db status <name>` with schema and `crsql_changes` metadata; the DB
|
`geth db status <name>` with schema and `crsql_changes` metadata; the DB
|
||||||
crate can extract typed local `crsql_changes` batches for future sync
|
crate and daemon can extract typed local `crsql_changes` batches through
|
||||||
|
`geth db changes <name>` for future sync
|
||||||
- local SQLite-backed KV commands: `geth kv create/set/get`
|
- local SQLite-backed KV commands: `geth kv create/set/get`
|
||||||
- local JSON document commands: `geth document create/status/set/get`
|
- local JSON document commands: `geth document create/status/set/get`
|
||||||
- local daemon-lifetime pubsub snapshots: `geth pubsub pub/sub`
|
- local daemon-lifetime pubsub snapshots: `geth pubsub pub/sub`
|
||||||
|
|
|
||||||
|
|
@ -248,8 +248,20 @@ pub enum PipeCommand {
|
||||||
|
|
||||||
#[derive(Debug, Subcommand)]
|
#[derive(Debug, Subcommand)]
|
||||||
pub enum DbCommand {
|
pub enum DbCommand {
|
||||||
Add { name: String, path: PathBuf },
|
Add {
|
||||||
Status { name: String },
|
name: String,
|
||||||
|
path: PathBuf,
|
||||||
|
},
|
||||||
|
Status {
|
||||||
|
name: String,
|
||||||
|
},
|
||||||
|
Changes {
|
||||||
|
name: String,
|
||||||
|
#[arg(long)]
|
||||||
|
after_db_version: Option<i64>,
|
||||||
|
#[arg(long, default_value_t = 100)]
|
||||||
|
limit: u32,
|
||||||
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
#[derive(Debug, Subcommand)]
|
#[derive(Debug, Subcommand)]
|
||||||
|
|
@ -469,6 +481,15 @@ fn request_for_command(command: Command) -> Result<ControlRequest> {
|
||||||
Command::Db { command } => match command {
|
Command::Db { command } => match command {
|
||||||
DbCommand::Add { name, path } => ControlRequest::DbAdd { name, path },
|
DbCommand::Add { name, path } => ControlRequest::DbAdd { name, path },
|
||||||
DbCommand::Status { name } => ControlRequest::DbStatus { name },
|
DbCommand::Status { name } => ControlRequest::DbStatus { name },
|
||||||
|
DbCommand::Changes {
|
||||||
|
name,
|
||||||
|
after_db_version,
|
||||||
|
limit,
|
||||||
|
} => ControlRequest::DbChanges {
|
||||||
|
name,
|
||||||
|
after_db_version,
|
||||||
|
limit,
|
||||||
|
},
|
||||||
},
|
},
|
||||||
Command::Document { command } => match command {
|
Command::Document { command } => match command {
|
||||||
DocumentCommand::Create { name } => ControlRequest::DocumentCreate { name },
|
DocumentCommand::Create { name } => ControlRequest::DocumentCreate { name },
|
||||||
|
|
@ -893,6 +914,24 @@ fn print_response(response: ControlResponse, json: bool) -> Result<()> {
|
||||||
}
|
}
|
||||||
println!("sync_status: {}", db.sync_status);
|
println!("sync_status: {}", db.sync_status);
|
||||||
}
|
}
|
||||||
|
ControlResponse::DbChanges { db, batch } => {
|
||||||
|
println!("db: {}", db.name);
|
||||||
|
println!("changes: {}", batch.changes.len());
|
||||||
|
println!(
|
||||||
|
"max_db_version: {}",
|
||||||
|
batch
|
||||||
|
.max_db_version
|
||||||
|
.map(|version| version.to_string())
|
||||||
|
.unwrap_or_else(|| "none".to_owned())
|
||||||
|
);
|
||||||
|
println!("schema_metadata: {}", batch.schema_metadata);
|
||||||
|
for change in batch.changes {
|
||||||
|
println!(
|
||||||
|
"{}\t{}\t{}",
|
||||||
|
change.db_version, change.table_name, change.column_id
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
ControlResponse::KvCreated { kv } => {
|
ControlResponse::KvCreated { kv } => {
|
||||||
println!("created kv: {}", kv.name);
|
println!("created kv: {}", kv.name);
|
||||||
println!("id: {}", kv.id);
|
println!("id: {}", kv.id);
|
||||||
|
|
|
||||||
|
|
@ -1,5 +1,5 @@
|
||||||
use geth_auth::{AuthExplanation, AuthOp};
|
use geth_auth::{AuthExplanation, AuthOp};
|
||||||
use geth_db::DbResource;
|
use geth_db::{CrSqliteChangeBatch, DbResource};
|
||||||
use geth_document::{DocumentResource, DocumentState};
|
use geth_document::{DocumentResource, DocumentState};
|
||||||
use geth_keychain::KeychainOp;
|
use geth_keychain::KeychainOp;
|
||||||
use geth_kv::{KvEntry, KvResource};
|
use geth_kv::{KvEntry, KvResource};
|
||||||
|
|
@ -121,6 +121,11 @@ pub enum ControlRequest {
|
||||||
DbStatus {
|
DbStatus {
|
||||||
name: String,
|
name: String,
|
||||||
},
|
},
|
||||||
|
DbChanges {
|
||||||
|
name: String,
|
||||||
|
after_db_version: Option<i64>,
|
||||||
|
limit: u32,
|
||||||
|
},
|
||||||
KvCreate {
|
KvCreate {
|
||||||
name: String,
|
name: String,
|
||||||
},
|
},
|
||||||
|
|
@ -165,7 +170,7 @@ pub enum ControlRequest {
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
|
||||||
#[serde(tag = "type", rename_all = "kebab-case")]
|
#[serde(tag = "type", rename_all = "kebab-case")]
|
||||||
pub enum ControlResponse {
|
pub enum ControlResponse {
|
||||||
Status(StatusResponse),
|
Status(StatusResponse),
|
||||||
|
|
@ -262,6 +267,10 @@ pub enum ControlResponse {
|
||||||
DbStatus {
|
DbStatus {
|
||||||
db: DbResource,
|
db: DbResource,
|
||||||
},
|
},
|
||||||
|
DbChanges {
|
||||||
|
db: DbResource,
|
||||||
|
batch: CrSqliteChangeBatch,
|
||||||
|
},
|
||||||
KvCreated {
|
KvCreated {
|
||||||
kv: KvResource,
|
kv: KvResource,
|
||||||
},
|
},
|
||||||
|
|
@ -427,5 +436,15 @@ mod tests {
|
||||||
decode_request(&encode_request(&request).expect("encode")).expect("decode"),
|
decode_request(&encode_request(&request).expect("encode")).expect("decode"),
|
||||||
request
|
request
|
||||||
);
|
);
|
||||||
|
|
||||||
|
let request = ControlRequest::DbChanges {
|
||||||
|
name: "notes".to_owned(),
|
||||||
|
after_db_version: Some(7),
|
||||||
|
limit: 10,
|
||||||
|
};
|
||||||
|
assert_eq!(
|
||||||
|
decode_request(&encode_request(&request).expect("encode")).expect("decode"),
|
||||||
|
request
|
||||||
|
);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -758,6 +758,25 @@ pub fn handle_request(
|
||||||
db: db_resource_from_stored(&stored)?,
|
db: db_resource_from_stored(&stored)?,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
ControlRequest::DbChanges {
|
||||||
|
name,
|
||||||
|
after_db_version,
|
||||||
|
limit,
|
||||||
|
} => {
|
||||||
|
geth_db::validate_db_name(&name).map_err(|_| NodeError::InvalidDbName(name.clone()))?;
|
||||||
|
let stored = store
|
||||||
|
.get_db_resource_by_name(&name)?
|
||||||
|
.ok_or_else(|| NodeError::DbNotFound(name.clone()))?;
|
||||||
|
let batch = geth_db::extract_crsqlite_changes(
|
||||||
|
Path::new(&stored.path),
|
||||||
|
after_db_version,
|
||||||
|
limit,
|
||||||
|
)?;
|
||||||
|
Ok(ControlResponse::DbChanges {
|
||||||
|
db: db_resource_from_stored(&stored)?,
|
||||||
|
batch,
|
||||||
|
})
|
||||||
|
}
|
||||||
ControlRequest::KvCreate { name } => {
|
ControlRequest::KvCreate { name } => {
|
||||||
geth_kv::validate_kv_name(&name).map_err(|_| NodeError::InvalidKvName(name.clone()))?;
|
geth_kv::validate_kv_name(&name).map_err(|_| NodeError::InvalidKvName(name.clone()))?;
|
||||||
let resource_id = format!("resource:kv:{name}");
|
let resource_id = format!("resource:kv:{name}");
|
||||||
|
|
|
||||||
|
|
@ -463,6 +463,44 @@ fn db_add_and_status_register_local_db_metadata() {
|
||||||
other => panic!("unexpected response: {other:?}"),
|
other => panic!("unexpected response: {other:?}"),
|
||||||
}
|
}
|
||||||
|
|
||||||
|
let response = geth_node::handle_request(
|
||||||
|
&node,
|
||||||
|
geth_control::ControlRequest::DbChanges {
|
||||||
|
name: "notes".to_owned(),
|
||||||
|
after_db_version: None,
|
||||||
|
limit: 10,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
.expect("db changes");
|
||||||
|
match response {
|
||||||
|
geth_control::ControlResponse::DbChanges { db, batch } => {
|
||||||
|
assert_eq!(db.name, "notes");
|
||||||
|
assert_eq!(batch.changes.len(), 1);
|
||||||
|
assert_eq!(batch.max_db_version, Some(3));
|
||||||
|
assert_eq!(batch.changes[0].table_name, "notes");
|
||||||
|
assert_eq!(batch.changes[0].column_id, "body");
|
||||||
|
assert!(batch.schema_metadata.contains("schema_hash="));
|
||||||
|
}
|
||||||
|
other => panic!("unexpected response: {other:?}"),
|
||||||
|
}
|
||||||
|
|
||||||
|
let response = geth_node::handle_request(
|
||||||
|
&node,
|
||||||
|
geth_control::ControlRequest::DbChanges {
|
||||||
|
name: "notes".to_owned(),
|
||||||
|
after_db_version: Some(3),
|
||||||
|
limit: 10,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
.expect("db changes after latest");
|
||||||
|
match response {
|
||||||
|
geth_control::ControlResponse::DbChanges { batch, .. } => {
|
||||||
|
assert!(batch.changes.is_empty());
|
||||||
|
assert_eq!(batch.max_db_version, None);
|
||||||
|
}
|
||||||
|
other => panic!("unexpected response: {other:?}"),
|
||||||
|
}
|
||||||
|
|
||||||
assert!(
|
assert!(
|
||||||
geth_node::handle_request(
|
geth_node::handle_request(
|
||||||
&node,
|
&node,
|
||||||
|
|
|
||||||
|
|
@ -96,10 +96,10 @@ and file sync trees are future work.
|
||||||
`geth-db` currently registers local SQLite paths as DB resources and reports
|
`geth-db` currently registers local SQLite paths as DB resources and reports
|
||||||
local-only sync status plus a read-only SQLite schema summary/hash. It also
|
local-only sync status plus a read-only SQLite schema summary/hash. It also
|
||||||
inspects `crsql_changes` metadata when that table or view exists, reporting
|
inspects `crsql_changes` metadata when that table or view exists, reporting
|
||||||
change count, columns, and max `db_version`. The crate can extract read-only
|
change count, columns, and max `db_version`. The crate and local daemon can
|
||||||
typed change batches from `crsql_changes` with schema metadata for future sync
|
extract read-only typed change batches from `crsql_changes` with schema metadata
|
||||||
messages. Loading cr-sqlite, applying remote changes, and DB sync are future
|
through `db changes`. Loading cr-sqlite, applying remote changes, and DB sync
|
||||||
work.
|
are future work.
|
||||||
|
|
||||||
`geth-kv` currently provides a SQLite-backed local fallback for named KV stores
|
`geth-kv` currently provides a SQLite-backed local fallback for named KV stores
|
||||||
through `kv create/set/get`. Iroh Documents namespaces, prefix authorization
|
through `kv create/set/get`. Iroh Documents namespaces, prefix authorization
|
||||||
|
|
|
||||||
|
|
@ -306,7 +306,7 @@ Automerge documents.
|
||||||
- `geth db status <name>` reports local path, schema metadata, and sync state.
|
- `geth db status <name>` reports local path, schema metadata, and sync state.
|
||||||
- Nonexistent paths and invalid names produce clear errors.
|
- Nonexistent paths and invalid names produce clear errors.
|
||||||
|
|
||||||
- `[ ]` cr-sqlite change extraction.
|
- `[x]` cr-sqlite change extraction.
|
||||||
Acceptance criteria:
|
Acceptance criteria:
|
||||||
- `[x]` DB status detects whether `crsql_changes` exists.
|
- `[x]` DB status detects whether `crsql_changes` exists.
|
||||||
- `[x]` DB status reports `crsql_changes` row count, columns, and max
|
- `[x]` DB status reports `crsql_changes` row count, columns, and max
|
||||||
|
|
@ -314,8 +314,7 @@ Automerge documents.
|
||||||
- `[x]` Tests use a temp SQLite DB and deterministic fixture changes.
|
- `[x]` Tests use a temp SQLite DB and deterministic fixture changes.
|
||||||
- `[x]` The module can extract typed change batches from `crsql_changes`.
|
- `[x]` The module can extract typed change batches from `crsql_changes`.
|
||||||
- `[x]` Schema hash/version metadata is included in extracted change batches.
|
- `[x]` Schema hash/version metadata is included in extracted change batches.
|
||||||
- `[ ]` Extracted batches are exposed through a daemon control command or sync
|
- `[x]` Extracted batches are exposed through `geth db changes`.
|
||||||
protocol.
|
|
||||||
|
|
||||||
- `[ ]` DB sync over Iroh.
|
- `[ ]` DB sync over Iroh.
|
||||||
Acceptance criteria:
|
Acceptance criteria:
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue