Add safe file-root apply
This commit is contained in:
parent
bab13cc0a1
commit
5c91635ea2
8 changed files with 394 additions and 14 deletions
|
|
@ -142,9 +142,10 @@ Roadmap items should be actionable and checkable:
|
|||
remote tree metadata and CAS tree bytes, recording a peer-qualified remote
|
||||
root with path `remote:<node>:<name>` without applying files or overwriting
|
||||
same-named local roots. Background live-sync refreshes authorized remote file
|
||||
roots from sync-status `cas-tree:<name>` watermarks. Durable local
|
||||
file-conflict records can be listed and resolved manually; automatic
|
||||
cross-node conflict detection and file application are still roadmap work.
|
||||
roots from sync-status `cas-tree:<name>` watermarks. `geth cas root apply`
|
||||
materializes missing files/directories from a CAS tree without deleting extras
|
||||
or overwriting local edits, and records conflicts for manual resolution.
|
||||
Richer three-way file application remains roadmap work.
|
||||
- DB resources can be registered locally and report local-only status plus a
|
||||
read-only SQLite schema summary/hash and `crsql_changes` metadata when
|
||||
present. The DB crate and daemon can extract typed read-only `crsql_changes`
|
||||
|
|
|
|||
10
README.md
10
README.md
|
|
@ -108,9 +108,10 @@ The bootstrap implementation provides:
|
|||
`unpin`, `cleanup`, `providers`, `list`; remote fetch accepts
|
||||
`--bearer-secret <secret>`
|
||||
- local CAS tree objects describe file trees and are stored as CAS blobs
|
||||
- local file-root commands: `geth cas root add/list/scan/sync`; root sync pulls
|
||||
authorized remote tree metadata and CAS tree bytes into a peer-qualified
|
||||
remote root without writing files
|
||||
- local file-root commands: `geth cas root add/list/scan/sync/apply`; root sync
|
||||
pulls authorized remote tree metadata and CAS tree bytes into a peer-qualified
|
||||
remote root, and apply materializes a tree without deleting files or
|
||||
overwriting local edits
|
||||
- local file conflict metadata commands:
|
||||
`geth cas conflict record/list/resolve`
|
||||
- local DB resource registration: `geth db add <name> <path>` and
|
||||
|
|
@ -180,6 +181,9 @@ local daemon skip unchanged or unauthorized streams.
|
|||
File roots advertise `cas-tree:<name>` watermarks when the caller has
|
||||
`cas.fetch`; background live-sync imports updated tree metadata and CAS tree
|
||||
bytes into peer-qualified remote roots without writing files.
|
||||
`geth cas root apply <root> --to <path>` can then materialize that tree locally:
|
||||
it creates missing directories/files, never deletes extra files, never
|
||||
overwrites differing local files, and records conflicts for manual resolution.
|
||||
Named KV stores participate in the same live-sync loop once they exist locally:
|
||||
manual `geth kv sync <node-id> <name>` and background ticks require `kv.read`
|
||||
on the remote `resource:kv:<name>` and import only remote entries that are not
|
||||
|
|
|
|||
|
|
@ -300,6 +300,13 @@ pub enum CasRootCommand {
|
|||
#[arg(long)]
|
||||
bearer_secret: Option<String>,
|
||||
},
|
||||
Apply {
|
||||
source: String,
|
||||
#[arg(long)]
|
||||
to: PathBuf,
|
||||
#[arg(long)]
|
||||
dry_run: bool,
|
||||
},
|
||||
}
|
||||
|
||||
#[derive(Debug, Subcommand)]
|
||||
|
|
@ -751,6 +758,15 @@ fn request_for_command(command: Command) -> Result<ControlRequest> {
|
|||
name,
|
||||
bearer_secret,
|
||||
},
|
||||
CasRootCommand::Apply {
|
||||
source,
|
||||
to,
|
||||
dry_run,
|
||||
} => ControlRequest::CasRootApply {
|
||||
source,
|
||||
target: to,
|
||||
dry_run,
|
||||
},
|
||||
},
|
||||
CasCommand::Conflict { command } => match command {
|
||||
CasConflictCommand::Record {
|
||||
|
|
@ -1295,6 +1311,32 @@ fn print_response(response: ControlResponse, json: bool) -> Result<()> {
|
|||
println!("reason: {reason}");
|
||||
println!("note: {note}");
|
||||
}
|
||||
ControlResponse::CasRootApplied {
|
||||
source,
|
||||
target,
|
||||
files_written,
|
||||
dirs_created,
|
||||
conflicts,
|
||||
dry_run,
|
||||
note,
|
||||
} => {
|
||||
println!("applied file root: {source}");
|
||||
println!("target: {}", target.display());
|
||||
println!("dry_run: {dry_run}");
|
||||
println!("files_written: {files_written}");
|
||||
println!("dirs_created: {dirs_created}");
|
||||
println!("conflicts: {}", conflicts.len());
|
||||
for conflict in conflicts {
|
||||
println!(
|
||||
"{}\t{}\t{}\t{}",
|
||||
conflict.id,
|
||||
conflict.path,
|
||||
conflict.kind.as_str(),
|
||||
conflict.status.as_str()
|
||||
);
|
||||
}
|
||||
println!("note: {note}");
|
||||
}
|
||||
ControlResponse::CasConflictRecorded { conflict } => {
|
||||
println!("recorded conflict: {}", conflict.id);
|
||||
print_file_conflict(&conflict);
|
||||
|
|
|
|||
|
|
@ -86,6 +86,11 @@ pub enum ControlRequest {
|
|||
name: String,
|
||||
bearer_secret: Option<String>,
|
||||
},
|
||||
CasRootApply {
|
||||
source: String,
|
||||
target: PathBuf,
|
||||
dry_run: bool,
|
||||
},
|
||||
CasConflictRecord {
|
||||
root: String,
|
||||
path: String,
|
||||
|
|
@ -403,6 +408,15 @@ pub enum ControlResponse {
|
|||
reason: String,
|
||||
note: String,
|
||||
},
|
||||
CasRootApplied {
|
||||
source: String,
|
||||
target: PathBuf,
|
||||
files_written: usize,
|
||||
dirs_created: usize,
|
||||
conflicts: Vec<FileConflict>,
|
||||
dry_run: bool,
|
||||
note: String,
|
||||
},
|
||||
CasConflictRecorded {
|
||||
conflict: FileConflict,
|
||||
},
|
||||
|
|
@ -1444,6 +1458,30 @@ mod tests {
|
|||
response
|
||||
);
|
||||
|
||||
let request = ControlRequest::CasRootApply {
|
||||
source: "remote-node-peer-notes".to_owned(),
|
||||
target: PathBuf::from("/tmp/notes"),
|
||||
dry_run: true,
|
||||
};
|
||||
assert_eq!(
|
||||
decode_request(&encode_request(&request).expect("encode")).expect("decode"),
|
||||
request
|
||||
);
|
||||
|
||||
let response = ControlResponse::CasRootApplied {
|
||||
source: "remote-node-peer-notes".to_owned(),
|
||||
target: PathBuf::from("/tmp/notes"),
|
||||
files_written: 1,
|
||||
dirs_created: 1,
|
||||
conflicts: Vec::new(),
|
||||
dry_run: true,
|
||||
note: "safe apply".to_owned(),
|
||||
};
|
||||
assert_eq!(
|
||||
decode_response(&encode_response(&response).expect("encode")).expect("decode"),
|
||||
response
|
||||
);
|
||||
|
||||
let request = ControlRequest::CasConflictResolve {
|
||||
conflict_id: "file-conflict:notes:1".to_owned(),
|
||||
resolution: "keep-local".to_owned(),
|
||||
|
|
|
|||
|
|
@ -3,8 +3,8 @@ pub mod service;
|
|||
use base64::Engine;
|
||||
use geth_auth::{AuthExplanation, AuthOp, AuthOpKind};
|
||||
use geth_cas::{
|
||||
BlobInfoSummary, FileConflict, FileConflictKind, FileConflictResolution, FileConflictStatus,
|
||||
FileRoot, FileRootScan, LocalCas, hash_path,
|
||||
BlobInfoSummary, CasTreeEntryKind, CasTreeObject, FileConflict, FileConflictKind,
|
||||
FileConflictResolution, FileConflictStatus, FileRoot, FileRootScan, LocalCas, hash_path,
|
||||
};
|
||||
use geth_config::{GethConfig, GethPaths, RelayMode};
|
||||
use geth_control::{
|
||||
|
|
@ -43,7 +43,7 @@ use geth_types::{
|
|||
ResourceName, SshCertId, SshCertRequestId, UnixMillis,
|
||||
};
|
||||
use std::collections::{BTreeMap, VecDeque};
|
||||
use std::path::Path;
|
||||
use std::path::{Component, Path, PathBuf};
|
||||
use std::sync::{Arc, Mutex};
|
||||
use std::time::Duration;
|
||||
use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader};
|
||||
|
|
@ -3397,6 +3397,11 @@ pub fn handle_request(
|
|||
},
|
||||
})
|
||||
}
|
||||
ControlRequest::CasRootApply {
|
||||
source,
|
||||
target,
|
||||
dry_run,
|
||||
} => apply_file_root_tree(&store, node, &source, target, dry_run),
|
||||
ControlRequest::CasConflictRecord {
|
||||
root,
|
||||
path,
|
||||
|
|
@ -4628,6 +4633,203 @@ fn file_conflict_from_stored(stored: StoredFileConflict) -> Result<FileConflict,
|
|||
})
|
||||
}
|
||||
|
||||
fn apply_file_root_tree(
|
||||
store: &Store,
|
||||
node: &LocalNode,
|
||||
source: &str,
|
||||
target: PathBuf,
|
||||
dry_run: bool,
|
||||
) -> Result<ControlResponse, NodeError> {
|
||||
geth_cas::validate_file_root_name(source)?;
|
||||
let source_root = store
|
||||
.get_file_root_by_name(source)?
|
||||
.ok_or_else(|| NodeError::ResourceNotFound(format!("file-root:{source}")))?;
|
||||
let tree_hash = source_root
|
||||
.latest_tree_hash
|
||||
.as_ref()
|
||||
.ok_or_else(|| NodeError::ResourceNotFound(format!("file-root-tree:{source}")))?;
|
||||
let tree_hash = BlobHash::new(tree_hash.clone());
|
||||
let cas = LocalCas::new(node.paths.cas_dir());
|
||||
let tree_bytes = cas.read_bytes(&tree_hash)?;
|
||||
let tree: CasTreeObject = geth_codec::decode_canonical(&tree_bytes)?;
|
||||
|
||||
if target.exists() && !target.is_dir() {
|
||||
return Err(NodeError::Cas(geth_cas::CasError::TreeRootNotDirectory(
|
||||
target.display().to_string(),
|
||||
)));
|
||||
}
|
||||
if !dry_run {
|
||||
std::fs::create_dir_all(&target)?;
|
||||
}
|
||||
|
||||
let mut files_written = 0;
|
||||
let mut dirs_created = 0;
|
||||
let mut conflicts = Vec::new();
|
||||
for entry in &tree.entries {
|
||||
let out = safe_tree_output_path(&target, &entry.path)?;
|
||||
match entry.kind {
|
||||
CasTreeEntryKind::Directory => {
|
||||
if out.exists() {
|
||||
if !out.is_dir() {
|
||||
conflicts.push(record_apply_conflict(
|
||||
store,
|
||||
&source_root,
|
||||
&entry.path,
|
||||
&tree_hash,
|
||||
"remote directory conflicts with a local non-directory path",
|
||||
dry_run,
|
||||
)?);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
dirs_created += 1;
|
||||
if !dry_run {
|
||||
std::fs::create_dir_all(&out)?;
|
||||
}
|
||||
}
|
||||
CasTreeEntryKind::File => {
|
||||
let Some(blob) = &entry.blob else {
|
||||
conflicts.push(record_apply_conflict(
|
||||
store,
|
||||
&source_root,
|
||||
&entry.path,
|
||||
&tree_hash,
|
||||
"remote file entry is missing a CAS blob hash",
|
||||
dry_run,
|
||||
)?);
|
||||
continue;
|
||||
};
|
||||
if out.exists() {
|
||||
if !out.is_file() {
|
||||
conflicts.push(record_apply_conflict(
|
||||
store,
|
||||
&source_root,
|
||||
&entry.path,
|
||||
&tree_hash,
|
||||
"remote file conflicts with a local non-file path",
|
||||
dry_run,
|
||||
)?);
|
||||
continue;
|
||||
}
|
||||
if hash_path(&out)? != *blob {
|
||||
conflicts.push(record_apply_conflict(
|
||||
store,
|
||||
&source_root,
|
||||
&entry.path,
|
||||
&tree_hash,
|
||||
"local file differs from remote CAS tree; refusing to overwrite",
|
||||
dry_run,
|
||||
)?);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
files_written += 1;
|
||||
if !dry_run {
|
||||
let bytes = cas.read_bytes(blob)?;
|
||||
write_file_atomic(&out, &bytes)?;
|
||||
set_executable_bit(&out, entry.executable)?;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Ok(ControlResponse::CasRootApplied {
|
||||
source: source.to_owned(),
|
||||
target,
|
||||
files_written,
|
||||
dirs_created,
|
||||
conflicts,
|
||||
dry_run,
|
||||
note: "safe file-root apply creates missing paths but never deletes extra files or overwrites differing local files; conflicts are recorded for manual resolution".to_owned(),
|
||||
})
|
||||
}
|
||||
|
||||
fn safe_tree_output_path(root: &Path, relative: &str) -> Result<PathBuf, NodeError> {
|
||||
let relative_path = Path::new(relative);
|
||||
if relative_path.is_absolute() {
|
||||
return Err(NodeError::Cas(geth_cas::CasError::NonRelativeTreePath(
|
||||
relative.to_owned(),
|
||||
)));
|
||||
}
|
||||
for component in relative_path.components() {
|
||||
if !matches!(component, Component::Normal(_)) {
|
||||
return Err(NodeError::Cas(geth_cas::CasError::NonRelativeTreePath(
|
||||
relative.to_owned(),
|
||||
)));
|
||||
}
|
||||
}
|
||||
Ok(root.join(relative_path))
|
||||
}
|
||||
|
||||
fn record_apply_conflict(
|
||||
store: &Store,
|
||||
root: &StoredFileRoot,
|
||||
path: &str,
|
||||
remote_tree: &BlobHash,
|
||||
detail: &str,
|
||||
dry_run: bool,
|
||||
) -> Result<FileConflict, NodeError> {
|
||||
let created_at = geth_store::now_ms();
|
||||
let stored = StoredFileConflict {
|
||||
conflict_id: generated_file_conflict_id(
|
||||
&root.name,
|
||||
path,
|
||||
FileConflictKind::ConcurrentEdit.as_str(),
|
||||
created_at,
|
||||
),
|
||||
root_name: root.name.clone(),
|
||||
resource_id: root.resource_id.clone(),
|
||||
path: path.to_owned(),
|
||||
kind: FileConflictKind::ConcurrentEdit.as_str().to_owned(),
|
||||
status: FileConflictStatus::Open.as_str().to_owned(),
|
||||
base_tree_hash: None,
|
||||
local_tree_hash: None,
|
||||
remote_tree_hash: Some(remote_tree.to_string()),
|
||||
detail: detail.to_owned(),
|
||||
resolution: None,
|
||||
resolution_note: None,
|
||||
created_at_ms: created_at,
|
||||
resolved_at_ms: None,
|
||||
};
|
||||
if !dry_run {
|
||||
store.upsert_file_conflict(&stored)?;
|
||||
}
|
||||
file_conflict_from_stored(stored)
|
||||
}
|
||||
|
||||
fn write_file_atomic(path: &Path, bytes: &[u8]) -> Result<(), NodeError> {
|
||||
if let Some(parent) = path.parent() {
|
||||
std::fs::create_dir_all(parent)?;
|
||||
}
|
||||
let tmp = path.with_extension(format!(
|
||||
"geth-tmp-{}",
|
||||
geth_crypto::blake3_hex(path.display().to_string().as_bytes())
|
||||
));
|
||||
std::fs::write(&tmp, bytes)?;
|
||||
std::fs::rename(tmp, path)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
fn set_executable_bit(path: &Path, executable: bool) -> Result<(), NodeError> {
|
||||
use std::os::unix::fs::PermissionsExt;
|
||||
let mut permissions = std::fs::metadata(path)?.permissions();
|
||||
let mut mode = permissions.mode();
|
||||
if executable {
|
||||
mode |= 0o111;
|
||||
} else {
|
||||
mode &= !0o111;
|
||||
}
|
||||
permissions.set_mode(mode);
|
||||
std::fs::set_permissions(path, permissions)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(not(unix))]
|
||||
fn set_executable_bit(_path: &Path, _executable: bool) -> Result<(), NodeError> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn ensure_resource_exists(store: &Store, resource_id: &str) -> Result<(), NodeError> {
|
||||
if store
|
||||
.list_resources()?
|
||||
|
|
|
|||
|
|
@ -394,6 +394,91 @@ fn cas_file_root_add_list_scan_detects_changes() {
|
|||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cas_file_root_apply_writes_missing_files_and_records_conflicts() {
|
||||
let home = tempfile::tempdir().expect("tempdir");
|
||||
let paths = geth_config::GethPaths::from_home(home.path());
|
||||
let node = geth_node::init_node(&paths).expect("init node");
|
||||
let source_path = home.path().join("source-root");
|
||||
std::fs::create_dir_all(source_path.join("dir")).expect("create source root");
|
||||
std::fs::write(source_path.join("a.txt"), b"remote").expect("write remote a");
|
||||
std::fs::write(source_path.join("dir").join("b.txt"), b"remote b").expect("write remote b");
|
||||
|
||||
geth_node::handle_request(
|
||||
&node,
|
||||
geth_control::ControlRequest::CasRootAdd {
|
||||
name: "remote-node-peer-shared".to_owned(),
|
||||
path: source_path,
|
||||
},
|
||||
)
|
||||
.expect("add source root");
|
||||
let tree = match geth_node::handle_request(
|
||||
&node,
|
||||
geth_control::ControlRequest::CasRootScan {
|
||||
name: "remote-node-peer-shared".to_owned(),
|
||||
},
|
||||
)
|
||||
.expect("scan source root")
|
||||
{
|
||||
geth_control::ControlResponse::CasRootScanned { scan } => scan.tree.hash,
|
||||
other => panic!("unexpected response: {other:?}"),
|
||||
};
|
||||
|
||||
let target_path = home.path().join("target-root");
|
||||
std::fs::create_dir_all(&target_path).expect("create target root");
|
||||
std::fs::write(target_path.join("a.txt"), b"local edit").expect("write local a");
|
||||
|
||||
let response = geth_node::handle_request(
|
||||
&node,
|
||||
geth_control::ControlRequest::CasRootApply {
|
||||
source: "remote-node-peer-shared".to_owned(),
|
||||
target: target_path.clone(),
|
||||
dry_run: false,
|
||||
},
|
||||
)
|
||||
.expect("apply source root");
|
||||
match response {
|
||||
geth_control::ControlResponse::CasRootApplied {
|
||||
files_written,
|
||||
dirs_created,
|
||||
conflicts,
|
||||
note,
|
||||
..
|
||||
} => {
|
||||
assert_eq!(files_written, 1);
|
||||
assert_eq!(dirs_created, 1);
|
||||
assert_eq!(conflicts.len(), 1);
|
||||
assert_eq!(conflicts[0].path, "a.txt");
|
||||
assert_eq!(conflicts[0].remote_tree, Some(tree));
|
||||
assert!(note.contains("never deletes"));
|
||||
}
|
||||
other => panic!("unexpected response: {other:?}"),
|
||||
}
|
||||
assert_eq!(
|
||||
std::fs::read(target_path.join("dir").join("b.txt")).expect("read applied b"),
|
||||
b"remote b"
|
||||
);
|
||||
assert_eq!(
|
||||
std::fs::read(target_path.join("a.txt")).expect("read local a"),
|
||||
b"local edit"
|
||||
);
|
||||
|
||||
let response = geth_node::handle_request(
|
||||
&node,
|
||||
geth_control::ControlRequest::CasConflictList {
|
||||
root: Some("remote-node-peer-shared".to_owned()),
|
||||
},
|
||||
)
|
||||
.expect("list apply conflicts");
|
||||
match response {
|
||||
geth_control::ControlResponse::CasConflictList { conflicts } => {
|
||||
assert_eq!(conflicts.len(), 1);
|
||||
assert_eq!(conflicts[0].path, "a.txt");
|
||||
}
|
||||
other => panic!("unexpected response: {other:?}"),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cas_file_conflict_record_list_and_resolve() {
|
||||
let home = tempfile::tempdir().expect("tempdir");
|
||||
|
|
|
|||
|
|
@ -133,9 +133,13 @@ pull authorized remote file-root tree metadata with `geth cas root sync <node>
|
|||
remote CAS tree bytes and records a peer-qualified remote root whose path is
|
||||
`remote:<node>:<name>` without applying files. File roots also participate in
|
||||
the daemon background live-sync loop through authorized `cas-tree:<name>`
|
||||
watermarks. The daemon also has durable local file-conflict records with
|
||||
explicit resolution choices; future cross-node file sync will create those
|
||||
records automatically instead of silently applying ambiguous remote changes.
|
||||
watermarks. `geth cas root apply <root> --to <path>` is the first conservative
|
||||
materialization path: it creates missing files and directories from the CAS tree,
|
||||
does not delete extra local files, does not overwrite differing local files, and
|
||||
records conflicts for manual resolution. The daemon also has durable local
|
||||
file-conflict records with explicit resolution choices; future cross-node file
|
||||
sync will extend those records instead of silently applying ambiguous remote
|
||||
changes.
|
||||
|
||||
As a bootstrap network path, `geth cas fetch <node-id> <hash>` dials an
|
||||
imported signed peer card over the daemon-owned Iroh control ALPN. The serving
|
||||
|
|
|
|||
|
|
@ -519,8 +519,12 @@ and future group key evolution.
|
|||
or overwriting same-named local roots.
|
||||
- `[x]` Background live-sync imports updated authorized file-root trees from
|
||||
sync-status `cas-tree:<name>` watermarks and stores per-peer cursors.
|
||||
- `[ ]` Future completion applies file-root sync safely with conflict
|
||||
detection and explicit resolution.
|
||||
- `[x]` `geth cas root apply <root> --to <path>` materializes missing files
|
||||
and directories from a CAS tree without deleting extras or overwriting local
|
||||
edits.
|
||||
- `[x]` Apply records durable conflicts for differing local paths.
|
||||
- `[ ]` Future completion adds a richer three-way apply using base/local/remote
|
||||
trees for automatic safe updates and deletes.
|
||||
|
||||
- `[~]` Conflict handling.
|
||||
Acceptance criteria:
|
||||
|
|
|
|||
Loading…
Reference in a new issue