Artifacts and staleness¶
A proxy, thumbnail, transcode, or render that PostProject knows was produced by a recorded activity is a managed artifact. PostProject can tell a host whether that artifact still reflects its inputs, explain the answer, and say whether the recorded recipe has missing information. The model is described in artifact knowledge and reproducibility; this guide shows the workflow.
Evaluation reads recorded knowledge only. It never opens media files, so it is cheap enough for a UI and gives the same answer on every machine. Whether the artifact’s files are reachable is a separate question answered by resolution.
Record completed outputs¶
After the host produces media, register its sources and actual producing recipe in one edit. Choose an existing logical asset for each representation, or explicitly create a new original asset. The task returns both identities and the activity ID; commit makes them durable together.
const pp_output_source_t outputs[] = {
{NULL, PP_REPRESENTATION_ORIGINAL, render, NULL, NULL, NULL},
{NULL, PP_REPRESENTATION_ORIGINAL, thumbnail, NULL, "example:thumbnail", NULL}};
const pp_activity_parameter_t parameter = {"example:recipe", "width", width};
const pp_activity_recipe_t recipe = {
.kind = "example:generate", .tool_name = "Fixture generator",
.tool_version = "1.0", .parameters = ¶meter, .parameter_count = 1};
pp_output_identity_t identities[2];
pp_activity_id_t activity;
status = pp_transaction_register_outputs(edit, outputs, 2, &recipe,
identities, &activity, &error);
pp_commit_receipt_t receipt = {0};
if (status == PP_OK) status = pp_transaction_commit(edit, &receipt, &error);
/* All returned identities become durable together after explicit commit. */
postproject::ActivitySpec recipe{};
recipe.kind = "example:generate";
recipe.tool = postproject::ToolIdentity{"Fixture generator", "1.0", std::nullopt};
POSTPROJECT_TRY_ASSIGN(auto edit, production.edit());
POSTPROJECT_TRY_ASSIGN(const auto registered, edit.registerOutputs(
{{render}, {thumbnail, std::nullopt, postproject::RepresentationKind::original,
"example:thumbnail", std::nullopt}},
recipe, {{"example:recipe", "width", postproject::MetadataValue::signedInteger(1920)}}));
POSTPROJECT_TRY_ASSIGN(const auto receipt, edit.commit());
// No inputs: this operation generated new media. A render supplies input IDs.
with production.edit(message="Record completed generation") as edit:
registered = edit.register_outputs(
(
(None, RepresentationKind.ORIGINAL, render),
(None, RepresentationKind.ORIGINAL, thumbnail, "example:thumbnail"),
),
ActivitySpec(
"example:generate",
tool=ToolIdentity("Fixture generator", "1.0"),
parameters={("example:recipe", "width"): 1920},
),
)
receipt = edit.commit()
# Inputs are empty because this operation generated new media.
# For a render from recorded sources, pass their IDs in ActivitySpec.inputs.
let mut recipe = OutputRecipe::new(
ActivityKind::new("example:generate")?,
std::iter::empty::<postproject_core::RepresentationId>(),
)?;
recipe.tool = Some(ToolIdentity::new(
"Fixture generator",
Some("1.0".to_owned()),
None,
)?);
recipe.parameters.push((
MetadataProperty::new(
VocabularyId::new("example:recipe")?,
PropertyId::new("width")?,
),
MetadataValue::i64(1920),
));
let prepared = prepare_outputs(
&[
OutputSource::new(
None,
RepresentationKind::Original,
MediaSource::File(render),
),
OutputSource::new(
None,
RepresentationKind::Original,
MediaSource::File(thumbnail),
),
],
recipe,
)?;
let identities = prepared.identities();
let activity = prepared.activity().id();
// Media I/O finishes before acquiring the edit's write lock.
let receipt = {
let mut edit = prepared.stage(production.begin_transaction()?)?;
edit.commit()?
};
cat > parameters.json <<'JSON'
[{"vocabulary":"example:recipe","property":"width","value":{"type":"i64","value":1920}}]
JSON
postproject --json artifact register outputs.pproj render.dat thumbnail.dat \
--activity-kind example:generate --tool-name "Fixture generator" \
--tool-version 1.0 --parameters parameters.json > registered.json
# Add --asset ID --kind derived --input REPRESENTATION_ID for a recorded input.
# Without --asset, each output explicitly creates a new original asset.
jq '{activity_id, outputs, commit_receipt}' registered.json
Pass consumed representation IDs as inputs, with optional roles. Empty inputs describe generation. Parameters belong to this actual activity and may repeat. A failed preparation stages no partial output. Registration creates neither a job nor an authored dependency observation. If an existing one-output job is being completed, retain its lease and complete it in the same edit; see job completion.
Evaluate an artifact¶
An evaluation returns one knowledge state — current, stale, indeterminate, or diverged — with structured reasons for anything that is not current, and a separate recorded recipe report listing each missing condition:
static pp_error_code_t inspect_artifact(const pp_production_t *production,
const pp_representation_id_t *artifact_id,
pp_error_t **error) {
pp_artifact_evaluation_t *evaluation = NULL;
pp_artifact_reproducibility_t *report = NULL;
pp_representation_id_t evaluated_id;
pp_artifact_knowledge_state_t state = 0;
uint32_t visited = 0;
uint8_t truncated = 0;
uint64_t reason_count = 0;
pp_error_code_t status = pp_production_evaluate_artifact(
production, *artifact_id, UINT32_C(64), UINT32_C(1000), &evaluation,
error);
if (status == PP_OK) {
status = pp_artifact_evaluation_get(
evaluation, &evaluated_id, &state, &visited, &truncated, &reason_count,
error);
}
if (status == PP_OK) {
printf("artifact state: %u, reasons: %llu\n", state,
(unsigned long long)reason_count);
status = pp_production_artifact_reproducibility(
production, *artifact_id, &report, error);
}
if (status == PP_OK) {
uint8_t reproducible = 0;
uint8_t has_activity = 0;
pp_activity_id_t activity_id;
const char *activity_kind = NULL;
uint64_t issue_count = 0;
status = pp_artifact_reproducibility_get(
report, &evaluated_id, &reproducible, &has_activity, &activity_id,
&activity_kind, &issue_count, error);
if (status == PP_OK) {
printf("recorded recipe complete: %u, missing conditions: %llu\n", reproducible,
(unsigned long long)issue_count);
}
}
pp_artifact_reproducibility_release(report);
pp_artifact_evaluation_release(evaluation);
return status;
}
void inspect_artifact(const postproject::Production &production,
const postproject::RepresentationId &artifact_id) {
const auto evaluation = production.evaluateArtifact(artifact_id, 64, 1000).value();
std::cout << "artifact state: " << static_cast<std::uint32_t>(evaluation.state)
<< '\n';
for (const auto &reason : evaluation.reasons) {
std::cout << "reason: " << static_cast<std::uint32_t>(reason.kind()) << '\n';
}
const auto reproducibility = production.artifactReproducibility(artifact_id).value();
std::cout << "recorded recipe complete: " << reproducibility.reproducible
<< ", missing conditions: " << reproducibility.issues.size()
<< '\n';
}
def inspect_artifact(production: Production, artifact_id: RepresentationId) -> None:
evaluation = production.evaluate_artifact(
artifact_id, max_depth=64, max_representations=1000
)
print(f"artifact state: {evaluation.state.name}")
for reason in evaluation.reasons:
print(f"reason: {reason.kind.name}")
reproducibility = production.artifact_reproducibility(artifact_id)
print(
f"recorded recipe complete: {reproducibility.reproducible}, "
f"missing conditions: {len(reproducibility.issues)}"
)
fn inspect_artifact(production: &SqliteProduction, artifact_id: RepresentationId) -> Result<()> {
let evaluation =
production.evaluate_artifact(artifact_id, ArtifactEvaluationLimits::new(64, 1_000)?)?;
println!("artifact state: {:?}", evaluation.state());
for reason in evaluation.reasons() {
println!("reason: {reason:?}");
}
let reproducibility = production.artifact_reproducibility(artifact_id)?;
println!(
"recorded recipe complete: {}, missing conditions: {}",
reproducibility.is_reproducible(),
reproducibility.issues().len()
);
Ok(())
}
postproject --json artifact evaluate production.pproj "$SEQUENCE_ID" \
--max-depth 64 --max-representations 1000 |
jq '{state, reasons, visited_representations, truncated}'
postproject --json artifact reproducibility production.pproj "$SEQUENCE_ID" |
jq '{reproducible, issues}'
The API calls this report “reproducibility.” It checks recorded recipe completeness, without checking tool availability, codecs, media bytes or an execution environment. A positive report does not promise deterministic or executable reproduction.
Traversal through upstream artifacts is bounded by depth and by the number of representations visited. A bound that is reached is reported as a reason and makes the result indeterminate, never current.
After a source changes¶
Staleness starts with a recorded change. When a camera original is replaced in place, the host records the new fingerprint observation; from then on every artifact produced from the old content evaluates as stale, with a reason naming the input edge, the fingerprint domain, the snapshot, and the current value:
static pp_error_code_t explain_stale_proxy(
pp_production_t *production, const pp_resource_id_t *original_resource,
const pp_representation_id_t *original_id, const char *original_path,
const pp_representation_id_t *proxy_id, pp_artifact_knowledge_state_t *state,
uint32_t *out_reason_kinds, uint32_t *out_issue_kinds, pp_error_t **error) {
pp_transaction_t *transaction = NULL;
pp_read_session_t *view = NULL;
pp_artifact_evaluation_t *evaluation = NULL;
pp_artifact_reproducibility_t *report = NULL;
pp_representation_id_t evaluated_id;
uint32_t visited = 0;
uint8_t truncated = 0;
uint64_t reason_count = 0;
uint8_t value[8];
/* Record what the host observes now: the original's bytes changed. */
pp_error_code_t status =
pp_production_read_session(production, &view, error);
if (status == PP_OK) {
status = pp_read_session_begin_edit(view, NULL, &transaction, error);
}
pp_read_session_release(view);
view = NULL;
if (status == PP_OK && !host_fingerprint(original_path, value)) {
status = PP_ERROR_IO;
}
if (status == PP_OK) {
status = pp_transaction_record_resource_fingerprint(
transaction, *original_resource, HOST_RESOURCE_ALGORITHM, 1, value,
sizeof value, error);
}
if (status == PP_OK) {
status = pp_transaction_record_representation_fingerprint(
transaction, *original_id, HOST_REPRESENTATION_ALGORITHM, 1, value,
sizeof value, error);
}
if (status == PP_OK) {
pp_commit_receipt_t commit_receipt;
status = pp_transaction_commit(transaction, &commit_receipt, error);
}
if (status == PP_OK) {
status = pp_production_evaluate_artifact(
production, *proxy_id, UINT32_C(64), UINT32_C(1000), &evaluation, error);
}
if (status == PP_OK) {
status =
pp_artifact_evaluation_get(evaluation, &evaluated_id, state, &visited,
&truncated, &reason_count, error);
}
for (uint64_t i = 0; status == PP_OK && i < reason_count; ++i) {
pp_artifact_reason_kind_t kind = 0;
status = pp_artifact_evaluation_get_reason_kind(evaluation, i, &kind, error);
pp_artifact_reason_t reason = {0};
if (status == PP_OK) {
status = pp_artifact_evaluation_get_reason(evaluation, i, kind, &reason, error);
}
if (status == PP_OK) {
*out_reason_kinds |= reason.kind < 32 ? UINT32_C(1) << reason.kind : 0;
printf("reason %u\n", kind);
if (kind == PP_ARTIFACT_REASON_FINGERPRINT_CHANGED ||
kind == PP_ARTIFACT_REASON_FINGERPRINT_EVIDENCE_MISSING) {
printf("%s edge: %s v%u, snapshot %llu bytes, now %llu\n",
reason.edge_kind == PP_ARTIFACT_EDGE_INPUT ? "input" : "output",
reason.fingerprint_algorithm != NULL ? reason.fingerprint_algorithm
: "-",
reason.fingerprint_version,
(unsigned long long)reason.snapshot_value_length,
(unsigned long long)reason.current_value_length);
}
}
}
if (status == PP_OK) {
status = pp_production_artifact_reproducibility(production, *proxy_id,
&report, error);
}
if (status == PP_OK) {
uint8_t reproducible = 0;
uint8_t has_activity = 0;
pp_activity_id_t activity_id = {0};
const char *activity_kind;
uint64_t issue_count = 0;
status = pp_artifact_reproducibility_get(
report, &evaluated_id, &reproducible, &has_activity, &activity_id,
&activity_kind, &issue_count, error);
/* Issues name what a regeneration would still be missing. */
for (uint64_t i = 0; status == PP_OK && i < issue_count; ++i) {
pp_artifact_reproducibility_issue_kind_t kind = 0;
status = pp_artifact_reproducibility_get_issue_kind(report, i, &kind, error);
pp_artifact_reproducibility_issue_t issue = {0};
if (status == PP_OK) {
status = pp_artifact_reproducibility_get_issue(report, i, kind, &issue, error);
}
if (status == PP_OK) {
*out_issue_kinds |= issue.kind < 32 ? UINT32_C(1) << issue.kind : 0;
printf("reproducibility issue %u\n", issue.kind);
}
}
}
pp_artifact_reproducibility_release(report);
pp_artifact_evaluation_release(evaluation);
pp_transaction_release(transaction);
return status;
}
postproject::ArtifactEvaluation evaluate_after_change(
postproject::Production &production,
const postproject::Representation &original,
const postproject::Fingerprint &new_resource_fingerprint,
const postproject::Fingerprint &new_representation_fingerprint,
const postproject::RepresentationId &proxy_id) {
// Record what the host observed after the original was re-exported.
auto transaction = production.readSession().value().edit().value();
transaction.recordResourceFingerprint(original.resources.front().id,
new_resource_fingerprint).value();
transaction.recordRepresentationFingerprint(original.id,
new_representation_fingerprint).value();
transaction.commit().value();
const auto evaluation = production.evaluateArtifact(proxy_id).value();
if (evaluation.state == postproject::ArtifactKnowledgeState::stale) {
for (const auto &reason : evaluation.reasons) {
std::cout << "stale because " << static_cast<std::uint32_t>(reason.kind())
<< '\n';
if (const auto *changed = std::get_if<postproject::ArtifactFingerprintChanged>(&reason.value())) {
std::cout << changed->fingerprint_algorithm << " on "
<< changed->representation_id.toString().value() << '\n';
}
}
}
// Reproducibility says whether the recorded activity suffices to redo it.
const auto reproducibility = production.artifactReproducibility(proxy_id).value();
for (const auto &issue : reproducibility.issues) {
std::cout << "cannot reproduce: " << static_cast<std::uint32_t>(
postproject::artifactReproducibilityIssueKind(issue).value())
<< '\n';
if (const auto *ambiguous = std::get_if<postproject::ReproducibilityProducerAmbiguous>(&issue)) {
std::cout << ambiguous->activityCount() << " recorded producers\n";
}
}
return evaluation;
}
def record_fingerprints(
production: Production, representation: Representation, path: Path
) -> None:
# Observations in the caller's own fingerprint domain.
with production.read_session() as view, view.edit() as transaction:
(resource,) = view.representation(representation.id).resources
digest = hashlib.sha256(path.read_bytes()).digest()
transaction.record_resource_fingerprint(
resource.id, Fingerprint("example-sha256", 1, digest)
)
transaction.record_representation_fingerprint(
representation.id,
Fingerprint(
"example-sha256-representation", 1, hashlib.sha256(digest).digest()
),
)
transaction.commit()
def explain_proxy(
production: Production, proxy_id: RepresentationId
) -> tuple[ArtifactEvaluation, ArtifactReproducibility]:
evaluation = production.evaluate_artifact(
proxy_id, max_depth=64, max_representations=1000
)
print(f"proxy is {evaluation.state.name}")
for reason in evaluation.reasons:
print(f" {reason.kind.name}")
if isinstance(reason, ArtifactFingerprintChanged):
print(
f" {reason.fingerprint_algorithm}: "
f"{reason.snapshot_value.hex()[:16]} -> {reason.current_value.hex()[:16]}"
)
# Missing recorded recipe information; execution is a separate check.
reproducibility = production.artifact_reproducibility(proxy_id)
for issue in reproducibility.issues:
print(f" recorded recipe missing: {issue.kind.name}")
if isinstance(issue, ReproducibilityToolMissing):
print(f" record the tool used by activity {issue.activity_id}")
return evaluation, reproducibility
fn record_changed_file(
production: &mut SqliteProduction,
asset_id: AssetId,
representation_id: RepresentationId,
path: &Path,
) -> Result<postproject_core::CommitReceipt> {
// Record the new resource observation first.
let view = production.read_session()?;
let resource_id = view.read().resources(representation_id)?[0].id();
let observed = fingerprint_file(path)?;
{
let mut transaction = production.begin_edit(view.decision_base())?;
transaction.record_resource_fingerprint(resource_id, observed.fingerprint())?;
transaction.commit()?;
}
// Then recompute the representation fingerprint from current resources.
let view = production.read_session()?;
let representation = view
.read()
.representations(asset_id)?
.into_iter()
.find(|representation| representation.id() == representation_id)
.expect("representation belongs to the asset");
let aggregate = fingerprint_representation(
representation.content_structure(),
&view.read().resources(representation_id)?,
)?;
let mut transaction = production.begin_edit(view.decision_base())?;
transaction.record_representation_fingerprint(representation_id, &aggregate)?;
transaction.commit()
}
fn evaluate_proxy(
production: &SqliteProduction,
proxy_id: RepresentationId,
) -> Result<ArtifactKnowledgeState> {
// Evaluation reads recorded knowledge only; it never touches media files.
let evaluation =
production.evaluate_artifact(proxy_id, ArtifactEvaluationLimits::new(64, 1_000)?)?;
for reason in evaluation.reasons() {
match reason {
ArtifactKnowledgeReason::FingerprintChanged {
representation_id,
algorithm,
..
} => println!("{representation_id} changed ({algorithm})"),
other => println!("reason: {other:?}"),
}
}
let reproducibility = production.artifact_reproducibility(proxy_id)?;
for issue in reproducibility.issues() {
match issue {
ArtifactReproducibilityIssue::ParametersMissing { activity_id } => {
println!("activity {activity_id} recorded no parameters");
}
other => println!("reproducibility issue: {other:?}"),
}
}
Ok(evaluation.state())
}
printf 'regraded camera original' > rushes/A001.mov
DECISION_BASE=$(postproject --json inspect provenance.pproj | jq -r .decision_base)
postproject --decision-base "$DECISION_BASE" media fingerprint provenance.pproj "$RESOURCE_ID" rushes/A001.mov
# Evaluation compares the recorded input snapshots with current fingerprints.
postproject --json artifact evaluate provenance.pproj "$PROXY_ID" |
jq -r '.state, (.reasons[] | "\(.kind) \(.edge) \(.representation_id)")'
postproject --json artifact reproducibility provenance.pproj "$PROXY_ID" |
jq -r '"reproducible: \(.reproducible)", (.issues[] | .kind)'
A reason chain also covers artifacts produced from other artifacts, such as a render made from a stale proxy, and changes that arrive through a required dependency.
Evaluation never reads files, so it cannot notice a source that someone else replaced, for example graded media copied over the original by another application. When a source may have changed outside the host, observe it first and then evaluate. The observation reports whether the content changed, and records nothing when it did not. A worker also observes its inputs before it claims a job, so the activity’s snapshots describe the content it actually read.
Find and regenerate stale artifacts¶
To list every stale artifact, optionally only downstream of one source, page through the stale-artifact query described in bounded queries. To regenerate one, derive the job that would reproduce it from its producing activity and enqueue it explicitly; see plan and enqueue regeneration. Nothing is regenerated automatically.