Confidential Containers Attestation Implementation
For developer’s eyes only :-)
In the previous blog, I wrote about the process of secret retrieval by a confidential container, including the Request-Challenge-Attestation-Response (RCAR) handshake that forms the basis. Secure secret retrieval using remote attestation is a fundamental use case for the confidential containers project.
The following steps summarise the entire process:
- The Client initiates an RCAR handshake with the Server to get the attestation token (get_token)
- The Client uses the received token to retrieve the secret from the Server (get_resource)
As a developer, you may be interested in learning the internals of the confidential containers attestation implementation , contributing to its development, or using it in your projects.
In the following sections, I will discuss some of the implementation (code) aspects of the components involved. I hope you find this useful if you want to contribute code or reuse it for your project.
Overview of the Code Layout and Key Functions
Let’s start with the client side and focus on two main aspects — get_token and get_resource
I have used the 0.9.0 version of the code for this write-up. The latest code may be slightly different, but it should be easy to understand.
The primary component on the client-side is the attestation-agent (AA).
Code Location for client side components:
Repository: guest-components
File: attestation-agent/kbs_protocol/src/client/rcar_client.rs
This file contains the essential functions that initiate the RCAR handshake. Two key functions are get_token and get_resource.
get_token function:
The get_token function initiates the RCAR handshake and retrieves a token.
pub async fn get_token(&mut self) -> Result<(Token, TeeKeyPair)> {
if let Some(token) = &self.token {
if token.check_valid().is_err() {
self.repeat_rcar_handshake().await?;
}
} else {
self.repeat_rcar_handshake().await?;
}
assert!(self.token.is_some());
let token = self.token.clone().unwrap();
let tee_key = self.tee_key.clone();
Ok((token, tee_key))
}High-level flow of get_token by the attestation-agent:
get_token → rcar_handshake →
- Sends an auth request to the Server’s (Key Broker Service) auth_endpoint (
/auth) - Gets challenge from Key Broker Service (nonce)
- Generates public key pair
After the attestation-agent (AA) receives the attestation challenge from the Key Broker Service (KBS), it generates an ephemeral asymmetric key pair in the TEE. The TEE stores the private key. The public key and its details are exported, placed in the tee-pubkey field (refer to the source code below), and sent to the KBS with the attestation evidence. The hash of the tee-pubkey field must be included in the custom field of TEE evidence and signed by TEE hardware. This ensures that the public key is bound to the TEE.
Use the KBS Attestation Protocol Documentation as a reference.
rcar_handshake function:
This is the main function on the client-side.
/// Perform RCAR handshake with the given kbs host. If succeeds, the client will
/// store the token.
///
/// Note: if RCAR succeeds, the http client will record the cookie with the kbs server,
/// which means that this client can be then used to retrieve resources.
async fn rcar_handshake(&mut self) -> anyhow::Result<()> {
let auth_endpoint = format!("{}/{KBS_PREFIX}/auth", self.kbs_host_url);
let tee = match &self._tee {
ClientTee::Unitialized => {
let tee = self.provider.get_tee_type().await?;
self._tee = ClientTee::_Initializated(tee);
tee
}
ClientTee::_Initializated(tee) => *tee,
};
let request = Request {
version: String::from(KBS_PROTOCOL_VERSION),
tee,
extra_params: String::new(),
};
debug!("send auth request to {auth_endpoint}");
let challenge = self
.http_client
.post(auth_endpoint)
.header("Content-Type", "application/json")
.json(&request)
.send()
.await?
.json::<Challenge>()
.await?;
debug!("get challenge: {challenge:#?}");
let tee_pubkey = self.tee_key.export_pubkey()?;
let runtime_data = json!({
"tee-pubkey": tee_pubkey,
"nonce": challenge.nonce,
});
let runtime_data =
serde_json::to_string(&runtime_data).context("serialize runtime data failed")?;
let evidence = self
.generate_evidence(tee, runtime_data, challenge.nonce)
.await?;
debug!("get evidence with challenge: {evidence}");
let attest_endpoint = format!("{}/{KBS_PREFIX}/attest", self.kbs_host_url);
let attest = Attestation {
tee_pubkey,
tee_evidence: evidence,
};
debug!("send attest request.");
let attest_response = self
.http_client
.post(attest_endpoint)
.header("Content-Type", "application/json")
.json(&attest)
.send()
.await?;
match attest_response.status() {
reqwest::StatusCode::OK => {
let resp = attest_response.json::<AttestationResponseData>().await?;
let token = Token::new(resp.token)?;
self.token = Some(token);
}
reqwest::StatusCode::UNAUTHORIZED => {
let error_info = attest_response.json::<ErrorInformation>().await?;
bail!("KBS attest unauthorized, Error Info: {:?}", error_info);
}
_ => {
bail!(
"KBS Server Internal Failed, Response: {:?}",
attest_response.text().await?
);
}
}
Ok(())
}generate_evidence function:
The generate_evidence function uses the public key and nonce as runtime_data to generate the evidence using TEE-specific implementation.
- Sends evidence to KBS attest endpoint (
/attest) - Receives token on success.
You can see the token format here.
This function calls the TEE-specific get_evidence function.
async fn generate_evidence(
&self,
tee: Tee,
runtime_data: String,
nonce: String,
) -> Result<String> {
debug!("Challenge nonce: {nonce}");
let mut hasher = Sha384::new();
hasher.update(runtime_data);
let ehd = match tee {
// IBM SE uses nonce as runtime_data to pass attestation_request
Tee::Se => nonce.into_bytes(),
_ => hasher.finalize().to_vec(),
};
let tee_evidence = self
.provider
.get_evidence(ehd)
.await
.context("Get TEE evidence failed")
.map_err(|e| Error::GetEvidence(e.to_string()))?;
Ok(tee_evidence)
}
}The Attester interface describes the TEE-specific get_evidence implementations, and all supported Attesters (e.g., Azure vTPMs for TDX, SNP, AMD SEV-SNP) implement it.
pub trait Attester {
/// Call the hardware driver to get the Hardware specific evidence.
/// The parameter `report_data` will be used as the user input of the
/// evidence to avoid reply attack.
async fn get_evidence(&self, report_data: Vec<u8>) -> Result<String>;
}Example Attester Implementation for Azure CVM:
impl Attester for AzSnpVtpmAttester {
async fn get_evidence(&self, report_data: Vec<u8>) -> anyhow::Result<String> {
let report = vtpm::get_report()?;
let quote = vtpm::get_quote(&report_data)?;
let certs = imds::get_certs()?;
let vcek = certs.vcek;
let evidence = Evidence {
quote,
report,
vcek,
};
Ok(serde_json::to_string(&evidence)?)
}
}get_resource function:
The resource is retrieved from KBS using the get_resource method.
The client sends the request to the /resource endpoint of KBS with the path to the resource.
async fn get_resource(&mut self, resource_uri: ResourceUri) -> Result<Vec<u8>> {
let remote_url = format!(
"{}/{KBS_PREFIX}/resource/{}/{}/{}",
self.kbs_host_url, resource_uri.repository, resource_uri.r#type, resource_uri.tag
);
for attempt in 1..=KBS_GET_RESOURCE_MAX_ATTEMPT {
debug!("KBS client: trying to request KBS, attempt {attempt}");
let res = self
.http_client
.get(&remote_url)
.send()
.await
.map_err(|e| Error::HttpError(format!("get failed: {e}")))?;
match res.status() {
reqwest::StatusCode::OK => {
let response = res
.json::<Response>()
.await
.map_err(|e| Error::KbsResponseDeserializationFailed(e.to_string()))?;
let payload_data = self
.tee_key
.decrypt_response(response)
.map_err(|e| Error::DecryptResponseFailed(e.to_string()))?;
return Ok(payload_data);
}
// ...
}
}
}This completes a basic overview of the client-side implementation available in the attestation-agent component.
Now, let’s focus on the server-side implementation. The server-side is implemented primarily in the Key Broker Service (KBS) and Attestation Service (AS) components.
Code Location for the server-side components:
Repository: trustee
Two key folders: kbs and attestation-service
Let’s understand what happens in the KBS when the AA sends a request to the /attest endpoint. The payload for the /attest call is below.
[2023-11-09T04:48:23Z INFO api_server::http::attest]
Cookie 003ee77cfa3445e6909c99af028cd579 attestation
Json(Attestation { tee_pubkey: TeePubKey
{ kty: "RSA", alg: "RSA1_5",
k_mod: "xbNB2aciq-oSzpCoaHL2RQjY9sjM4hU_qyss6TSvEuaa6cHrgPQWqTQgJM4JuoXiw6gy5_Mw_ehvtqlfG42ABnL2Y1nQbNih1PQTw09xI5OOXyKaYX1IBbHDBNjIZvKcP8hgbp2bjff8arUTUKqst16KyrVc1G95YEx_reJuZS2V8GbSwzAtKGP8c99VvX12Su8r5t_CQN8a-QmflLHQIjuGFYEWixwH7b55-PUgszyXuPkxZR0jj5vA3Y0griXwg5SvWx1wEq_VQvSYmiVw3XH-50B-OoI-sZCRmBvz1JlgseYmCLNn3WI8rPUJuFJFVIq35Qo6dJBOypTfmXYtBw",
k_exp: "AQAB" }, tee_evidence: "{\"quote\":{\"signature\"::[...],\"message\": [...]},
\"report\":[...],\"vcek\":
\"-----BEGIN CERTIFICATE-----\\nMIIFTDCCAvugAwIBAgIBADBGBgkqhkiG9w0BAQowOaAPMA0GCWCGSAFlAwQCAgUA\noRwwGgYJKoZIhvcNAQEIMA0GCWCGSAFlAwQCAgUAogMCATCjAwIBATB7MRQwEgYD\nVQQLDAtFbmdpbmVlcmluZzELMAkGA1UEBhMCVVMxFDASBgNVBAcMC1NhbnRhIENs\nYXJhMQswCQYDVQQIDAJDQTEfMB0GA1UECgwWQWR2YW5jZWQgTWljcm8gRGV2aWNl\nczESMBAGA1UEAwwJU0VWLU1pbGFuMB4XDTIzMDEyNDE4NDkyOVoXDTMwMDEyNDE4\nNDkyOVowejEUMBIGA1UECwwLRW5naW5lZXJpbmcxCzAJBgNVBAYTAlVTMRQwEgYD\nVQQHDAtTYW50YSBDbGFyYTELMAkGA1UECAwCQ0ExHzAdBgNVBAoMFkFkdmFuY2Vk\nIE1pY3JvIERldmljZXMxETAPBgNVBAMMCFNFVi1WQ0VLMHYwEAYHKoZIzj0CAQYF\nK4EEACIDYgAEytSBUTFSfAnP2J5GfA16VsrMVWKNCs0gU3ZYqtQ8736ZJLGYPLzR\nEGzWISSVSwaXAkXMCTq2VZG7r5N+RbJqYhhw7z45++McYnDOY8cMxrEs5SpkaxZU\nIw8jYJR+N4Tco4IBFjCCARIwEAYJKwYBBAGceAEBBAMCAQAwFwYJKwYBBAGceAEC\nBAoWCE1pbGFuLUIwMBEGCisGAQQBnHgBAwEEAwIBAzARBgorBgEEAZx4AQMCBAMC\nAQAwEQYKKwYBBAGceAEDBAQDAgEAMBEGCisGAQQBnHgBAwUEAwIBADARBgorBgEE\nAZx4AQMGBAMCAQAwEQYKKwYBBAGceAEDBwQDAgEAMBEGCisGAQQBnHgBAwMEAwIB\nCDARBgorBgEEAZx4AQMIBAMCAXMwTQYJKwYBBAGceAEEBEAjNxnBOis0UU9ZhJCP\nfsCozdX7ntTOCRmnaLLn71NzjLsh2nKRC/GCYnwAMhT2dD3/W3uA46SmV1OuB2PF\nNk2tMEYGCSqGSIb3DQEBCjA5oA8wDQYJYIZIAWUDBAICBQChHDAaBgkqhkiG9w0B\nAQgwDQYJYIZIAWUDBAICBQCiAwIBMKMDAgEBA4ICAQCS12h0aXUwUxi7Tl89RgAJ\nkqhdpGMiHn5ENWrBaYaXwZruwm62iCQZas+jCz6a7fhM1s21rQ3mjZcDKApr3GbM\nwnmIDXuEWr+6xPyEXjlSAyg0uCVA8/1qS1tjsp1gy91M8oSecddbbld9qwBqdzty\nDeXlQ7isPLs/rEllUBEr+Aks50v+GEqMdezrL4EETU1qfmnpQbZjXnzl+zmIIoSd\nQRjyMNQR5frxEmjGZlUYTkjwN5FNNinTrEsMse8pJPGGwgec++RSVNgJl2lBjkbn\nram3mdDcEgUsHzQVoQdRBoCOh5+mFC1A9ZAUca4HQGpmvrN125bvNwG2J8zvitpj\nw+SsuH/WA7869ANMHqWdHPoHgtQGEzFM10Gsz+U1MdM7m8KRtYSKiaUQSzPsCF8x\nQLJvF4cCiFIoXMJvQ2YBfEqzY5qE9o/4GDza+bIfLqQSxGqlNONkTFkoiDf4IEDK\nfJJOdnJrfLM0MbVFSUf8Pu5+8cZRUrsK1Xd/iBwUZldzy9RevEhahaXy4bPwcz9v\n3u3Ae/gpsrxPQXpN4hmz74ZeNOORBgetlvKHmvPdKJHqL8Ys3tAgCkd8BHzT9DGY\naAIvit+vbLccKHRFGKIQPAzNrKBbXlJj5QC/SvtoV2vRQjD1EdVWzvFn3zVraYiY\\nrE1N4Gvw8+gFwhRn8LskLQ==\\n-----END CERTIFICATE-----\\n\"}KBS sends the evidence (tee_evidence) to the Attestation Service (AS) for evaluation.
The AttestationService defines the code for evaluating attestation evidence. Upon success, it returns the attestation token to the KBS, which then sends it to the AA.
/// Evaluate Attestation Evidence.
/// Issue an attestation results token which contains TCB status and TEE public key. Input parameters:
/// - `evidence`: TEE evidence bytes. This might not be the raw hardware evidence bytes. Definitions
/// are in the `verifier` crate.
/// - `tee`: concrete TEE type
/// - `runtime_data`: These data fields will be used to check against the counterpart inside the evidence.
/// The concrete way of checking is decided by the enum type. If this parameter is set to `None`, the comparison
/// will not be performed.
/// - `init_data`: These data fields will be used to check against the counterpart inside the evidence.
/// The concrete way of checking is decided by the enum type. If this parameter is set to `None`, the comparison
/// will not be performed.
/// - `hash_algorithm`: The hash algorithm that is used to calculate the digest of `runtime_data` and `init_data`.
/// - `policy_ids`: The policy IDs that are used to check this evidence. Any check that fails against a policy will
/// not cause this function to return an error. The result check against every policy will be included inside
/// the final Token returned by CoCo-AS.
#[allow(clippy::too_many_arguments)]
pub async fn evaluate(
&self,
evidence: Vec<u8>,
tee: Tee,
runtime_data: Option<Data>,
runtime_data_hash_algorithm: HashAlgorithm,
init_data: Option<Data>,
init_data_hash_algorithm: HashAlgorithm,
policy_ids: Vec<String>,
) -> Result<String> {
// ...
let claims_from_tee_evidence = verifier
.evaluate(&evidence, &report_data, &init_data_hash)
.await
.map_err(|e| anyhow!("Verifier evaluate failed: {e:?}"))?;
info!("{:?} Verifier/endorsement check passed.", tee);
let flattened_claims = flatten_claims(tee, &claims_from_tee_evidence)?;
debug!("flattened_claims: {:#?}", flattened_claims);
let tcb_json = serde_json::to_string(&flattened_claims)?;
let reference_data_map = self
.get_reference_data(flattened_claims.keys())
.await
.map_err(|e| anyhow!("Generate reference data failed: {:?}", e))?;
debug!("reference_data_map: {:#?}", reference_data_map);
let evaluation_report = self
.policy_engine
.evaluate(reference_data_map.clone(), tcb_json, policy_ids.clone())
.await
.map_err(|e| anyhow!("Policy Engine evaluation failed: {e}"))?;
info!("Policy check passed.");
let policies: Vec<_> = evaluation_report
.into_iter()
.map(|(k, v)| {
json!({
"policy-id": k,
"policy-hash": v,
})
})
.collect();
let reference_data_map: HashMap<String, Vec<String>> = reference_data_map
.into_iter()
.filter(|it| !it.1.is_empty())
.collect();
let token_claims = json!({
"tee": to_variant_name(&tee)?,
"evaluation-reports": policies,
"tcb-status": flattened_claims,
"reference-data": reference_data_map,
"customized_claims": {
"init_data": init_data_claims,
"runtime_data": runtime_data_claims,
},
});
let attestation_results_token = self.token_broker.issue(token_claims)?;
info!(
"Attestation Token ({}) generated.",
self._config.attestation_token_broker
);
Ok(attestation_results_token)
}This function calls the TEE-specific evidence verifier implementation to verify the evidence’s format, signature, etc. Then, it evaluates the claims presented in the evidence using the policy engine and reference value provider service (RVPS).
Here is an example of the Azure SNP evidence verifier code.
impl Verifier for AzSnpVtpm {
/// The following verification steps are performed:
/// 1. TPM Quote has been signed by AK included in the HCL variable data
/// 2. Attestation report_data matches TPM Quote nonce
/// 3. TPM PCRs' digest matches the digest in the Quote
/// 4. SNP report's report_data field matches hashed HCL variable data
/// 5. SNP Report is genuine
/// 6. SNP Report has been issued in VMPL 0
async fn evaluate(
&self,
evidence: &[u8],
expected_report_data: &ReportData,
expected_init_data_hash: &InitDataHash,
) -> Result<TeeEvidenceParsedClaim> {
let ReportData::Value(expected_report_data) = expected_report_data else {
bail!("unexpected empty report data");
};
if let InitDataHash::Value(_) = expected_init_data_hash {
warn!("Azure SNP vTPM verifier does not support verify init data hash, will ignore the input `init_data_hash`.");
}
let evidence = serde_json::from_slice::<Evidence>(evidence)
.context("Failed to deserialize Azure vTPM SEV-SNP evidence")?;
let hcl_report = HclReport::new(evidence.report)?;
verify_signature(&evidence.quote, &hcl_report)?;
verify_nonce(&evidence.quote, expected_report_data)?;
verify_pcrs(&evidence.quote)?;
let var_data_hash = hcl_report.var_data_sha256();
let snp_report = hcl_report.try_into()?;
verify_report_data(&var_data_hash, &snp_report)?;
let vcek = Vcek::from_pem(&evidence.vcek)?;
verify_snp_report(&snp_report, &vcek, &self.vendor_certs)?;
let mut claim = parse_tee_evidence(&snp_report);
extend_claim_with_tpm_quote(&mut claim, &evidence.quote)?;
Ok(claim)
}
}After the TEE-specific verifier successfully evaluates the evidence, the claims are returned to the caller, who retrieves the reference values from RVPS and then evaluates the claims against the policies.
async fn evaluate(
&self,
reference_data_map: HashMap<String, Vec<String>>,
input: String,
policy_ids: Vec<String>,
) -> Result<HashMap<String, PolicyDigest>, RegoError> {
let mut res = HashMap::new();
let policy_dir_path = self
.policy_dir_path
.to_str()
.ok_or_else(|| RegoError::PolicyDirPathToStringFailed)?;
for policy_id in &policy_ids {
let input = input.clone();
let policy_file_path = format!("{policy_dir_path}/{policy_id}.rego");
let policy = tokio::fs::read_to_string(policy_file_path.clone())
.await
.map_err(RegoError::ReadPolicyFileFailed)?;
let mut engine = regorus::Engine::new();
let policy_hash = {
use sha2::Digest;
let mut hasher = sha2::Sha384::new();
hasher.update(&policy);
let hex = hasher.finalize().to_vec();
hex::encode(hex)
};
// Add policy as data
engine
.add_policy(policy_id.clone(), policy)
.map_err(RegoError::LoadPolicyFailed)?;
let reference_data_map = serde_json::to_string(&reference_data_map)?;
let reference_data_map =
regorus::Value::from_json_str(&format!("{{\"reference\":{reference_data_map}}}"))
.map_err(RegoError::JsonSerializationFailed)?;
engine
.add_data(reference_data_map)
.map_err(RegoError::LoadReferenceDataFailed)?;
// Add TCB claims as input
engine
.set_input_json(&input)
.context("set input")
.map_err(RegoError::SetInputDataFailed)?;
let allow = engine
.eval_bool_query("data.policy.allow".to_string(), false)
.map_err(RegoError::EvalPolicyFailed)?;
if !allow {
return Err(RegoError::PolicyDenied {
policy_id: policy_id.clone(),
});
}
res.insert(policy_id.clone(), policy_hash);
}
Ok(res)
}Here is a flow diagram that you can use as a ready reference. Note that the PolicyEngine shown here is for the AS. There is also a policy engine for the KBS which I have left out for clarity.
Summary
In this article, we looked at the implementation aspects of the attestation process in the Trustee project. We explored the get_token and get_resource functions, which are pivotal in initiating the request-challenge-attestation-response (RCAR) handshake and requesting resources from the Key Broker Service (KBS).
We also examined the role of the Attestation Service (AS) in evaluating evidence and issuing attestation result tokens. The process involves multiple steps, including evidence verification by TEE-specific verifiers, obtaining reference data, and assessing claims against policies using the Policy Engine.
This blog has been lengthy, but hopefully, it will help you to understand the attestation implementation and pave the way for contributions :-).
Thanks and gratitude to Tobin and Magnus for helping me with reviews, answering code-related questions, and helping me understand the implementation details of remote attestation.
