From bbad9f255d76cf7dcd11d15800e508d9849826af Mon Sep 17 00:00:00 2001 From: Nelson Billing Date: Wed, 19 Feb 2020 15:32:08 -0800 Subject: Add optional new symbol upload API to sym_upload. Change-Id: I6a49e9f4a699fa6f5f8e9f0fc86afb4cb342a442 Reviewed-on: https://chromium-review.googlesource.com/c/breakpad/breakpad/+/1422400 Reviewed-by: Mark Mentovai Reviewed-by: Ivan Penkov Reviewed-by: Mike Frysinger --- docs/sym_upload_v2_protocol.md | 214 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 214 insertions(+) create mode 100644 docs/sym_upload_v2_protocol.md (limited to 'docs') diff --git a/docs/sym_upload_v2_protocol.md b/docs/sym_upload_v2_protocol.md new file mode 100644 index 00000000..d56a4c83 --- /dev/null +++ b/docs/sym_upload_v2_protocol.md @@ -0,0 +1,214 @@ +# Introduction + +The `sym_upload` tool is able to operate in `sym-upload-v2` protocol mode, in +addition to the legacy protocol (which will be referred to as `sym-upload-v1` +for the rest of this document). For now `sym-upload-v2` is HTTP/REST-based but +it could be extended to operate over gRPC instead, in the future. + +# Table of Contents +* [Why](#why) +* [How](#how) + * [Uploading](#uploading) + * [Uploading with `sym_upload`](#uploading-with-sym_upload) + * [Uploading with curl](#uploading-with-curl) + * [Serving the `sym-upload-v2` protocol](#serving-the-sym-upload-v2-protocol) + * [Authenticate using `key`](#authenticate-using-key) + * [Symbol `checkStatus`](#symbol-checkstatus) + * [Upload `create`](#upload-create) + * [Uploading the symbol file](#uploading-the-symbol-file) + * [Upload complete](#upload-complete) + + +# Why + +Using `sym_upload` in `sym-upload-v2` protocol mode has the following features +beyond `sym-upload-v1`: + * Authentication via `key` (arbitrary secret). + * Symbol identifier (product of `debug_file` and `debug_id`, as recorded in +output from `dump_syms`) can be checked against existing symbol information on +server. If it's present, then the upload is skipped entirely. + +# How + +## Uploading + +### Uploading with `sym_upload` + +Uploading in `sym-upload-v2` protocol mode is easy. Invoke `sym_upload` like +``` +$ ./sym_upload -p sym-upload-v2 [-k ] +``` + +Where `symbol-file` is a symbol file created by `dump_syms`, `API-URL` is the +URL of your `sym-upload-v2` API service (see next section for details), and +`API-key` is a secret known to your uploader and server. + +For more options see `sym_upload --help`. + +### Uploading with curl + +As an example, if: + * Your API's URL was "https://sym-upload-api". + * Your service has assigned you `key` "myfancysecret123". + * You wanted to upload the symbol file at "path/to/file_name", with +`debug_file` being "file_name" and `debug_id` being +"123123123123123123123123123". Normally you would read these values from +"path/to/file_name", which in turn was generated by `dump_syms`. + +Then you might run: +``` +$ curl https://sym-upload-api/symbols/file_name/123123123123123123123123123:checkStatus?key=myfancysecret123 +``` + +And, upon seeing that this `debug_file`/`debug_id` combo is missing from symbol +storage then you could run: +``` +$ curl --request POST https://sym-upload-api/uploads:create?key=myfancysecret123 +``` + +Which returns `upload_url` "https://upload-server/42?creds=shhhhh" and +`upload_key` "42". Next you upload the file directly like: +``` +$ curl -T path/to/file_name "https://upload-server/42?creds=shhhhh" +``` + +Once the HTTP PUT is complete, run: +``` +$ curl --header "Content-Type: application/json" \ + --request POST \ + --data '{symbol_id:{"debugFile":"file_name",'\ + '"debugId":"123123123123123123123123123"}}' \ + https://sym-upload-api/uploads/42:complete?key=myfancysecret123 +``` + +### Serving the `sym-upload-v2` Protocol + +The protocol is currently defined only in HTTP/REST. There are three necessary +REST operations to implement in your service: +* `/symbols//:checkStatus?key=` +* `/uploads:create?key=` +* `/uploads/:complete?key=` + +#### Authenticate Using `key` + +The query string arg `key` contains some secret that both the uploader and +server understand. It is up to the service implementer to decide on what +constitutes a valid `key`, how the uploader acquires one, and how to handle +requests made with invalid ones. + +#### Symbol `checkStatus` + +``` +/symbols//:checkStatus?key= +``` + +This operation expects an empty (or no) JSON payload in the request. + +This operation should return the status of the symbol file uniquely identified +by the given `debug_file` and `debug_id`. JSON schema: +``` +{ + "type": object", + "properties": { + "status": { + "type": "string", + "enum": ["STATUS_UNSPECIFIED", "MISING", "FOUND"], + "required": true + } + } +} +``` + +Where `MISSING` denotes that the symbol file does not exist on the server and +`FOUND` denotes that the symbol file exists on the server. + +#### Upload `create` + +``` +/uploads:create?key= +``` + +This operation expects an empty (or no) JSON payload in the request. + +This operation should return a URL that uploader can HTTP PUT their symbol file +to, along with an "upload key" that can be used to notify the service once the +file upload is completed. JSON schema: +``` +{ + "type": "object", + "properties": { + "upload_url": { + "type: "string", + "required": true + }, + "upload_key": { + "type": "string", + "required": true + } + } +} +``` + +Since this REST API operation can be authenticated via the `key` query string +arg, the service can return a URL that encodes permission delegation to the +upload endpoint resource and thereby constrain the ability to upload to those +with valid `key`s. + +#### Uploading the Symbol File + +Note that the actual symbol upload step is _not_ part of the REST API. The +upload URL obtained in the above operation is meant to be used as the endpoint +for a normal HTTP PUT request for the contents of the symbol file. Once that +HTTP PUT request is completed use the upload `complete` operation. + +#### Upload `complete` + +``` +/uploads/:complete?key= +``` + +This operation expects a JSON payload in the HTTP request body with the +following schema: +``` +{ + "type": "object", + "properties": { + "symbol_id": { + "type": "object", + "properties": { + "debug_file": { + "type": "string", + "required": true + }, + "debug_id": { + "type": "string", + "required": true + } + } + } + } +} +``` + +This operation should cause the symbol storage back-end (however implemented) +to consume the symbol file identified by `upload_key`. It is up to the service +implementation to decide how uploads are assigned `upload_key`s and how to +retrieve a completed upload by its `upload_key`. If the symbol file cannot be +found, is malformed, or the operation cannot be completed for any other reason +then an HTTP error will be returned. JSON schema of non-error responses: +``` +{ + "type": "object", + "properties": { + "result": { + "type": string, + "enum": ["RESULT_UNSPECIFIED", "OK", "DUPLICATE_DATA"], + "required": true + } + } +} +``` + +Where `OK` denotes that the symbol storage was updated with the new symbol file +and `DUPLICATE_DATA` denotes that the symbol file data was identical to data +already in symbol storage and therefore nothing changed. -- cgit v1.2.1