Add optional new symbol upload API to sym_upload.

Change-Id: I6a49e9f4a699fa6f5f8e9f0fc86afb4cb342a442
Reviewed-on: https://chromium-review.googlesource.com/c/breakpad/breakpad/+/1422400
Reviewed-by: Mark Mentovai <mark@chromium.org>
Reviewed-by: Ivan Penkov <ivanpe@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

1 files changed, 214 insertions, 0 deletions

diff --git a/docs/sym_upload_v2_protocol.md b/docs/sym_upload_v2_protocol.md
new file mode 100644
index 00000000..d56a4c83
--- /dev/null
+++ b/docs/sym_upload_v2_protocol.md
@@ -0,0 +1,214 @@
+# Introduction
+
+The `sym_upload` tool is able to operate in `sym-upload-v2` protocol mode, in
+addition to the legacy protocol (which will be referred to as `sym-upload-v1`
+for the rest of this document). For now `sym-upload-v2` is HTTP/REST-based but
+it could be extended to operate over gRPC instead, in the future.
+
+# Table of Contents
+* [Why](#why)
+* [How](#how)
+   * [Uploading](#uploading)
+     * [Uploading with `sym_upload`](#uploading-with-sym_upload)
+     * [Uploading with curl](#uploading-with-curl)
+     * [Serving the `sym-upload-v2` protocol](#serving-the-sym-upload-v2-protocol)
+       * [Authenticate using `key`](#authenticate-using-key)
+       * [Symbol `checkStatus`](#symbol-checkstatus)
+       * [Upload `create`](#upload-create)
+       * [Uploading the symbol file](#uploading-the-symbol-file)
+       * [Upload complete](#upload-complete)
+
+
+# Why
+
+Using `sym_upload` in `sym-upload-v2` protocol mode has the following features
+beyond `sym-upload-v1`:
+  * Authentication via `key` (arbitrary secret).
+  * Symbol identifier (product of `debug_file` and `debug_id`, as recorded in
+output from `dump_syms`) can be checked against existing symbol information on
+server. If it's present, then the upload is skipped entirely.
+
+# How
+
+## Uploading
+
+### Uploading with `sym_upload`
+
+Uploading in `sym-upload-v2` protocol mode is easy. Invoke `sym_upload` like
+```
+$ ./sym_upload -p sym-upload-v2 [-k <API-key>] <symbol-file> <API-URL>
+```
+
+Where `symbol-file` is a symbol file created by `dump_syms`, `API-URL` is the
+URL of your `sym-upload-v2` API service (see next section for details), and
+`API-key` is a secret known to your uploader and server.
+
+For more options see `sym_upload --help`.
+
+### Uploading with curl
+
+As an example, if:
+  * Your API's URL was "https://sym-upload-api".
+  * Your service has assigned you `key` "myfancysecret123".
+  * You wanted to upload the symbol file at "path/to/file_name", with
+`debug_file` being "file_name" and `debug_id` being
+"123123123123123123123123123". Normally you would read these values from
+"path/to/file_name", which in turn was generated by `dump_syms`.
+
+Then you might run:
+```
+$ curl https://sym-upload-api/symbols/file_name/123123123123123123123123123:checkStatus?key=myfancysecret123
+```
+
+And, upon seeing that this `debug_file`/`debug_id` combo is missing from symbol
+storage then you could run:
+```
+$ curl --request POST https://sym-upload-api/uploads:create?key=myfancysecret123
+```
+
+Which returns `upload_url` "https://upload-server/42?creds=shhhhh" and
+`upload_key` "42". Next you upload the file directly like:
+```
+$ curl -T path/to/file_name "https://upload-server/42?creds=shhhhh"
+```
+
+Once the HTTP PUT is complete, run:
+```
+$ curl --header "Content-Type: application/json" \
+    --request POST \
+    --data '{symbol_id:{"debugFile":"file_name",'\
+        '"debugId":"123123123123123123123123123"}}' \
+    https://sym-upload-api/uploads/42:complete?key=myfancysecret123
+```
+
+### Serving the `sym-upload-v2` Protocol
+
+The protocol is currently defined only in HTTP/REST. There are three necessary
+REST operations to implement in your service:
+* `/symbols/<debug_file>/<debug_id>:checkStatus?key=<key>`
+* `/uploads:create?key=<key>`
+* `/uploads/<upload_key>:complete?key=<key>`
+
+#### Authenticate Using `key`
+
+The query string arg `key` contains some secret that both the uploader and
+server understand. It is up to the service implementer to decide on what
+constitutes a valid `key`, how the uploader acquires one, and how to handle
+requests made with invalid ones.
+
+#### Symbol `checkStatus`
+
+```
+/symbols/<debug_file>/<debug_id>:checkStatus?key=<key>
+```
+
+This operation expects an empty (or no) JSON payload in the request.
+
+This operation should return the status of the symbol file uniquely identified
+by the given `debug_file` and `debug_id`. JSON schema:
+```
+{
+    "type": object",
+    "properties": {
+        "status": {
+            "type": "string",
+            "enum": ["STATUS_UNSPECIFIED", "MISING", "FOUND"],
+            "required": true
+        }
+    }
+}
+```
+
+Where `MISSING` denotes that the symbol file does not exist on the server and
+`FOUND` denotes that the symbol file exists on the server.
+
+#### Upload `create`
+
+```
+/uploads:create?key=<key>
+```
+
+This operation expects an empty (or no) JSON payload in the request.
+
+This operation should return a URL that uploader can HTTP PUT their symbol file
+to, along with an "upload key" that can be used to notify the service once the
+file upload is completed. JSON schema:
+```
+{
+    "type": "object",
+    "properties": {
+        "upload_url": {
+            "type: "string",
+            "required": true
+        },
+        "upload_key": {
+            "type": "string",
+            "required": true
+        }
+    }
+}
+```
+
+Since this REST API operation can be authenticated via the `key` query string
+arg, the service can return a URL that encodes permission delegation to the
+upload endpoint resource and thereby constrain the ability to upload to those
+with valid `key`s.
+
+#### Uploading the Symbol File
+
+Note that the actual symbol upload step is _not_ part of the REST API. The
+upload URL obtained in the above operation is meant to be used as the endpoint
+for a normal HTTP PUT request for the contents of the symbol file. Once that
+HTTP PUT request is completed use the upload `complete` operation.
+
+#### Upload `complete`
+
+```
+/uploads/<upload_key>:complete?key=<key>
+```
+
+This operation expects a JSON payload in the HTTP request body with the
+following schema:
+```
+{
+    "type": "object",
+    "properties": {
+        "symbol_id": {
+            "type": "object",
+            "properties": {
+                "debug_file": {
+                    "type": "string",
+                    "required": true
+                },
+                "debug_id": {
+                    "type": "string",
+                    "required": true
+                }
+            }
+        }
+    }
+}
+```
+
+This operation should cause the symbol storage back-end (however implemented)
+to consume the symbol file identified by `upload_key`. It is up to the service
+implementation to decide how uploads are assigned `upload_key`s and how to
+retrieve a completed upload by its `upload_key`. If the symbol file cannot be
+found, is malformed, or the operation cannot be completed for any other reason
+then an HTTP error will be returned. JSON schema of non-error responses:
+```
+{
+    "type": "object",
+    "properties": {
+        "result": {
+            "type": string,
+            "enum": ["RESULT_UNSPECIFIED", "OK", "DUPLICATE_DATA"],
+            "required": true
+        }
+    }
+}
+```
+
+Where `OK` denotes that the symbol storage was updated with the new symbol file
+and `DUPLICATE_DATA` denotes that the symbol file data was identical to data
+already in symbol storage and therefore nothing changed.