Test structerr metadata via interceptors
structerr: Add interceptors to inject metadata as error details
It is quite hard right now to properly test whether metadata is injected as expected as it is only used for logging purposes. This is further complicated by the fact that by default, the metadata is only attached to log messages that are dispatched asynchronously after we have already returned the error to the client. This causes flakiness in one of our tests.
Metadata is really not supposed to be seen by clients at all: they are intended only for logging purposes and are thus not stable information that any client may rely on. So it is not immediately obvious how we can remedy that situation. There are multiple alternatives:
- Don't test error metadata at all when it crosses the gRPC
boundary. This is the easiest to implement, but it may make it
hard to see what's going on.
- Provide a mechanism to make the logging synchronous. We cannot do
this in production code as we rely on the `grpc/stats.End` message
to obtain information on how many bytes we have sent and received,
and that can only ever be available after we have sent the final
error.
- Convert error metadata into error details. This may cause clients
to start parsing them though and thus rely on implemenation
details that are subject to change.
This commit opts for the last option, but makes the logic conditional so that we only convert error metadata when using our test server setup. It has the significant downside that errors are now different depending on whether we are testing or not. But the interceptors really only add additional information, which should hopefully be fine.