I do not have too much experience with these ring buffers, these are just my ideas after 1year of kernel hobby os work.
1. I cannot really help with this.
Probably it is worth for very specific situations, i do not know what exactly your needs there.
But i would guess that these syscalls are heavily IO specific so, in my OS that is not such a trouble haha.
2.
Regarding the size of the packages: I think when you initialise your queue from the user-space side,
you could have a parameter to define the request/response package sizes?
In this case the caller needs to know the max size they are willing.
Packing:
The nicest would be (IMO) if all syscall has their own struct, so you could just memcpy their content into the ring buffer.
Also you can go with the max param count yes,
Or also just store the number of parameters in a numeric field,
then with pointer-magic you just read it from-to the correct region.
struct {
int param_count;
int params[];
// and the content can even contain the length of the param and then the actual param's value,
// so you can even put an integer field, then a char array, in a single package
}
3. You need to validate them manually based on the given syscall that was requested.
Whether they have the correct value range, modes, etc.
Validate the output buffer, whether the kernel has correct read/write access to it, whether it is long enough to read/write the requested amount of data from/into it.
The same if your parameter is being a pointer to a struct,you need to follow the pointer and check that struct for validity.
I hope you can find some useful stuff in it