C++ question!
I have a class that produces gigabytes of data per second by emitting it character by character due to the structure of the VCD format. I want to let consumers of that API stream data to: file, (shared) memory, socket, ...
How? I have a few options. 1/3
Conversation
1. take std::ostream&. pro: it's standard, i guess. con: have you seen iostream? con: wchar_t ofstream filenames subtly broken in some configurations con: hard to expose for FFI
2. take FILE*. pro: everyone knows that. con: no RAII. con: windows has no funopen() or fmemopen() 2/3
1
12
3. have an internal std::string, caller responsible for flushing buffer. pro: obvious. pro: works even on bare metal. pro: easy to expose via FFI. con: extra copy. con: caller has to care about spooling explicitly
Opinions?
8
12
looks like std::string is the only workable option, thanks everyone! this is what I already implemented but I wanted to see if I'm missing something
3
10
Replying to
Use an internal std::string to implement a ring buffer and provide an API producing std::string_view slices. Alternatively, use std::vector<char> and std::span<char>. Don't expose the internal collection type. You can convert to pointer + length slices for FFI.
1
Replying to
You could use a pair of iterators to represent the slice, which is closer to how FFI has to do it anyway.
1
Replying to
right. but then the problem becomes sizing the ring buffer. the caller does not have enough information to make a well informed choice, neither does the callee
1
Replying to
It can produce in between 1 and N bytes of output until it reaches the end of the data though, right? So the caller's choice of size is a latency/memory vs. throughput choice, with diminishing returns for doing larger chunks at the same time.
1
You can make an API like read(stream) where it returns a slice of at least 1 byte and finally signals that it has come to an end. Choose some arbitrary buffer size and let the caller override it when they create the stream. If you really wanted you could do it in parallel.
Replying to
hmm. I kind of like the self-sizedness of std::string (the underlying buffer being as large or as small as your flush frequency) but this could make sense
1
Replying to
If you were going to use an interface like a C++ stream, I think you should *provide* one rather than taking one as an argument. It lets you decide to do it in a thread or even a separate process with the buffer in shared memory or even just an OS pipe.
1
Show replies