http_request with custom stream

Aug 12, 2014 at 2:23 PM
Is it possible to implement a custom stream for http_request that doesn't require all of the bytes of the body to be in memory?

The server I'm talking to requires a MIME wrapper around the file I want to send in the body. From reading the documentation, it seems like my only options are
  1. read in the entire file into a std::string, and add MIME prefix and suffix contents (resource hog for large files, potential out of memory).
  2. rewrite the file into a temporary file that has the MIME prefix and suffix contents (potential out of disk space for large files, can be a performance killer as well).
I'd prefer to leave my file alone on disk, and have a custom stream that would dynamically add the MIME prefix and suffix contents (headers, delimiters, etc.) on the fly.

Is this possible? Is there any example code anywhere?

( Apologies if this is documented or discussed elsewhere - I couldn't find it. )
Aug 13, 2014 at 6:10 PM
Hi kkostrzewa,

No apologies necessary this is a good discussion question :)

In addition to the options you mention here are a few more that I could think of:
  1. If it is safe for you to edit the original file that you want to wrap what you could do is open a stream for writing to the file. Then first write the MIME prefix at the beginning, seek to the end, and then write the MIME suffix. Then close the file and re-open for reading passing the stream in as the request body. This approach has several positive characteristics, first there will be no reading or copying of the file data since it will only be read into memory when temporarily needed to transfer over the wire. Second there is no potential for out of memory because only one chunk of the file will be read into memory at a time, using our file_buffer/file_stream (cpprest\filestream.h). The one drawback is you would have to modify the file on disk, not sure if this is OK for your situation.
  2. Another potential option is to implement a stream that works by first providing the MIME prefix, then reading in the file, and finally providing the MIME suffix. This probably could be built reusing our file_buffer/file_stream classes but would be more work and require learning and understanding about how our streams are implemented. To do this I'd take a look at the cpprest\filestream.h, cpprest\astreambuf.h, and streams.h source files.
Steve
Aug 13, 2014 at 7:00 PM
Edited Aug 13, 2014 at 7:01 PM
Thanks for the quick reply. I'm probably going to go with the 2nd approach. I'm trying to upload a PDF to a web service from an exe, and the library that I'm using generates the PDF on disk. To go with approach #1 means that I have to rewrite the entire PDF with the MIME prefix/suffix because you can only append to a file.

The second approach will also allow me to base64 encode (or other kind of transform) the underlying file, per the web service requirements.

I would love to see some sample code beyond reading through filestream.h and astreambuf.h if anybody has it.
Aug 14, 2014 at 5:59 PM
Hi kkostrzewa,

Maybe I'm missing something, but why can you only append to a file with approach 1? If you have to do base64 encoding as well then yes I see that this approach won't work.

The only examples we have of implementing a stream are in the actual product but there are several different ones. You can take a look at the following:

Producer Consumer (cpprest\producerconsumerstream.h): This is a memory-based stream buffer that supports both reading and writing a sequence of bytes. A single consumer and producer pair can communicate via the buffer from different threads.
File Streams (cpprest\filestream.h) : Supports input and output operations for files.
Container Streams (cpprest\containerstream.h): This is a stream that is backed by a basic STL container. It is a memory-based stream that supports both reading and writing.
Raw Pointer Streams (cpprest\rawptrstream.h) : A stream buffer that is based on a raw pointer and block size. The buffer cannot be expanded or contracted, it has a fixed capacity. This stream buffer class is used to create a memory backed stream that supports both reading and writing sequence of characters from a fixed block size.
Interop Streams (cpprest\interopstream.h) : Supports interop between STL iostreams or WinRT streams and Casablanca async streams.

For more examples of using streams you can take a look at some of the test cases located under Release\tests\Functional\streams.

Steve
Aug 14, 2014 at 7:08 PM
Thanks a ton for the examples in the product. That'll make writing this a lot easier.

When I said "only append", I'm not aware of a CreateFile flag that allows me to insert into the middle (or at the beginning) of a file. I can only overwrite.

For example, if a 4 byte file on disk has
byte[0] = 'A'
byte[1] = 'B'
byte[2] = 'C'
byte[3] = 'D'

and I open the file for writing, seek to position 0, and then write out 'x', the file looks like this
byte[0] = 'x'
byte[1] = 'B'
byte[2] = 'C'
byte[3] = 'D'

When I need to prepend data to a file, I have to account for the overwrite by looping through the file (backwards), reading a chunk at the old position and writing it at the new position. Then I've got "free space" at the beginning of the file to overwrite.

If I'm wrong about that, and there is a way to insert at the beginning of a file using the Win32 API, then please tell me! I've been following this file pattern in my code for years.
  • Kevin
Aug 15, 2014 at 5:40 PM
Hi Kevin,

You can read or write to a file at any position by specifying an offset through and OVERLAPPED structure to APIs like ReadFile and WriteFile. This is what we do in our implementation.

Another option is you can just use the file buffer/stream provided by the C++ Rest SDK, it supports seeking. For examples search for 'seek' in the test cases in Release\tests\Functional\streams\fstreambuf_tests.cpp.

Steve