Bad alloc exception when reading a stream (from a big file)

Jul 28, 2015 at 3:05 PM
Edited Jul 28, 2015 at 4:04 PM
Hi,
I have a basic_istream(fileStream1) which I get from a file_stream.I have a loop where I read this stream into a producer_consumer buffer(pcb) :
                   pcb.set_buffer_size(10485760, std::ios_base::out);//10mb
                    size_t count = 10485760;          //10 mb
        while (sizef2>0)
        {
            nread=fileStream1.read(pcb, count).get();
            sizef2 -= nread;
        }
if the file is 850 Mo it runs well but if the original file is 1,7Go I have a bad alloc exception when reading (when I have read about 900Mo). I don't know how to solve this problem...I have tryed other values for count but I have always this exception.I have 8Go Ram on my pc and a virtual memory between 8Go and 16 Go.Is there another way to do the same function without throwing this exception ?
Aug 18, 2015 at 11:55 AM
I had this exception because I was in 32 bits and there is less memory ,but is it possible to read the first stream with alloc,acquire and release to use less memory ? I didn't found examples how to use alloc,acquire and release to read a stream...
Coordinator
Aug 21, 2015 at 7:39 PM
Since it looks like you're only reading, you should only need to use acquire and release.

http://microsoft.github.io/cpprestsdk/class_concurrency_1_1streams_1_1streambuf.html#ad19730d8637ba56baf9283041ea68bc3
http://microsoft.github.io/cpprestsdk/class_concurrency_1_1streams_1_1streambuf.html#a4e1645fcdf1b616a795c55de338ec26b

For example usage, you can look at our tests inside Release\tests\functional\streams\memstream_tests.cpp.
Aug 24, 2015 at 9:30 AM
thank you ,I have looked at the exemple in memstream_tests.cpp but I haven't understand the whole code.

so I have tryed this :
streambuf<unsigned char> sb = fileStream1.streambuf();
sb.set_buffer_size(10485760, std::ios_base::in);//10mb
size_t count2 = 0;
unsigned char *ptr=0;
while (sizef2 > 0)
                {

                    
                    bool r = sb.can_read();
                    bool r2=sb.acquire(ptr, count2);
                    if (ptr)
                    {
                        pcb.putn_nocopy(ptr, count2).get();
                        sb.release(ptr, count2);
                    }

                    sizef2 -= count2;
                }
but ptr =0 and r2=false atfer the call to acquire, so I don't understand (although sb.can_read returns true).
Aug 24, 2015 at 9:49 AM
someone told me that my first code :
nread=fileStream1.read(pcb, count).get();
takes only 10 Mb . so if it takes only 10 Mb here I don't need to use acquire and release anymore
Aug 24, 2015 at 9:57 AM
No I have just tested now and the previous code takes 10 Mb more for each loop that's why it takes so much memory as the size of the file....
Aug 24, 2015 at 10:14 AM
I have thought that it is perhaps pcb (the buffer where I write) that takes 10 Mb more each time so I can perhaps do nothing against that and it is perhaps unuseless to use acquire and release to read the stream.
Coordinator
Aug 24, 2015 at 11:57 PM
What is your actual application here? The loop you have above will write the entire contents of the file into (what I assume is) an in-memory producer-consumer buffer. This will consume as much memory as the file size, since that's what you're asking the program to do.

Can you post more of your actual code, either inline or via pastebin/gist? In order to avoid the memory issue, you will need to rework your processing logic to work on a "streaming" basis.
Aug 25, 2015 at 4:16 PM
I upload a message with this file : so before my code fileStream1.read I have other code to write in pcb the beginning of the message and after fileSTream1.read I write the end of the message and after all I read the pcb in a basic_istream and I upload with chunk of 100 Mo.
the biggest part of the application runs I wanted to know only if I can spare memory.( I don't upload files with a size bigger than 2Gb) .
Coordinator
Aug 25, 2015 at 10:05 PM
Without seeing the full code, I don't think I can help any more than I've already done above. Sorry :(.
Aug 26, 2015 at 8:15 AM
Edited Aug 26, 2015 at 8:15 AM
below there is a big part of the code : (enough to understand)
file_stream<unsigned char>::open_istream(szpath).then([this, pathf, lsizef](task<basic_istream<unsigned char>> fileStream1task)
        {
            
            try
            {
                basic_istream<unsigned char>fileStream1 = fileStream1task.get();
                .......
                producer_consumer_buffer<unsigned char> pcb(1048576);//1mb
                pcb.set_buffer_size(10485760, std::ios_base::out);//10mb
                CString boundary("--toto");
                CString boundary2 = CString("--") + boundary + CString("\n");
                CString filepart("Content-Disposition: form-data; name=\"file\"; filename=\"");
                filepart += filename + CString("\"\n");
                filepart += CString("Content-Type: application/octet-stream\n\n");
                std::basic_string<unsigned char> Toboundary;
                std::basic_string<unsigned char> Toboundary2;
                std::basic_string<unsigned char> To;
                Str2Data(Toboundary, boundary2);
                
                pcb.putn_nocopy(Toboundary.c_str(), Toboundary.length()).get();
                
                sumsize += Toboundary.length();
                Str2Data(To, filepart);
                pcb.putn_nocopy(To.c_str(), To.length()).get();
                
                size_t count = 10485760;//10 mb
                size_t count2 = 0;
                unsigned char *ptr=0;

                size_t nread;
                size_t sizef2 = (size_t)lsizef;
                while (sizef2 > 0)
                {

                    
                    nread = fileStream1.read(pcb, count).get();

                    

                    
                    sizef2 -= nread;

                    
                }
                fileStream1.close();
                
                sumsize += lsizef;
                

                CString iscryptpart(_T("Content-Disposition: form-data; name=\"isCrypted\"\n\n"));
                CString isc(_T("false"));
                iscryptpart += isc;
                iscryptpart += CString(_T("\n"));

                To.clear();
                this->Str2Data(To, CString("\n"));
                pcb.putn_nocopy(To.c_str(), To.length()).get();
                //fileStream.print(To);
                sumsize += To.size();
                pcb.putn_nocopy(Toboundary.c_str(), Toboundary.length()).get();
                //fileStream.print(Toboundary);
                sumsize += Toboundary.size();
                To.clear();
                this->Str2Data(To, iscryptpart);
                pcb.putn_nocopy(To.c_str(), To.length()).get();
                
                sumsize += To.size();
                pcb.putn_nocopy(Toboundary2.c_str(), Toboundary2.length()).get();
                
                sumsize += Toboundary2.size();

                //fileStream.close();
                pcb.close(std::ios::out);
                pcb.set_buffer_size(104857600, std::ios_base::in);//100mb
                basic_istream<unsigned char> Stream(pcb);
                
                requete.headers().set_content_length(sumsize);
                requete.set_request_uri(u);
                requete.set_body(Stream);
                client.request(requete).then(....
                ....
                
lsizef is the size of the file.(fileStream1)
Aug 26, 2015 at 8:25 AM
I write data in pcb before and after writing fileSTream1 (the file).
after that, I put pcb in a basic_istream and I upload it.
Coordinator
Aug 27, 2015 at 11:00 PM
The way this code is structured does the following:
  • open a file (~0 memory)
  • create a pcb (~0 memory)
  • prep the pcb with some data (~small memory)
  • read the entire file into the pcb in a loop (=filesize memory)
  • close file (~0 memory)
  • send the pcb out in a request (~0 memory).
With this approach, you are reading the entire file into memory before starting the transfer (forcing us to buffer the whole thing). If you want to support large files, you need to change your approach to start the request immediately. This way the data doesn't have to be all kept in memory at the same time.

You will probably want to specify Transfer-Encoding: "chunked":
response.headers().add(header_names::transfer_encoding, U("chunked"));

Take a look at some of the tests in Release\tests\functional\http\client\response_stream_tests.cpp and Release\tests\functional\http\client\request_stream_tests.cpp for examples.
Aug 28, 2015 at 9:10 AM
I have tryed with read_to_end instead of my loop(fileStream1.read_to_end(pcb)) but if I don't put a get() to read_to_end, it doesn't run, so it takes so much memory with the get().
the problem is that I write other smalls datas into the pcb after writing the file into the pcb....SO I don't see how to procceed.and I have seen too that passing pcb or a string by reference to a lambda is not ok.(because I have tryed too :(fileStream1.read_to_end(pcb).then([this,&pcb,....])