Atmega 2560 UART serial data limit

Question

Hello I am working atmega 2560 with SIM800L. I require to extract data of 100KB from server but i am able achieve only 5KB (RAM constraints) but I have SD card to store the data. I am trying to implement it using circular buffer. Can anyone guide me through the process?

I tried concept of circular buffer but i am unable to receive data when buffer size is less actual server data

I want to extract chunks of data and store it in SD card

Answer 1

The issue is that you need to be able to receive and buffer incoming data whilst the filesystem is busy writing to the SD card.

For that to happen and assuming that you are not using an RTOS (and probably even if you are using an RTOS), your serial input will need to be interrupt driven where the UART Rx interrupt handler places data into a FIFO buffer (normally implemented as a circular buffer as you suggest). The size of the buffer will need to be sufficient to store as many characters as could be received for the duration the file system is busy dealing with the SD card write. Bearing in mind that SD card latency is likely to be variable and non-deterministic depending on physical constraints of the card. For example it is usual for a card to "stall" for a few milliseconds when writing across specific physical boundaries, even when otherwise data can be written very fast.

In this case the serial data stream rate is likely to be low compared to the SD card latency, which means that you can likely get away with a small buffer. For example at 9600,N,8,1, a buffer of 128 bytes will allow for 133ms of filesystem latency. That is probably sufficient for any SD card you are likely to use.

Filesystems work most efficiently when presented with data matching the logical sector size. Typically for a small filesystem 512 byte buffers are used. So your software should read from the serial FIFO until it has collated a 512 byte block, then write that to the SD card. While that is happening, the interrupt driven serial driver can concurrently place data into the FIFO (for up to 133ms in the example given above), then when you return to collating a new block, the code will immediately transfer the buffered data to the collating block and continue collecting further data in real-time (i.e. as it arrives) until the block is again filled.

An alternative, though less generalised solution (i.e. it is specific to this particular task) would be to use double buffering, where the UART Rx interrupt handler writes to one of two 512 byte buffers, and when a buffer is filled, sets a flag polled by the main thread that triggers a write of that buffer to the file system. Meanwhile the interrupt handler switches to the alternate buffer and fills that while the SD card is being written to. The disadvantage of that approach is that it requires two relatively large buffers, and does not easily allow other processing you might want to perform on the incoming serial data other than simply logging it.

If you need help specifically with buffered interrupt driven serial I/O, that probably deserves a new and more specific question.