Does there is an easier method for LIFO?

[rado99]

Hello,
I save logs in the SPIFFS via FILE_APPEND, but I want to visualise them LIFO by \n. What is the easiest method to do that, because I think the option to load the whole File in RAM and than to convert it is not a good idea.
Thank you in advance.

[1TekPro]

Only 2 things come to mind:
esp8266/Arduino#5511 might work for you. (easy method but might be bugged)
And
If you know what will be written, you could painstakingly create a virtual directory by creating pointer indexes each and every time something is to be written. (difficult method)

[rado99]

I think the second variant is good for me:
I have records from temperature meter, so I can do an folder with Unique ID every record. Although, it will help me with other functionality I planned - to delete the old records. But, does FS.h will not see a problem with a folder with 1000-2000 files minimum? It will something like a SQL table.

[1TekPro]

@rado99 If each chunk of data is written with append and then read, spiff size, max file size, file name length (including directory) and buffer might become an issue. Other than that, I don't foresee an issue. Ext2/Ext3 can hold about 32k files/directories per directory. Just be sure to edit "#define AFPDIR 64" in FS.h

[rado99]

OK, I will realize it tomorrow and will write what I did. The algorithm I think to use:
File = new FILE(unique_ID,FILE_WRITE(to create a new file)).
And then, when I have saved the current id in the calling function, I will start reading them from the last one.

AFPDIR is for the maximum files per directory?

[1TekPro]

@rado99 It is for the expected number of files per directory. Normally this wouldn't need to be changed, but for your application it does.

[stickbreaker]

@rado99 This is what I use,

/* readLineReverse
  f  : a file opened in READ
  buf : pointer to buffer to receive data
  maxBufLen : size of buf, a NULL is appended, so max Data Length is -1
  result : length of data in buf, 0=at beginning of file, no more data available, or param error.
*/  
int readLineReverse(File &f, char * buf, uint32_t maxBufLen){
    if ((buf==NULL)||(maxBufLen<2)||(f==NULL)){
        return 0; // bad param
    }

#define REVERSE_BUFFER_SIZE 64
#define REVERSE_BUFFER_COUNT 2 // must be >=2

    static uint8_t revBuf[REVERSE_BUFFER_COUNT][REVERSE_BUFFER_SIZE];
    static uint32_t bufOfs[REVERSE_BUFFER_COUNT]={0};
    static uint32_t bufLen[REVERSE_BUFFER_COUNT]={0};
    static uint32_t cBuf=0;
    static File * oldFile =NULL;

    uint32_t outPos = 0; // amount of data copied to outbound buffer
    uint32_t pos = 0;

    if(f && !f.isDirectory()){
        uint32_t endPos = f.position(); //current position in file, where found line will end.

        if(endPos == 0) {
            buf[0]= '\0';
            return 0; // at beginning of file
        }
        
        uint32_t startPos = 0; // where this line starts in file
        int32_t foundBuffer = -1;
        if((oldFile==&f)&&(bufOfs[cBuf] <= endPos)&&((bufOfs[cBuf]+bufLen[cBuf])>endPos)){// data already in ram
            pos = endPos - bufOfs[cBuf];
        }
        else {
            pos = 0;
            bufOfs[cBuf] = endPos;
        }
        oldFile = &f;
        bool done = false;
        bool first = true;
        while(!done){
            while( !done && (pos > 0) ){
                pos--;
                done = (revBuf[cBuf][pos] =='\n')&&(!first);
                first=false;
            }
            if(!done && (bufOfs[cBuf] ==0)) {// at beginning of file, so done!
                done = true;
                pos--; // correct for increment in Done
            }
            if(done){ // move data into buffer
                foundBuffer = cBuf; //buffer holding oldest data, possibly cached for next iteration
                pos++; // next char after '\n'
                startPos = pos + bufOfs[cBuf];
                outPos = 0;
                uint32_t moveLen;
                if(bufOfs[cBuf] + bufLen[cBuf] > endPos){
                    moveLen = (endPos - bufOfs[cBuf])-pos;
                }
                else moveLen = bufLen[cBuf]-pos;
                
                if(( moveLen +outPos)>= maxBufLen) {
                    moveLen = (maxBufLen-1) -outPos;
                }
                if (moveLen > 0){ //data in this buffer
                    memmove(&buf[outPos],&revBuf[cBuf][pos],moveLen);
                    outPos += moveLen; // number of char add to out buf
                }
                while((( startPos + outPos) < endPos)&&( outPos < (maxBufLen-1))){ //more data required
                    if(cBuf > 0) cBuf--;
                    else cBuf = (REVERSE_BUFFER_COUNT-1);
                    if( bufOfs[cBuf] == (startPos + outPos)){ // data in this buffer
                        moveLen = bufLen[cBuf];
                        if( (startPos + outPos + moveLen)>endPos){ // maximum data available
                            moveLen = endPos - (startPos + outPos);
                        }
                        if(( moveLen + outPos) >= maxBufLen){ // output buffer too small
                            moveLen = (maxBufLen-1) - outPos;
                        }
                        if( moveLen > 0){ 
                            memmove(&buf[outPos],&revBuf[cBuf][0],moveLen);
                            outPos += moveLen;
                        }
                    }
                    else { // read next into this buffer
                        bufOfs[cBuf] = startPos + outPos;
                        f.seek(bufOfs[cBuf],SeekSet);
                        bufLen[cBuf] = REVERSE_BUFFER_SIZE;
                        bufLen[cBuf] = f.read((uint8_t*)&revBuf[cBuf][0],bufLen[cBuf]);
                        if(cBuf == foundBuffer) foundBuffer = -1;
                        cBuf++;
                        cBuf %= REVERSE_BUFFER_COUNT;
                    }
                }
            }
            else { // need to go back one more block
                uint32_t oldBuf = cBuf;
                cBuf++;
                cBuf %= REVERSE_BUFFER_COUNT;
                bufLen[cBuf] = REVERSE_BUFFER_SIZE;
                bufOfs[cBuf] = bufOfs[oldBuf];
                if(bufLen[cBuf] > bufOfs[cBuf]){ // would backup past start of file
                    bufLen[cBuf] = bufOfs[cBuf];
                }
                bufOfs[cBuf] -= bufLen[cBuf];
                f.seek(bufOfs[cBuf],SeekSet);
                bufLen[cBuf] = f.read((uint8_t*)&revBuf[cBuf][0],bufLen[cBuf]);
                done = (bufLen[cBuf] == 0);
                pos = bufLen[cBuf];
            }
        }
        f.seek(startPos,SeekSet); // position for next
        if(foundBuffer != -1) cBuf=foundBuffer; //cached data is valid for next iteration
        buf[outPos]='\0';
        return outPos;
    }
    else {//
        Serial.printf(" objected passed as file parameter is not a file.\n");
    }
    buf[0] ='\0';
    return 0;
}

// usage
  
    uint32_t count;
    File file = SPIFFS.open("/hello.txt", FILE_READ);
    file.seek(0,SeekEnd);
    char buf[512];
    do{
        count = readLineReverse(file,buf,sizeof(buf));
        Serial.printf("%s",buf);
        }while(count>0);
    file.close();

readLineReverse() will return upto sizeof(buf)-1 characters, including \n. if 'string' is larger than buffer, then only buflen-1 is returned. The filePointer is positioned at the beginning of the returned line, so you can use the builtin file.read() to reread the whole line.

Chuck.

[rado99]

@stickbreaker
Thank you so much, it works, but it is very laggy ( I need to resend around 500KB file). Do you have a code, that appends the file, but in the beginning?

[1TekPro]

@rado99
I haven't personally attempted this yet, but maybe something like this... (edit: sorry about code formatting my phone doesn't let me)

void rewind(FILE *stream);

Then read the first chunk of data (need to know size for buffer)

File fs = SPIFFS.open("log", "r");
char buf[52];
while (fs.available()) {
fs.read((uint8_t *)buf, 49); // read 49 bytes into buf
if (buf[48] == '\n') {
} else {
buf[49] = fs.read();
if (buf[49] == '\n') {
buf[49] = 0;
} else {
buf[50] = fs.read();
buf[50] = 0;
}
}
}
fs.close();

[stickbreaker]

@rado99 the lagging could be related to SPIFFS, it has no index so Directed file offset result in rereading the file from the beginning. I just modified that code to use Relative Seek Offsets, SPIFFS might handle them better.
readlineReverse()

Chuck.

[1TekPro]

True. The more data you put into spiffs, the longer it takes to read. Spiffs is actually just 1 file.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This stale issue has been automatically closed. Thank you for your contributions.