CertStoreBearSSL.cpp is only file that pervents compile with -D FS_NO_GLOBALS

[isadora-6th]

Basic Infos

• This issue complies with the issue POLICY doc.
• I have read the documentation at readthedocs and the issue is not addressed there.
• I have tested that the issue is present in current master branch (aka latest git).
• I have searched the issue tracker for a similar issue.
• If there is a stack dump, I have decoded it.
• I have filled out all fields below.

Platform

• Hardware: NodeMCU-V1.0 (Esp8266)
• Core Version: espressif8266@2.6.2 , framework-arduinoespressif8266@3.20704.0 (newest platformio can download by version number)
• Development Env: Platformio
• Operating System: Windows

Settings in IDE

• Module: Nodemcu
• Flash Mode: dio
• Flash Size: 4MB
• lwip Variant: v2 Lower Memory
• Reset Method: nodemcu
• Flash Frequency: 40Mhz
• CPU Frequency: 160MHz
• Upload Using: SERIAL
• Upload Speed: other

Problem Description

Older project version used SDFat as lib needed for work.
I updated ESP8266 core and package, and currentrly using ESP8266SdFat
Older lib had no defined namespace for it, so i need to prevent different FS in global scope.

To prevent issue with different FS difinitons i used this flag
-D FS_NO_GLOBALS
So all code compiled ok except 1 fileCertStoreBearSSL.cpp
located at path
C:\.platformio\packages\framework-arduinoespressif8266\libraries\ESP8266WiFi\src\CertStoreBearSSL.cpp
pathcing file by adding fs:: before all FS stuff usage solved issue, all compiled fine.

[env:nodemcuv2]
platform = espressif8266@2.6.2
platform_packages = 
    platformio/framework-arduinoespressif8266@3.20704.0
board = nodemcuv2
framework = arduino
build_flags = 
    -D FS_NO_GLOBALS
lib_deps =
    ESP8266WiFi
    SPI
    ESP8266SdFat

Sketch

#include <Arduino.h>
#include <ESP8266WiFi.h>
#include <SdFat.h>
/* Compiles only with -D FS_NO_GLOBALS */
using namespace sdfat;
File file;
/*------------------------------*/
void setup(){

}

void loop(){

}

Debug Messages

Processing nodemcuv2 (platform: espressif8266@2.6.2; board: nodemcuv2; framework: arduino)
-----------------------------------------------------------------------------------------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif8266/nodemcuv2.html
PLATFORM: Espressif 8266 (2.6.2) > NodeMCU 1.0 (ESP-12E Module)
HARDWARE: ESP8266 80MHz, 80KB RAM, 4MB Flash
PACKAGES:
 - framework-arduinoespressif8266 3.20704.0 (2.7.4)
 - tool-esptool 1.413.0 (4.13)
 - tool-esptoolpy 1.20800.0 (2.8.0)
 - toolchain-xtensa 2.40802.200502 (4.8.2)
LDF: Library Dependency Finder -> http://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 30 compatible libraries
Scanning dependencies...
Dependency Graph
|-- <ESP8266WiFi> 1.0
|-- <SPI> 1.0
|-- <ESP8266SdFat> 1.1.0
|   |-- <SPI> 1.0
Building in release mode
Compiling .pio\build\nodemcuv2\lib863\ESP8266WiFi\CertStoreBearSSL.cpp.o
C:\.platformio\packages\framework-arduinoespressif8266\libraries\ESP8266WiFi\src\CertStoreBearSSL.cpp:79:30: error: 'int BearSSL::CertStore::initCertStore' is not a static member of 'class BearSSL::CertStore'
 int CertStore::initCertStore(FS &fs, const char *indexFileName, const char *dataFileName) {
                              ^
C:\.platformio\packages\framework-arduinoespressif8266\libraries\ESP8266WiFi\src\CertStoreBearSSL.cpp:79:30: error: 'FS' was not declared in this scope
C:\.platformio\packages\framework-arduinoespressif8266\libraries\ESP8266WiFi\src\CertStoreBearSSL.cpp:79:30: note: suggested alternative:
In file included from C:\.platformio\packages\framework-arduinoespressif8266\libraries\ESP8266WiFi\src\CertStoreBearSSL.h:26:0,
                 from C:\.platformio\packages\framework-arduinoespressif8266\libraries\ESP8266WiFi\src\CertStoreBearSSL.cpp:20:
C:\.platformio\packages\framework-arduinoespressif8266\cores\esp8266/FS.h:197:7: note:   'fs::FS'
 class FS
       ^
C:\.platformio\packages\framework-arduinoespressif8266\libraries\ESP8266WiFi\src\CertStoreBearSSL.cpp:79:36: error: expected primary-expression before ',' token
 int CertStore::initCertStore(FS &fs, const char *indexFileName, const char *dataFileName) {
                                    ^
C:\.platformio\packages\framework-arduinoespressif8266\libraries\ESP8266WiFi\src\CertStoreBearSSL.cpp:79:38: error: expected primary-expression before 'const'
 int CertStore::initCertStore(FS &fs, const char *indexFileName, const char *dataFileName) {
                                      ^
C:\.platformio\packages\framework-arduinoespressif8266\libraries\ESP8266WiFi\src\CertStoreBearSSL.cpp:79:65: error: expected primary-expression before 'const'
 int CertStore::initCertStore(FS &fs, const char *indexFileName, const char *dataFileName) {
                                                                 ^
C:\.platformio\packages\framework-arduinoespressif8266\libraries\ESP8266WiFi\src\CertStoreBearSSL.cpp:79:89: error: expression list treated as compound expression in initializer [-fpermissive]
 int CertStore::initCertStore(FS &fs, const char *indexFileName, const char *dataFileName) {
                                                                                         ^
C:\.platformio\packages\framework-arduinoespressif8266\libraries\ESP8266WiFi\src\CertStoreBearSSL.cpp:79:91: error: expected ',' or ';' before '{' token
 int CertStore::initCertStore(FS &fs, const char *indexFileName, const char *dataFileName) {
                                                                                           ^
C:\.platformio\packages\framework-arduinoespressif8266\libraries\ESP8266WiFi\src\CertStoreBearSSL.cpp:230:1: error: expected '}' at end of input
 }
 ^
*** [.pio\build\nodemcuv2\lib863\ESP8266WiFi\CertStoreBearSSL.cpp.o] Error 1

Patched File log.

Processing nodemcuv2 (platform: espressif8266@2.6.2; board: nodemcuv2; framework: arduino)
-----------------------------------------------------------------------------------------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif8266/nodemcuv2.html
PLATFORM: Espressif 8266 (2.6.2) > NodeMCU 1.0 (ESP-12E Module)
HARDWARE: ESP8266 80MHz, 80KB RAM, 4MB Flash
PACKAGES:
 - framework-arduinoespressif8266 3.20704.0 (2.7.4)
 - tool-esptool 1.413.0 (4.13)
 - tool-esptoolpy 1.20800.0 (2.8.0)
 - toolchain-xtensa 2.40802.200502 (4.8.2)
LDF: Library Dependency Finder -> http://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 30 compatible libraries
Scanning dependencies...
Dependency Graph
|-- <ESP8266WiFi> 1.0
|-- <SPI> 1.0
|-- <ESP8266SdFat> 1.1.0
|   |-- <SPI> 1.0
Building in release mode
Compiling .pio\build\nodemcuv2\lib863\ESP8266WiFi\CertStoreBearSSL.cpp.o
Retrieving maximum program size .pio\build\nodemcuv2\firmware.elf
Checking size .pio\build\nodemcuv2\firmware.elf
Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM:   [===       ]  32.9% (used 26912 bytes from 81920 bytes)
Flash: [===       ]  25.3% (used 263820 bytes from 1044464 bytes)
====================================================== [SUCCESS] Took 3.86 seconds ======================================================

Terminal will be reused by tasks, press any key to close it.

so I add fs::FS to lines
79, 100, 105, 178, 190, 195

Patched file

/*
CertStoreBearSSL.cpp - Library for Arduino ESP8266
Copyright (c) 2018 Earle F. Philhower, III

This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.

This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
*/

#include "CertStoreBearSSL.h"
#include <memory>


#ifdef DEBUG_ESP_SSL
#define DEBUG_BSSL(fmt, ...)  DEBUG_ESP_PORT.printf_P((PGM_P)PSTR( "BSSL:" fmt), ## __VA_ARGS__)
#else
#define DEBUG_BSSL(...)
#endif

namespace BearSSL {

extern "C" {
// Callback for the x509 decoder
static void dn_append(void *ctx, const void *buf, size_t len) {
  br_sha256_context *sha1 = (br_sha256_context*)ctx;
  br_sha256_update(sha1, buf, len);
}
}


CertStore::~CertStore() {
free(_indexName);
free(_dataName);
}

CertStore::CertInfo CertStore::_preprocessCert(uint32_t length, uint32_t offset, const void *raw) {
CertStore::CertInfo ci;

// Clear the CertInfo
memset(&ci, 0, sizeof(ci));

// Process it using SHA256, same as the hashed_dn
br_x509_decoder_context *ctx = new br_x509_decoder_context;
br_sha256_context *sha256 = new br_sha256_context;
if (!ctx || !sha256) {
  DEBUG_BSSL("CertStore::_preprocessCert: OOM\n");
  return ci;
}

br_sha256_init(sha256);
br_x509_decoder_init(ctx, dn_append, sha256, nullptr, nullptr);
br_x509_decoder_push(ctx, (const void*)raw, length);

// Copy result to structure
br_sha256_out(sha256, &ci.sha256);
ci.length = length;
ci.offset = offset;

// Clean up allocated memory
delete sha256;
delete ctx;

// Return result
return ci;
}

// The certs.ar file is a UNIX ar format file, concatenating all the 
// individual certificates into a single blob in a space-efficient way.
int CertStore::initCertStore(fs::FS &fs, const char *indexFileName, const char *dataFileName) {
int count = 0;
uint32_t offset = 0;

_fs = &fs;

// In case initCertStore called multiple times, don't leak old filenames
free(_indexName);
free(_dataName);

// No strdup_P, so manually do it
_indexName = (char *)malloc(strlen_P(indexFileName) + 1);
_dataName = (char *)malloc(strlen_P(dataFileName) + 1);
if (!_indexName || !_dataName) {
  free(_indexName);
  free(_dataName);
  return 0;
}
memcpy_P(_indexName, indexFileName, strlen_P(indexFileName) + 1);
memcpy_P(_dataName, dataFileName, strlen_P(dataFileName) + 1);

fs::File index = _fs->open(_indexName, "w");
if (!index) {
  return 0;
}

fs::File data = _fs->open(_dataName, "r");
if (!data) {
  index.close();
  return 0;
}

uint8_t magic[8];
if (data.read(magic, sizeof(magic)) != sizeof(magic) ||
    memcmp(magic, "!<arch>\n", sizeof(magic)) ) {
  data.close();
  index.close();
  return 0;
}
offset += sizeof(magic);

while (true) {
  uint8_t fileHeader[60];
  // 0..15 = filename in ASCII
  // 48...57 = length in decimal ASCII
  uint32_t length;
  if (data.read(fileHeader, sizeof(fileHeader)) != sizeof(fileHeader)) {
    break;
  }
  offset += sizeof(fileHeader);
  fileHeader[58] = 0;
  if (1 != sscanf((char *)(fileHeader + 48), "%d", &length) || !length) {
    break;
  }

  void *raw = malloc(length);
  if (!raw) {
    break;
  }
  if (data.read((uint8_t *)raw, length) != length) {
    free(raw);
    break;
  }

  // If the filename starts with "//" then this is a rename file, skip it
  if (fileHeader[0] != '/' || fileHeader[1] != '/') {
    CertStore::CertInfo ci = _preprocessCert(length, offset, raw);
    if (index.write((uint8_t *)&ci, sizeof(ci)) != (ssize_t)sizeof(ci)) {
      free(raw);
      break;
    }
    count++;
  }

  offset += length;
  free(raw);
  if (offset & 1) {
    uint8_t x;
    data.read(&x, 1);
    offset++;
  }
}
data.close();
index.close();
return count;
}

void CertStore::installCertStore(br_x509_minimal_context *ctx) {
br_x509_minimal_set_dynamic(ctx, (void*)this, findHashedTA, freeHashedTA);
}

const br_x509_trust_anchor *CertStore::findHashedTA(void *ctx, void *hashed_dn, size_t len) {
CertStore *cs = static_cast<CertStore*>(ctx);
CertStore::CertInfo ci;

if (!cs || len != sizeof(ci.sha256) || !cs->_indexName || !cs->_dataName || !cs->_fs) {
  return nullptr;
}

fs::File index = cs->_fs->open(cs->_indexName, "r");
if (!index) {
  return nullptr;
}

while (index.read((uint8_t *)&ci, sizeof(ci)) == sizeof(ci)) {
  if (!memcmp(ci.sha256, hashed_dn, sizeof(ci.sha256))) {
    index.close();
    uint8_t *der = (uint8_t*)malloc(ci.length);
    if (!der) {
      return nullptr;
    }
    fs::File data = cs->_fs->open(cs->_dataName, "r");
    if (!data) {
      free(der);
      return nullptr;
    }
    if (!data.seek(ci.offset, fs::SeekSet)) {
      data.close();
      free(der);
      return nullptr;
    }
    if (data.read((uint8_t *)der, ci.length) != ci.length) {
      free(der);
      return nullptr;
    }
    data.close();
    cs->_x509 = new X509List(der, ci.length);
    free(der);
    if (!cs->_x509) {
      DEBUG_BSSL("CertStore::findHashedTA: OOM\n");
      return nullptr;
    }

    br_x509_trust_anchor *ta = (br_x509_trust_anchor*)cs->_x509->getTrustAnchors();
    memcpy(ta->dn.data, ci.sha256, sizeof(ci.sha256));
    ta->dn.len = sizeof(ci.sha256);

    return ta;
  }
}
index.close();
return nullptr;
}

void CertStore::freeHashedTA(void *ctx, const br_x509_trust_anchor *ta) {
CertStore *cs = static_cast<CertStore*>(ctx);
(void) ta; // Unused
delete cs->_x509;
cs->_x509 = nullptr;
}

}

I think I should had make a Pull Request but never done any before,
For now i will try create patcher cmdlet to solve this for myself.
(due to C++ can't just unuse namespace)

[earlephilhower]

Core Version: espressif8266@2.6.2 , framework-arduinoespressif8266@3.20704.0 (newest platformio can download by version number)

@ivankravets , sorry to drag you in here but it seems the last PIO release for the ESP8266 Arduino is very old. Do you need to run something on your end to pick up the new releases, or did we break something you were using to detect new releases? We're on 2.7.4 now...

@isadora-6th, doing a pull request is pretty simple if you're familiar with git. See https://docs.github.com/en/free-pro-team@latest/desktop/contributing-and-collaborating-using-github-desktop/creating-an-issue-or-pull-request . Basically you'll fork the repo, clone it locally, make your changes to a branch, push the branch to your fork on GH, and then on the GW website for the repo homepage you'll have a "Create pull request" button you can use to get one in.

If that's too much, let us know and we can do it. But, since you have a fix, it'd be nice to get your name attached to it.

We should also have a NO_GLOBALS build somewhere in CI to catch this kind of thing, too.

[isadora-6th]

@earlephilhower, About new releases, 2.6.2 is platform version
framework-arduinoespressif8266 3.20704.0 (2.7.4)
2.7.4 as i understod arduino user-friendly version instead of 3.20704.0

As I can see from this part

CONFIGURATION: https://docs.platformio.org/page/boards/espressif8266/nodemcuv2.html
PLATFORM: Espressif 8266 (2.6.2) > NodeMCU 1.0 (ESP-12E Module)  <------ This line
HARDWARE: ESP8266 80MHz, 80KB RAM, 4MB Flash
PACKAGES:
 - framework-arduinoespressif8266 3.20704.0 (2.7.4)   <------ And this line
 - tool-esptool 1.413.0 (4.13)
 - tool-esptoolpy 1.20800.0 (2.8.0)
 - toolchain-xtensa 2.40802.200502 (4.8.2)

Made a pull request, thank you for instructions, they are very clear
#7685