Progmem F() Strings Manager

[Hiddenvision]

Again, Not an issue but more of an Improvement thought.

I read that using Progmem for read only strings is all cool and dandy
but the compiling/linking process does not optimise repeated text used.

I was toying with a Strings manager concept that would handle all the static strings.
So you could perhaps litter your code with F_(55) where strings are required.
Save all the Indexed Strings in a txt file with a little tool to (add/search/replace/delete) entries.

F_(int i); being a simple function to return a constructed string.

An example optimizedStrings.h file

const char HTTP[] PROGMEM = "http:";
const char DOTCOM[] PROGMEM = ".com";
const char DOTCOUK[] PROGMEM = ".co.uk";
const char WEBSITE[] PROGMEM = "visitme";
const char HELPSITE[] PROGMEM = "helpme";
const char THANKS[] PROGMEM = "Thanks";
const char MR[] PROGMEM = "Mr:";
const char MRS[] PROGMEM = "Mrs:";
const char SMITH[] PROGMEM = "Smith";
const char PAGE[] PROGMEM = "Page ";
const char NOTHING[] PROGMEM = "Nothing ";
const char UNIQUE[] PROGMEM = "unique";
const char UNKNOWN[] PROGMEM = "UNKNOWN";

String F_(int i){
 switch (i) {
  case 1:     return (String) FPSTR(MR) + FPSTR(SMITH);
  case 2:     return (String) FPSTR(MRS) + FPSTR(SMITH);
  case 3:     return (String) FPSTR(HTTP) + FPSTR(WEBSITE) + FPSTR(DOTCOM);
  case 3:     return (String) FPSTR(HTTP) + FPSTR(HELPSITE) + FPSTR(DOTCOUK); 
  case 55:   return (String) FPSTR(NOTHING) + FPSTR(UNIQUE);
  case 721: return (String) FPSTR(PAGE) + FPSTR(UNIQUE) + FPSTR(THANKS); 
  default: return (String) FPSTR(UNKNOWN) + String(i); 
 }
}

end of optimizedStrings.h

As you can see from this random example trying to create and maintain this sort of thing by hand would be a nightmare but if technically this could save some space then to automate the creation of the builder function and const char progmem bits, is relatively easy to do from my point.
As a precompile process all the user strings can be studied and duplicates optimised.
Simple little exe to buz thru an indexed txt file doing the pre optimising stuff and outputting a fresh include file ready for compile.

Just wondering if this would be worth looking at or would it be a wasted folly.
I am no pro coder, I'll slap on more code than sunblock, until the resources run low,
and I'll be the first to say I can over complicate a problem that does not exist just for the fun of it.
It may even be the worst thing to even consider and produce horrendous output,
happy to take any comment.!

Now that I can do OTA > 500k, perhaps I wont even think of it again.
But wasted space is ditto.
I did this on the little Atmel devices when I was running low on space, 32k was easy to fill.!
That was not automated, but like I said, 32k is easy to fill, so I had limited text to manage.

It could be calculated that saving space takes time, so there is always a trade-off.

Hv

[Makuna]

I don't understand why you wouldn't do the top part, then just use those directly. Its seems more readable than F_(52).

[Hiddenvision]

@Makuna
Yes Readability goes out the door but then code is not to be read, it is to executed.!
And the more that can be squeezed in for execution the better on these little devices.
With a little manager to search for and SEE the strings F_(42) would look ok.
Personally I like short easy to read code and try to keep all the clutter backstage.

But comes the question of how much code space is used for joining strings.
Is there a letter count threshold to make it worthwhile.?
Who knows such things.?

Serial.Print(F("Mr. Smith"));
Serial.Print((String) F("Mr.") + F("Smith"));
Serial.Print(F_(1));

[Hiddenvision]

Actually what would be good is a resource file area in an unused section of flash.
With an update process and some method of mapping that area to an F_() type function.

Then all the strings could live in upper flash just below the FS and not take up any of the limited Sketch space. If what I hear about the Maximum Sketch size being 1MB then to off-load static resources from
the sketch area could be handy.

Plus, updates would be faster if no resources had changed. (Extra bonus)

After having thought for a while,
I guess something like the 20K Flash allocation for simulated EEprom,
Then a version of the EEPROM libs remapped and tweeked with an index lookup.
too crazy for this time of day ???

that's it, the coffee ran out, the effects of the coffee expired, I'm done for today.

Hv.

[tonton81]

speaking of “resources”, you shouldn’t be using the “String” class. Character arrays are always the best.

[Hiddenvision]

@tonton81
Great tip, I know strings are lazy but, well that just suits my use from VB.
https://hackingmajenkoblog.wordpress.com/2016/02/04/the-evils-of-arduino-strings/

If I ever do a production thing I would take more care.!
Or pay someone else to write it properly.

I played around with the EEPROM libs,
Actually just renamed everything containing EEPROM to RESOURCES.
Changed the Sector pointer. and it compiled.
But I had not time to actually make use of or test it .
After reading the code I think it may be a little heavy for pulling individual strings
but an interesting thought none the less.

Hv.

[vdeconinck]

Yes Readability goes out the door but then code is not to be read, it is to executed.!

I beg to disagree on that point.

[Hiddenvision]

@vdeconinck
no worries, that's what we are all here for.
But honestly how often do you need to "see" code once it is written and working.?

As far as maintenance, having all the resources in the one place seems like an easier approach.
Oh and when I mentioned Strings, I was being generic and just talking about things with letter in them.

But hey that's just my lazy mind stretching it's legs,
Personally I think the use of any undeclared constant should be outlawed.

[vdeconinck]

@Hiddenvision many experienced programmers agree that you should aim for 1.correctness 2.readabilty 3.performance, in that order, and 3. only if necessary ("premature optimization is the root of all evil").
Your proposal does not respect 2. and 3. (as you admitted) but it also does not respect 1. (nasty bug: you have two "case 3:" in your code).
In other words, you just proved my point: while you would think it's "working", it's not, and as it's not "readable" either, the bug went unnoticed and will byte you (or your users/customers) some time in the future.
You are very welcome to use such a manager in your code if you like but IMHO it wouldn't be wise to include it in this project (I admit I even fail to grasp how you could include it, as it's totally application dependent, but maybe that's just me).
Just my 2 cents.

[Hiddenvision]

I'd be switching 2 & 3 around myself.
But yep number 1 is in the right spot.
No point in writing code if it don't work right. (or at least with acceptable flaws!)

My background comes more from hacking and reverse engineering so prefer assembler if honest but c and such things do make portability easier.
As far as readability, I guess it is just different.
I would still rate a good resource manager over doing it by hand.

and being able to edit all the "Strings" in one single file, it worked for me
but then my mind is pretty shattered so I like stuff to be easy to find

But lets be real, bugs are our own creation mostly,
typing different things sometimes gives different results.
Re the two Case 3:'s
That's what copy and pasting on a browser does for ya,
But the compiler will speak to you in firm words if used in real life.
Correct it, move on and try not to do it again.
Plus if the file creation was automated then any errors will be software created and not down to Human intervention.

At the end of the day the compiler does not care about what stuff looks like as long as it fits thru the sieve.

But don't think I do not understand the need for well structured stuff in it's place.
Any public facing code should always be well dressed.

However, I would be curious to know if doing such a thing would result in a "space" saving.

[devyte]

@Hiddenvision you are correct: when strings are duplicate under normal use, the build process (supposedly) unifies duplicates. However, when those strings are moved to flash, then that optimization no longer works. Therefore, if there are duplicate strings (and we do have a few of those), then they are wastefully increasing the bin size. Removing duplicates therefore saves bin space.

As commented above, your original code structure has one Bad Idea: the switch statement with numerical index. This makes usage like F(55) very unreadable for posterity. True, code is to be executed, but after it executes the first time, that code has to be maintained. Bugs can appear, restructuring can be needed, rewrites, or even just a look to check on some behavior detail. It is because of these reasons that readability is important: code lives on after it has been written for the first time.

If strings are going to be moved to global space, one option is to have each string in its own variable, like you have above, and access them directly via that variable. The other option is to have all strings in a pointer array, and access via an index, in which case an enum with meaningful names is better than a numerical index that says nothing about the string. Here, you don't even need your function with the switch.

About your idea of a resource manager. I am currently doing a minor overhaul of our esp8266webserver, moving several strings to flash, others to global space and removing duplicates, and so on, and I have to say I find the process a bit painful, so I was thinking along similar lines. However, having a tool restructure your code for you makes me cringe a bit, especially when thinking about maintaining that tool cross-platform. Consider a subtle bug in that tool, which ends up messing up some user's code...
So, I thought of a simpler solution, at least as a first step: a string checker, that issues a build warning when it finds strings not in flash, or strings that are in flash that are duplicates.
Why just a checker? Most users who care enough about bin size also care about a warning-free build. At the very least, for those who don't care outright, they'll know that they could improve their code.
We could then eventually use that checker during our CI build process, and fail PRs that don't pass the check.

Problem for me is: there is no way I could justify spending the time to develop something like that on my end. Are you willing to tackle this yourself?

[everslick]

IMHO and experience: you run out of RAM long before you run out of flash. at least in general. there might be applications where this could be different, but then you can think about special deduplication strategies that are tailored to that use case.

[Hiddenvision]

@devyte Wise words indeed.
All of what people say is true, but there must be some compromise.

Just to be clear I was not thinking of anything that made changes to anyone's core code.

My thought,
You just type in a new string to the Resource-Manager and it give you a new index to use.
Or if you want to Check or edit the strings, enter an index and edit the string as desired.
No need to re-index so no need to back track thru other pages making changes.
Before compile, a separate "generated" include file is made.
the need for some sort of switch function is to "BUILD" the strings from the various parts that use duplicates. There maybe a better method to build these strings but something will need to exist.?

The Index's could be given "Names" but I fear that some of that may creep it's way into the binary depending on debug level as I am sure I see raw function names in the bin.

I would be happy to have a look at what is required to do this on a multi platform level but I would need to finish my own tool so it is publicly safe before I go creating larger monsters.

As mentioned my background is more taking stuff apart, understanding and improving,
and the tools I write are very unique to my needs and written as and while I need them.
Not to say my needs are unique, just the tools.!

I do "currently" have a HUGE amount of free time but not as much of the training and code discipline that maybe required (I score 0% in both!).
But glad to assist anyone else that may have more credentials.

@everslick
I think you maybe right, Ram is always desperate.
Then again almost everything is endangered on these embedded devices.
The first thing I do is run out of pins normally.

Most of my initial coding was done in .5k or 1k of PIC space.
Ahh, those long gone days of hunting for a 4 byte savings.
I used a few tricks on tightening up assembler code,
but again it was not as pretty to read but the silicon was happy.

Hv.

[devyte]

@Hiddenvision please contact me via gitter, there's something I'd like to discuss in addition to this :)

[Hiddenvision]

fire away.
you can email or PM on any place you find me.
Or post it here.

hv.