FreeBASIC CWstr

[Jeff Marshall]

to my understanding though internally using different definitions both return the same: an "any ptr" and a "byref wstring".

The CAST() as ANY PTR is irrelevant.  It actually doesn't matter for the string conversions.  What it allows is implicit cast to any ptr, and making the UDT passable to anything that would accept a pointer, which maybe makes passing a custom UDT to WINAPI functions easier, though there is a drawback.

For example, in this stripped down version:

Code: [Select]

sub procAnyPtr( byval arg as any ptr )
end sub

sub procDblPtr( byval arg as double ptr )
end sub

dim s as string
procAnyPtr( s ) '' error: type mismatch
procDblPtr( s ) '' error: type mismatch

type T
__ as integer
declare operator cast() as any ptr
end type

dim x as T
procAnyPtr( x ) '' No error - OK
procDblPtr( x ) '' No error - but probably should have

The implicit CAST() as ANY ptr, allows the UDT to be passed, without compiler error/warning to many kinds of parameters, at the cost any kind of type checking.  Possibly convenient, also possibly creating hard to find bugs.

[Jeff Marshall]

Typically i add something like this to the compiler´s code in appropriate places;

Code: [Select]

'*************************************************************************
' JK - check for dws
'*************************************************************************
      if env.clopt.dws then
dim jk__zz as zstring ptr

        if (dtype = FB_DATATYPE_STRUCT) then
          jk__zz = nd_text->subtype->id.name

          if (*jk__zz = "JK_CWSTR") or (*jk__zz = "CWSTR") or (*jk__zz = "CBSTR") or (*jk__zz = "DWSTR")then
            goto do_ustring     
          end if
        end if
      end if

I added #pragma dws", which sets compiler option "dws" when and as soon as found. All new code i added for getting rid of "**" is enclosed by an "IF ... END IF" clause. If "#pragma dws" is not present in the code to compile, my changes don´t become active. Initially this was a meant to be a switch for testing the compiler with and without my code, in case there were problems with the re-compiled compiler - fortunately there weren´t any. In fact this pragma is not necessary for what i want to do, but it could be re-used for other things or be removed entirely.

If this were the solution, and we are talking fbc compiler internals, then the #pragma should just affect the UDT.  Similar to internal macros symbSetUDTIsUnion() where a status bit is attached to the UDT's information.  That way you only need to test that the UDT's typedef has the bit set and it is not tied to a specific UDT name.  This makes the solution more generic.  I've come across this many times when fixing bugs in the compiler, having to identify a root cause and solve it there, rather than the end use.

Bottom line is, it's better to solve the problem at the point where it is caused, rather than at the point which you notice it.  Unfortunately, the point where you notice it is usually more obvious than the point at which it is caused.

[Jeff Marshall]

I think, i understand the problem (for the compiler) you describe with multiple cast operators. When working on additional string handling functions, which should work for the new type(s) and the existing ones as well i got this error many times. But i was able to avoid it by adapting my code. And using all of this for quite some time, i didn´t experience errors (ambiguous ...), which couldn´t be fixed by tweaking the code. So i always thought, it´s my bad coding (being not that experienced in FreeBASIC), rather than a possible compiler problem. The bottomline is: i understand the problem you describe, but for me personally it didn´t occur (as of now) - the compiler seems to do it right.

It would help to have an example of failing code coming from this problem! 

What "works" and what is "correct" are completely different.  While you may observe that LTRIM "works" with the new type, internally I would say that LEFT is more "correct".  And it all has to do with implicit CAST overload resolution.

If we are implementing the dynamic wide string type (dwstring) as a UDT, then for implicit CASTing it boils down to:

Code: [Select]

type T
    __ as integer
   declare operator CAST() as wstring ptr
   declare CAST as string
end type
because "WSTRING PTR" and "STRING" are the only 2 types that fbc really knows how to handle.  But this is what introduces the ambiguous call that fbc (currently) does not know how to resolve.

If you look at https://github.com/jayrm/dwstring/blob/master/tests.bas , it provides an example.  Let me know if you have any trouble compiling or using.  It seems to me that the pattern for PB'ers is to include everything from one file, containing the complete implementation.  I understand the benefits.  For me, there is benefit having the interface (.bi) file separate from the implementation (.bas) file.

I don't know what your tests are, but the reason you didn't encounter any errors, is that you are specifically looking at dwstring.bi (and it's variants) as the "correct" way to implement the class.  Jose has done a superb job at that, but he is also working around all the current rules and quirks of fbc, and trying to make something usable within those constraints.

I guess what I'm trying to say is that Jose's implementations are very good for the current state of fbc, but that doesn't necessarily make them "correct", if it's possible to say such a thing as "correct".  If fbc has different (hopefully better) rules, then the implementation will be different.  The rules of the compiler define the required implementation, not the other way around.  Fortunately, we are in a position to set the rules.

[Jeff Marshall]

I didn´t have a look at how the compiler resolves overloaded functions or operators. If there really is a problem with the new type, it must be fixed. But IMHO making this a general thing for all types is not a good idea. I would rather be notified about a problem (ambiguous ...), so i´m forced to fix it exactly the way i want it to work, instead of the compiler making "guesses" (even if these follow rules), which might result in malfunctioning code under the hood. A pragma would switch this feature on or off for all types in use (which as described above i would avoid). They only thing that would make sense to me, is having a new keyword like "DEFAULT" to mark a function or operator as the default one to take, if the compiler cannot resolve it, e.g:

Code: [Select]

PRIVATE OPERATOR (OVERLOAD) DEFAULT DWSTR.CAST() BYREF AS WSTRING

This gives individual control to the coder without making (maybe unwanted, because the coder isn´t aware of the ambiguity at all) guesses. If someone decides to code "DEFAULT", then he must be aware of it and then it is his responsibility.

Currently, fbc's #pragma's currently affect the compiler's decisions in a global way, they are not specific to context.  I am thinking that, if there is no formal way to declare the behaviour in the TYPE's syntax, then a #pragma could help control it, but it should be for the UDT specifically, and not globally.  Or use the #prgram PUSH/POP mechanism for it.

The "default" overload resolution basically follows what would happen in a C++ program.  It doesn't have to happen that way, it's just what we've chosen as a baseline, so far.

Specifically, in the fbc compiler code, we'd be looking at symb-proc.bas:symbFindCastOvlProc() and maybe symbFindClosestOvlProc().  Rather than finding all possible matches, an algorithm to find the best possible match, even if there are multiple matches that would work.

[Juergen Kuehlwein]

Jeff,


a lot to read and digest, for me (Germany) it´s past midnight now - more tomorrow...


JK

[Juergen Kuehlwein]

Jeff,

But, you genuinely seem to have an enthusiasm for improvement and I think we both want to see a solution here.

Yes!


First of all you must know i live in Germany and i´m not a native speaker, so there is always a chance of misunderstandings and incorrect wording on my side, especially with a quite complicated matter like this.


Then you must know, that my first and main goal was having a working solution, maybe not perfect from a compiler coder´s perspective - but perfectly working. So i coded a quick (and dirty) approach in order to get working seamlessly, what we currently have. My initial intention was to keep changes as minimal as possible, because i thought a minimal approach would be more likely to be accepted and integrated into master than a generic one, which would require changing much more code.


@post #119

What we really want, is that fbc knows we want to use use the WSTRING version of LTRIM long before we ever get to rtlStrLTrim. Which means that the issue is earlier on in the translation.

I know and i understand that the place i applied my changes is not the correct place for a generic solution. But i intentionally coded a specific solution for these specific variations of a dynamic wide string type we have. I didn´t want to code a generic solution.

So what i coded basically works for what we have, but it doesn´t resolve inconsistencies and quirks in fbc. We are aware of these inconsistencies and as you said José sure had a hard time to find workarounds.

If you tell me you want a generic solution - fine. This opens a totally new perspective


@post #120
Maybe my wording was misleading. Of course "any ptr" and "byref wstring" are not the same, i know that. What i wanted to say was, José and Marc define the internal data buffer differently, both have two cast operators, nevertheless José´s cast operators return the same types as Marc´s (one of them returns an "any ptr" and the other one returns a "byref wstring").

Regarding "any ptr" i cannot tell where and why it is needed, but i can tell that linking fails, when you outcomment it. Maybe it is necessary for "MultiByteToWideChar" and "WideCharToMultiByte", which are called in José´s code. José is the creator and mastermind of all of this - maybe he can tell us.

As far as i know there are intrinsic ansi/wide conversion functions, maybe implementing these, would let us get rid of the need for casting to any ptr.


@post # 121
forget about that pragma thing, i don´t want it, i don´t need it. As already said, it was security and debugging thing for my personal use and maybe one more argument to convince you to accept changes in the compiler. Implementing a pragma you can just switch off my code. I didn´t expect you to want and to accept more than minimal changes to the compiler.


@post #122
As said above my first goal was to have a working solution not a "correct" one. A correct one requires much more changes to the compiler than i did - i understand that.

Jose has done a superb job at that, but he is also working around all the current rules and quirks of fbc, and trying to make something usable within those constraints.

What else should he(we) have done? Left and Right can be overloaded, the others can´t.

If we are implementing the dynamic wide string type (dwstring) as a UDT, then for implicit CASTing it boils down to:

type T
    __ as integer
   declare operator CAST() as wstring ptr
   declare CAST as string
end type

because "WSTRING PTR" and "STRING" are the only 2 types that fbc really knows how to handle.  But this is what introduces the ambiguous call that fbc (currently) does not know how to resolve.

Maybe i still don´t understand the problem, but neither José nor Marc have or need a cast as string operator. In fact you don´t need to implicitly cast the data to a string type as you do in your code. If you really want to do that (converting from wide to ansi isn´t always a lossles conversion) the compiler does it for you automatically.

e.g

Code: [Select]

dim s as string
dim u as ustring

  u = "123"
  s = u
 
print s 

works as expect with José´s CWSTR type. In my opinion because it isn´t a lossless conversion, there shouldn´t be any implicit casting form wide to ansi, there should be an error message (type mismatch, ambiguous..., whatever). This helps avoiding possible code malfunction coming from an inadverted conversion.

Please have a closer look at Jose´s string helper functions (AfxStr.inc) in his WINFBX suite, or have look at the attached file, which contains similar string handling functions built-in in my IDE. These functions work for all available string types (STRING, ZSTRING, WSTRING and USTRING) ansi to wide and wide to ansi conversions are done automatically. If you use my IDE you must add #include "ustring.inc" to be able to use the built-in dynamic wide string type (USTRING, essentially a clone of José´s CWSTR) and  the mentioned string helper functions. There is more information in the help file (FreeBASIC/...)



@post #123

but it should be for the UDT specifically

So why use a pragma for this, why not add to the syntax of types in general. In parser-proc.bas there is a function "cProcHeader" parsing a procedure´s head line. Why not parse it for "DEFAULT", and set the attrib parameter accordingly (requires to add "FB_SYMBATTRIB_DEFAULT" to FB_SYMBATTRIB enum) and check for this attribute later on, if needed for overload resolution in the functions you mentioned.

This way ("DEFAULT" as a keyword) you must have it inside the necessary definitions for a type, a pragma could be anywhere in code, and it´s easy to oversee it when debugging. Another point is, you can use #define(s) for the types name (e.g "USTRING"). The only way to make a pragma type specific i can think of, is using the type´s name. Having #define(s) then could cause confusion.

 

Well, to summarize it:

- we want to have a working and "correct" solution for dynamic wide strings
- we have a working solution (José´s and Marc´s code + my adaptions for the compiler), which can deal with the current inconsistencies in fbc. This solution is specific to the new dynamic wide string type, it doesn´t solve existing problems in fbc, it is inconsistent in itself - but it works
- you want to have a more generic approach for UDTs and strings in general fixing those quirks and inconsistencies in fbc

Do you agree so far ?

As mentioned above either i still don´t understand it or there might be a misconception on your side (prove me wrong) in that there is a need for a "cast as string" operator (which definitely causes problems). To my understanding, there is no need for such an operator.


It´s not a problem for me, if you don´t accept my changes just as they are right now - let´s do it better. Personally i would like to have three things:

- the new type should be implemented as included file (José, Marc, my clone of José´s, maybe others)
- "USTRING" should become a reserved word for a #define for the new type (so everyone can code whatever he prefers: #define ustring JK_CWSTR/CWSTR/CBSTR/DWSTR/whatever). In other words, nobody should name his version of the new type "USTRING".
- it should allow for a seamless integration into fbc (being able to use the same syntax just like with other string types) and (if possible) it shouldn´t break already existing code, which implements José´s or Marc´s code.


Maybe it is possible to get rid of the cast as any ptr operator, then there would be only one cast operator left, so no chance for ambiguities anymore.


JK



PS: José could you please explain (possibly once more) what for is the cast as any ptr operator needed.

[José Roca]

> PS: José could you please explain (possibly once more) what for is the cast as any ptr operator needed.

To be able to use a CWSTR directly with some Windows API functions without having to use casting, e.g.

Code: [Select]

' // Writing to a file
DIM cwsFilename AS CWSTR = "тест.txt"
DIM cwsText AS CWSTR = "Дмитрий Дмитриевич Шостакович"
DIM hFile AS HANDLE = CreateFileW(cwsFilename, GENERIC_WRITE, 0, NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL)
IF hFile THEN
   DIM dwBytesWritten AS DWORD
   DIM bSuccess AS LONG = WriteFile(hFile, cwsText, LEN(cwsText) * 2, @dwBytesWritten, NULL)
   CloseHandle(hFile)
END IF

In the Windows API there are parameters declared as LPCWSTR, LPCVOID, LPBYTE, WCHAR PTR, etc., and some may cause an error or a warning if you pass a WSTRING PTR instead of ANY PTR.

[José Roca]

BTW I also don't use * and ** instead of only * by caprice. I needed to have the equivalents to VARPTR and STRPTR. I first used & to emulate VARPTR, but then I could not use & to get the address of the class, so I removed it. You may like or not my workaround, but it works. Also, being a COM programmer, I'm comforable using double indirection. For those not comfortable with it, they can use the vptr and sptr methods.

For example, in this code:

Code: [Select]

DIM cwsFilename AS CWSTR = "тест.txt"
DIM cwsText AS CWSTR = "Дмитрий Дмитриевич Шостакович"
DIM hFile AS HANDLE = CreateFileW(cwsFilename, GENERIC_WRITE, 0, NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL)
IF hFile THEN
   DIM dwBytesWritten AS DWORD
   DIM bSuccess AS LONG = WriteFile(hFile, cwsText, LEN(cwsText) * 2, @dwBytesWritten, NULL)
   CloseHandle(hFile)
END IF

hFile = CreateFileW(cwsFilename, GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, FILE_FLAG_SEQUENTIAL_SCAN, NULL)
IF hFile THEN
   DIM dwFileSize AS DWORD = GetFileSize(hFile, NULL)
   IF dwFileSize THEN
      DIM cwsOut AS CWSTR = WSPACE(dwFileSize \ 2)
      DIM bSuccess AS LONG = ReadFile(hFile, *cwsOut, dwFileSize, NULL, NULL)
      CloseHandle(hFile)
      PRINT cwsOut
   END IF
END IF

I'm using *cwsOut with ReadFile.

[Juergen Kuehlwein]

Thanks José,


so the cast to any ptr operator could basically be dropped, if Jeff still sees a problem here. You added it for convenience, but without it explicit casting to the required type would work as well  - no other effects, than being not so convenient.

Is this correct ?

BTW I also don't use * and ** instead of only * by caprice. I needed to have the equivalents to VARPTR and STRPTR

I think we all know and understand that. The good thing about changing the compiler is, that indirection is not necessary anymore and the other good thing is, that you can still use indirection, if you want. You can have it both ways then


JK

[José Roca]

> Is this correct ?

Yes, it will be called only when the target is a pointer other than a wstring pointer. If you remove it, then you will need to use casting in some cases.

I will be very pleased if support for dynamic unicode strings is implemented in the compiler and more still if they also add support for BSTRings.

Every time that somebody asked for dynamic unicode string support, the replies were that it was unlikely to happen, so I decided to write my own class. The first one that I wrote was CBSTR, that deals with BSTRings. Then, to try to make it to work faster, I started to write a string builder class. I used an UBYTE PTR instead of a WSTRING PTR because it was intended to work with ansi and unicode. Then I decided to relegate CBSTR for use with COM and convert the string builder into a class to work with dynamic null terminated strings.

[Juergen Kuehlwein]

Ok José,


i tried removing the cast as any ptr operator, but this breaks a lot of the string helper functions (essentially the same as yours with some variations) in that now "**" is needed in places, it wasn´t necessary before - even with the adapted compiler version. So this cast as any ptr operator is maybe used in more places than we think right now.

Having a cast as any operator let´s me get away with my fairly simple changes to the compiler, removing it breaks integration in places where it worked before. So integrating CWSTR into the compiler seems to work mostly because of this operator.


Jeff,

is that true, and is it that, what is giving you headaches, with the current version of the new type and my changes to the compiler ? Without a cast as any ptr operator, the only cast operator left is cast as byref wstring, and then it needs different coding to get it to work. Is it this, what you consider as the "correct" way ?

Regarding the intrinsic string functions ("TRIM", MID", etc.) "as any" is the easy way, but there is no type checking at all and you could literally pass anything, even wrong types introducing hard to find bugs. You want better type checking there, so that only "approved" data is passed, therefore the cast as any operator should be removed or type checking shouldn´t let pass as any pointers - right ?


JK

 

[José Roca]

i tried removing the cast as any ptr operator, but this breaks a lot of the string helper functions (essentially the same as yours with some variations) in that now "**" is needed in places, it wasn´t necessary before - even with the adapted compiler version. So this cast as any ptr operator is maybe used in more places than we think right now.

I did mean that you can remove it from the class and the class will still work, not that it is not needed by my framework. If you remove it, you will have to modify some of the code. How much? I don't know and I don't care, because I'm not going to remove it.

[Marc Pons]

@Juergen
you are reactiving my interrest on that subject ( after 2 years)
My new DWSTR  include file is on github https://github.com/marpon/DWSTR
i have cleaned some mistakes from it

I've also done some simple tests with linux64, it seems working but not exhaustive tests done.
(i use linux not very often) and did not notice a real need for linux, coders have the opportunity to use UTF8 ...


@Jeff
happy to see you on that topic
i think it is only 2 solutions to include dynamic wide strings into FreeBasic

first option ,
           add a new type accepted by the parser as   dynamic wide string, with  all the code to work with included into compiler
           it is my prefered option for sure

second option,
          make minimal tweaks to help external codes to work in simplest way as today
          that's what Juergen is providing, i understand it's not as "correct" as the first option
          but with few tweaks on the native string handling functions it can make the deal (mainly for umbigous pointers)

@José

Yes, it will be called only when the target is a pointer other than a wstring pointer. If you remove it, then you will need to use casting in some cases.

i aggree with you, cast to any ptr simplilies way of doing in various case, what interest avoiding the ** or * for some cases if we have to explicitly cast in other cases?

[Jeff Marshall]

@Juergen, no, the cast as any ptr is not giving me headaches.  What I think of it or my personal preference, is irrelevant.  It's available as a choice for implementing the class, and the designer can make whatever choices they want.  I did look at many of the classes in WinFBX, and the use of '* operator' is used in a very consistent way to get a pointer to the underlying type.  And '**' obviously dereferences the pointer.  This pattern would work nicely for a variety of underlying types, not just wstring ptr.

Yes, I am looking for a solution that is a more generic than what you are proposing.  Swapping the data type based on the specific TYPE name at the last moment just before calling intrinsic string ignores many other issues related to UDT's, conversions, string/wstring disambiguation.  But if you really think this is the solution, then keep at it, create a pull request, and get the feedback from the other developers also.

I re-read your idea about the "DEFAULT" specifier and maybe something like that could work, with UDT's in general, not just [w]string's.  Allowing the programmer to tell the compiler what the preferred implicit conversion should be, and not be so strict about type matching.  Again, it will take time to develop.

@Marc, thanks,

first option, having built-in dynamic wide string is likely where we are headed.  But José is correct, the typical response is that it is unlikely to be done.  I predict it would take me about 4 months to add built in dwstring, if I work on nothing else but that.  We know the work involved tend to not promise something that is going to get delivered soon.

second option, what Juergen is providing is just not there yet, in my opinion.  It needs more.  Believe me, the quick and dirty fixes in the compiler end up being 10 year old bugs, or features that are now so difficult to implement they never get started.

So I don't know, I wish I had better news.  If it's just me, next 1.06 release probably will take about 2 months to get out, and updating bindings probably another 3 months, and maybe dwstrings after that for 4 months.

third option
In the meantime, I don't know, maybe just use the framework as-is, because that's the best option immediately available.  José has been generous enough to share in the hopes that it is useful.

I appreciate you guys welcoming me here and the discussion.

[Juergen Kuehlwein]

Jeff,

I appreciate you guys welcoming me here and the discussion.

ditto!



It´s amazing what José did, isn´t it ?

And '**' obviously dereferences the pointer.  This pattern would work nicely for a variety of underlying types, not just wstring ptr.

you need "**" everytime, where automatic casting (cast byref as wstring, in our case) doesn´t work. Apart from this you can use it like a native variable type. So maybe a clean solution would be to fix this - make implicit casting work everywhwere. This is, what you would prefer.

But i´m asking myself, is this really necessary everywhere and does it make sense? Coming for PowerBASIC with an Assembler background, initially i considered the extensive type casting needed in FreeBASIC a burden, e.g in PowerBASIC wparam and lparam of the SendMessage API are definded as LONG and you can pass nearly anything of long size to it without getting complains of the compiler. In Assembler there is essentially no type casting needed at all. In FreeBASIC i must cast even a LONG to wparam/lparam respectively. Ok - this language doesn´t take a relaxed stance on type casting, which enforces coding discipline and helps avoiding errors (to name to pros).

But why then would it be desirable to have automatic (implicit) casting from UDTs to other types, when it is possible to cast or convert explicitly and when casting is enforced everywhere else? This is inconsistent IMHO (if you don´t reduce the need for type casting everywhere else).


In our case we have a working extension of dynamic zero terminated wide strings closing an existing shortcoming in FreeBASIC. And we want to integrate it seamlessly into the compiler, which is a special case of a class-like UDT. We want to able to use just like all the other already existing string types. In all other cases i can live with (i would even prefer) the need of having to code castings or conversions.


I think initially nobody thought of the possibility of creating a string type the way José did it, therefore (and maybe, but this is speculation, on purpose) the code doesn´t account for UDTs in this case. I wouldn´t call it a bug. It´s just something nobody thought of being possible or even necessary. Looking at the compiler´s code i see some places, where this (automatic casting) already works, and i see many places, where it cannot work, because of how it is coded.

So i see three options:

- make the compiler work seamlessly with what we have
- make it work in general (re-work UDT type casting everywhere)
- make all statements overload-able like "LEFT" an others.

But if you really think this is the solution

I don´t think it is the solution, but it is a solution (the best one we have so far).

Swapping the data type based on the specific TYPE name

I explicitly check for UDT as datatype and i check for the name ("jk_cwstr, dwstring, cwstr, cbstr") of this udt, all other UDTs are rejected. Checking for the type´s name, this is what the compiler does for all other variable types too (e.g. "LONG"). "Long" is a reserved word as everybody knows, so where is the problem ? Establish "USTRING" (and the underlying type names) as reserved words for dynamic wide strings and everything will be fine.


My approach is a quick and dirty fix - yes. But as a first step it would allow using José´s dynamic wide strings just like native variable types (without the need for the otherwise extremly clever "**" workaround construct). The compiler can evolve (just like everthing else). Including my changes into the next release, doesn´t mean accepting it forever as it is, but accepting it, until we have something better. It doesn´t establish anything, which cannot be evolved any further and it doesn´t establish in what direction this evolution would have to go.

You could announce it as what it is: a first step, a still pragma isolated part (if we want to leave it as it is) of the compiler for those who want to use it. It doesn´t affect all others.


I´m confident to get this type running for Linux too, so it would really add value for all users not only the Windows fraction.



In the meantime i did a lot of tests with my changes and there will be code for thoroughly testing every aspect of the new type and it´s integration into the compiler. I hope to be able to present code and test code soon.


I cannot guarantee for an exact schedule, because i caught a cold this week and don´t feel very well, so i don´t know how much progress i can make in the next days. As soon as i´m ready, i will post here what i have. Then there must be tests for LINUX (i already have someone for this, but of course Jeff you are invited to test also). When everything is ok, i will push it to the repo and create a pull request.


JK