Specification of FFI

[chorman0773]

What are the operational semantics of the following rust program:

extern "C"{
   fn foo();
}

fn main(){
    foo()
}

Two specific questions are asked here:

1. What is the behaviour of calling a "foreign" function that is defined and exported (#[no_mangle] or #[export_name]) in the same program and "accessible" via a (potentially-indirectly) linked rlib or dylib
   a. A subquestion is what is the behaviour of calling a "foreign" function defined and exported in Rust that is linked via a staticlib/cdylib
2. What is the behaviour of calling a "foreign" function that isn't defined in Rust (but in some other, potentially assembly) language.

The answer to question (1) is obvious: It's equivalent to a call to that function. (1)(a) and (2) are a bit more interesting.

A previously discussed answer is that the the AM performs an implementation-defined AM operation, paramaterized by the link_name of the external function, and if it matches the export_name of an exported function defined by the program, it behaves as an equivalent operation to a call to that function. Does this handle every case? What about signature mismatch in the (1), (1)(a), and (2) cases? What happens if it's exported from a dylib, but also defined in a staticlib (either rust or non-rust)? Or defined in two dylib or a dylib+cdyib.

@rustbot label +S-pending-design +A-ffi

[RalfJung]

I think the "opsem" in the title is a misnomer. There isn't an operational semantics for FFI (unless you want to fix the language on the other side). The interaction with the outside world is necessarily going to be axiomatic in style.

A previously discussed answer is that the the AM performs an implementation-defined AM operation

That's not how I would put it.

Basically for every FFI call (and inline asm block, this is closely related to #422) the programmer picks a corresponding "step" (in quotes because there's a lot that can happen here, with non-determinism and I/O and whatnot, it's a very big step) on AM states and then they have the proof obligation to show that

• the same step could have been performed by pure Rust code (to ensure universal reasoning principles applied by the compiler remain correct)
• the actual code linked in under that name is a valid implementation of that step (this requires reasoning about the implementation-specific relation between concrete machine states and AM states, which is generally unstable and undocumented, but we can make some careful promises about it to allow certain FFI patterns)

[chorman0773]

That's not how I would put it.

This was at least one of the ways it was discussed previously. I would prefer it would use the direct implementation-defined correspondance because:
a) It actually makes it fairly direct when you call rust code from rust code, if you could have made the call without using FFI. We don't even have to leave the opsem world, and we don't have to decide how a user goes about proving that a function written in rust performs the operation that function normaly performs on the AM.
b) It emphasises the distinction between inline asm (which is more likely to go with a proof model) and function calls, being that implementations are very much allowed to (and very much do) analyze FFI calls.

[chorman0773]

And note that the analysis may not even require LTO: Under lccc's abi definition it's possible to inline a #[no_mangle] function that's also declare #[inline] even via an external call, if the definition is brought into the current CGU/TU in some way, usually by the IR file that defines the inlineable version being included by a call to another inlineable function.

[RalfJung]

This was at least one of the ways it was discussed previously.

Maybe it has been, but I have been using the description in #421 (comment) for years now, so that is also surely one of the ways it has been discussed for a long time.

Saying "it's implementation-defined" is basically useless as far as I am concerned, then we still have to figure out what we will say for rustc. Our job is to produce something akin to rust-lang/rfcs#3355, so we're not done when we say "implementation-defined", we have a primary implementation we care to specify. We still need to work out what that spec looks like, and then we can decide whether for some reason we want to allow other implementations to define a different spec (but I see no strong motivation for doing that).

The equivalent of doing xlang-LTO is to run optimizations on the assembler level, on the final object file. Both of these are totally legal so I am not sure a fundamentally different approach is needed for asm and FFI. The justification for xlang LTO is that LLVM knows both the C and Rust AM (or rather, both of them have been refined into LLVM IR), and so now it is in control of both sides of this interaction and can reason about what happens across that interaction.

[chorman0773]

We still need to work out what that spec looks like, and then we can decide whether for some reason we want to allow other implementations to define a different spec (but I see no strong motivation for doing that).

IMO, this is necessary because of cross-lang LTO (and in lccc, this is just "LTO", there's not much special going on), and differences in linking (lccc for example uses slightly different search directories on x86_64-unknown-linux-gnu than rustc does, because it assumes that unknown means unknown, rather than assuming it means either unknown or pc), both static linking and dynamic linking. There are also definitely differences in how different targets and compilers will react in some of the more interesting cases, and especially for Rust-Rust FFI, we can't just say that extern "Rust" matches regardless of compiler (though rustc could very well specify that for rustc, as could others).

Saying "it's implementation-defined" is basically useless as far as I am concerned

Keep in mind that this is constrained "implementaton-defined", assuming we constrain in the Rust-Rust case, at least within a crate graph.

The equivalent of doing xlang-LTO is to run optimizations on the assembler level, on the final object file. Both of these are totally legal so I am not sure a fundamentally different approach is needed for asm and FFI.

(The use of xlang-LTO here is slightly confusing to me so I'm going to use cross lang lto)
While you can equivalently run those optimizations at an assembler level, IMO the current wording in the reference would prohibit running it together on generated assembly/machine code, and assembly/machine code from inline-assembly, since the compiler is explicitly enjoined from assuming the contents of the assembly block match what is actually executed at runtime. The difference with FFI is that no such prohibition exists on FFI calls, so cross-lang lto is allowed. This is why I believe there is necesarily a fundamental difference in how they are specified.

[CAD97]

I think we actually agree on semantics here, and just disagree on how exactly to communicate them. I'm going to attempt to restate the definition such that it satisfies both Ralf and Connor's intuitions:

Rust-Rust FFI

This is a special case of Rust-Unknown FFI, documented purely as an expository measure. There is no specialization of behavior.

• There exists exactly one definition of a function with a given link name.
• Function declarations are associated with the function definition with the same link name in the manner typical for the target.
• When the function declaration is called:
  • The function signatures of the called declaration and the definition are ABI compatible.
  • The function definition receives the function call as if the declaration and definition refer to the same item.

Rust-Unknown FFI

• There exists exactly one definition of a function with a given link name.
• Function declarations are associated with the function definition with the same link name in the manner typical for the target.
• There exists a shared execution semantic between Rust and the FFI language.
  • For Rust-Rust FFI, this is trivial.
  • In the limit, the target bytecode/assembly semantics are shared.
  • This can be an intermediate IR between the AM and target bytecode, such as LLVM-IR or LCCC-XIR.
• When the function declaration is called:
  • The function signatures of the called declaration and the definition are ABI compatible.
  • The function definition receives the function call as if the declaration and definition refer to the same item.
  • Evaluation of the function definition executes some number of Rust AM operations by executing operations in the shared lowered semantic that implement the Rust AM operations.
  • Evaluating the function definition does not execute any operations in the shared lowered semantic which would modify Rust AM implementation state in a way not possible via Rust AM operations.

---

Cross-language and/or link-time optimization is necessarily done over the domain of some shared IR that both sides lower to. Optimizations that preserve observable behavior in the shared IR necessarily preserve behavior of both higher-level languages, as a correctly lowered translation is at least as constrained w.r.t. behavior as the pre-lowering representation (and usually more constrained).

To note, though, optimizing at the level of target bytecode/assembly is difficult, and requires some assumptions beyond that provided purely by the target semantics, because practically everything about assembly is theoretically observable. At a bare minimum for a von-neuman architecture target, you need to assume that the code segment is never accessed as data. Potentially the only safe transformation is relocation.

I think that addresses how FFI to raw asm (whether inline or not) is different from FFI to another higher level language. Code segments implemented in raw assembly can't be optimized without a proof that there's absolutely no possible way on the target machine to differentiate between the written and optimized versions. (The exact same as for high level languages, just in a domain where much more is potentially observable.) Any cross-language optimization that doesn't uphold that proof burden is simply incorrect.

You can of course claw back some opportunity for optimizarion by defining the target behavior (and thus the domain where optimizations are done) as being less constrained than "what the hardware does," but any such redefinition of course apply to any developer-authored assembly as well.

Saying "it's implementation-defined" is basically useless as far as I am concerned

Even for solely a single rustc, "implementation defined" has some useful meaning, though perhaps it would be better called "target defined." There are definitely things which behave in a reliable manner per target, but in a different way on different targets. Anything related to how the Rust AM gets lowered is "implementation defined," even though we absolutely do still guarantee some properties after lowering (e.g. that C-compatible FFI operates in a C-compatible fashion).

[chorman0773]

Potentially the only safe transformation is relocation.

Note that there are also link relaxations, which are a form of optimization. These are also done by the linker, though, hense the name.

[RalfJung]

Ah sure, we'll have to say something about linker name resolution I guess. I'll stay out of that discussion since I know basically nothing about this subject. I would expect that we specify something based on what rustc does; having other implementations use a different algorithm here seems quite problematic -- but this is so far outside my realm I really can't contribute much here.

I was mostly concerned with the abstract/theoretical code reasoning aspects of FFI: linking with foreign code, potentially code written in a different language. And here I think @CAD97 captured my intent quite well: we shouldn't have to say anything special like "assembly code cannot be optimized by the compiler" or "cross-lang inlining is permitted"; this should all fall out of higher-level reasoning principles such as "each part of the program is executed according to the spec of the language it is written in". I expect the most tricky part here will be ironing out just which part of the "Rust AM / assembly" correspondence implemented by rustc (i.e., the relational invariant maintained throughout the execution of any Rust program which always tracks which Rust AM state corresponds to the current physical machine state) we want to make any stable promises about. This will be heavily target-dependent and includes things like how the stack is laid out and which registers have special roles (such as a stack pointer). Saying much here is challenging since things lime inlining and outlining can make the low-level target stack look quite different from the Rust AM stack.

[chorman0773]

I would expect that we specify something based on what rustc does; having other implementations use a different algorithm here seems quite problematic -- but this is so far outside my realm I really can't contribute much here.

Name resolution rules at a minimum will depend on the object format. If the algorithm is specified too precisely, then you'd effectively lock things to ELF+COFF+WASM. However, there are definitely rust-level decisions that can affect this - for example, whether #[no_mangle] functions in dylibs get default or protected visibility. I highly doubt that there's a good reason to limit choice in this regard either way. Name mangling is also implicated, because users could reverse engineer mangled names from the compiler (or read documentation on the matter) and those symbols will be resolved differently between compilers that use different algorithms. I'm sure there are other subtle ways there can be a difference, beyond the obvious one of where names are looked up in the first place depending on what libraries are linked, and different implementations may pull in different libraries for different reasons.

There are also subtle differences between both static and dynamic link editors, and even configuration of the same linkers. Linking with or without setting an option that generates a DT_BIND_NOW tag, or using the LD_BIND_NOW environment variable determines whether libraries loaded by dlopen throughout a program can affect name resolution. Specifying every single possible combination of behaviours accross the already long list of link editors is likely a non-starter.

[chorman0773]

As a note about how different rust-level implementations might differ in symbol lookup, lccc's abi trivially endorses the calls in the following program as semantically equivalent (and can even do so operationally to an extent, as it does not leave the "rust domain" to perform the calls):

// crate foo: Compile without any `-C metadata` so as to get the crate name as the stem. Note that this behaviour is not guarantee by the abi, but a stem is not added without any `-C metadata` arguments by the lccc rust frontend

// No inline attribute, this is defined in the crate somewhere
pub fn bar(x: u32) -> i32{
    x as i32
}

// crate bar

extern "Rust"{
     fn _ZN3foo3barEu(x: u32) -> i32;
}

fn main(){
     assert_eq!(foo::bar(u32::MAX), unsafe{_ZN3foo3barEu(u32::MAX)});
}

In fact, a rule used by lccc for that implementation-defined choice might be to say: "Every function defined in rust that does not have the #[no_mangle] or #[inline] attribute (other than #[inline(never)]) and is either declared pub or is accessible to the current module may be called within that module via it's external name as given by https://lccc.lcdev.xyz/lcrust/abi/v0#name-mangling" (The lccc standard library in fact relies on this rule in several places, notably the implementation of Caller::location()).

This would be a different rule for name resolution than rustc has, and the program is more than likely going to create a link error on rustc. Thus, at the very least, the set of symbols exported from rust and their link names are unspecified.

[RalfJung]

Isn't that the kind of situation we want to avoid -- programs working with some compilers but not others? That seems like a compatibility / ecosystem split hazard.